Introduction
In the previous blogs, we explored how to scrape static web pages using Java and Jsoup. While Jsoup is an excellent tool for parsing HTML documents, it struggles with web pages that load content dynamically through JavaScript. Many modern websites rely heavily on JavaScript for displaying content, making traditional HTML parsing ineffective.
In this blog, we will look at how to scrape dynamic web pages in Java. To achieve this, we’ll explore Selenium, a powerful web automation tool, and show you how to use it for scraping dynamic content such as email addresses.
What Are Dynamic Web Pages?
Dynamic web pages load part or all of their content after the initial HTML page load. Instead of sending fully rendered HTML from the server, dynamic pages often rely on JavaScript to fetch data and render it on the client side.
Here’s an example of a typical dynamic page behavior:
- The initial HTML page is loaded with placeholders or a basic structure.
- JavaScript executes and fetches data asynchronously using AJAX (Asynchronous JavaScript and XML).
- Content is dynamically injected into the DOM after the page has loaded.
Since Jsoup fetches only the static HTML (before JavaScript runs), it won’t capture this dynamic content. For these cases, we need a tool like Selenium that can interact with a fully rendered web page.
Step 1: Setting Up Selenium for Java
Selenium is a browser automation tool that allows you to interact with web pages just like a real user would. It executes JavaScript, loads dynamic content, and can simulate clicks, form submissions, and other interactions.
Installing Selenium
To use Selenium with Java, you need to:
- Install the Selenium WebDriver.
- Set up a browser driver (e.g., ChromeDriver for Chrome).
First, add the Selenium dependency to your Maven pom.xml:
<dependencies>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.0.0</version>
</dependency>
</dependencies>
Next, download the appropriate browser driver. For example, if you are using Chrome, download ChromeDriver from here.
Make sure the driver is placed in a directory that is accessible by your Java program. For instance, you can set its path in your system’s environment variables or specify it directly in your code.
Step 2: Writing a Basic Selenium Email Scraper
Now, let’s write a simple Selenium-based scraper to handle a dynamic web page.
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class DynamicEmailScraper {
public static void main(String[] args) {
// Set the path to your ChromeDriver executable
System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
// Create a new instance of the Chrome driver
WebDriver driver = new ChromeDriver();
try {
// Open the dynamic web page
driver.get("https://example.com"); // Replace with your target URL
// Wait for the page to load and dynamic content to be fully rendered
Thread.sleep(5000); // Adjust this depending on page load time
// Extract the page source after the JavaScript has executed
String pageSource = driver.getPageSource();
// Regular expression to find emails
String emailRegex = "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,6}";
Pattern emailPattern = Pattern.compile(emailRegex);
Matcher emailMatcher = emailPattern.matcher(pageSource);
// Print out all found email addresses
while (emailMatcher.find()) {
System.out.println("Found email: " + emailMatcher.group());
}
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
// Close the browser
driver.quit();
}
}
}
Code Breakdown:
- We start by setting the path to ChromeDriver and creating an instance of
ChromeDriverto control the Chrome browser. - The
get()method is used to load the desired dynamic web page. - We use
Thread.sleep()to wait for a few seconds, allowing time for the JavaScript to execute and the dynamic content to load. (For a better approach, consider using Selenium’s explicit waits to wait for specific elements instead of relying onThread.sleep().) - Once the content is loaded, we retrieve the fully rendered HTML using
getPageSource(), then search for emails using a regex pattern.
Step 3: Handling Dynamic Content with Explicit Waits
In real-world scenarios, using Thread.sleep() is not ideal as it makes the program wait unnecessarily. A better way to handle dynamic content is to use explicit waits, where Selenium waits for a specific condition to be met before proceeding.
Here’s an improved version of our scraper using WebDriverWait:
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import java.time.Duration;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class DynamicEmailScraperWithWaits {
public static void main(String[] args) {
// Set the path to your ChromeDriver executable
System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
// Create a new instance of the Chrome driver
WebDriver driver = new ChromeDriver();
try {
// Open the dynamic web page
driver.get("https://example.com"); // Replace with your target URL
// Create an explicit wait
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
// Wait until a specific element (e.g., a div with class 'contact-info') is visible
WebElement contactDiv = wait.until(
ExpectedConditions.visibilityOfElementLocated(By.className("contact-info"))
);
// Extract the page source after the dynamic content has loaded
String pageSource = driver.getPageSource();
// Regular expression to find emails
String emailRegex = "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,6}";
Pattern emailPattern = Pattern.compile(emailRegex);
Matcher emailMatcher = emailPattern.matcher(pageSource);
// Print out all found email addresses
while (emailMatcher.find()) {
System.out.println("Found email: " + emailMatcher.group());
}
} finally {
// Close the browser
driver.quit();
}
}
}
How This Works:
- We replaced
Thread.sleep()with WebDriverWait to wait for a specific element (e.g., adivwith the classcontact-info) to be visible. - ExpectedConditions is used to wait until the element is available in the DOM. This ensures that the dynamic content is fully loaded before attempting to scrape the page.
Step 4: Extracting Emails from Specific Elements
Instead of searching the entire page source for emails, you might want to target specific sections where emails are more likely to appear. Here’s how to scrape emails from a particular element, such as a footer or contact section.
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class SpecificSectionEmailScraper {
public static void main(String[] args) {
// Set the path to your ChromeDriver executable
System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
// Create a new instance of the Chrome driver
WebDriver driver = new ChromeDriver();
try {
// Open the dynamic web page
driver.get("https://example.com"); // Replace with your target URL
// Wait for a specific section (e.g., the footer)
WebElement footer = driver.findElement(By.tagName("footer"));
// Extract text from the footer
String footerText = footer.getText();
// Regular expression to find emails
String emailRegex = "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,6}";
Pattern emailPattern = Pattern.compile(emailRegex);
Matcher emailMatcher = emailPattern.matcher(footerText);
// Print out all found email addresses in the footer
while (emailMatcher.find()) {
System.out.println("Found email: " + emailMatcher.group());
}
} finally {
// Close the browser
driver.quit();
}
}
}
Step 5: Handling AJAX Requests
Some websites load their content via AJAX requests. In these cases, you can use Selenium to wait for the AJAX call to complete before scraping the content. WebDriverWait can help detect when the AJAX call is done and the new content is available in the DOM.
Conclusion
In this blog, we covered how to scrape dynamic web pages using Selenium in Java. We explored how Selenium handles JavaScript, loads dynamic content, and how you can extract email addresses from these pages. Key takeaways include:
- Setting up Selenium for web scraping.
- Using explicit waits to handle dynamic content.
- Extracting emails from specific elements like footers or contact sections.
In the next blog, we’ll dive deeper into handling websites with anti-scraping mechanisms and how to bypass common challenges such as CAPTCHA and JavaScript-based blocking.