Category: Scraping using PHP

When building web applications, scraping data like emails from Google search results can be a valuable tool for marketing, lead generation, and outreach. In PHP, Guzzle, a powerful HTTP client, allows you to make HTTP requests to websites efficiently. In this blog, we’ll show you how to scrape emails from Google search results using Guzzle, covering setup, steps, and ethical considerations.

1. What is Guzzle?

Guzzle is a PHP HTTP client that simplifies sending HTTP requests and integrating with web services. It offers a clean API to handle requests, parse responses, and manage asynchronous operations. Using Guzzle makes web scraping tasks easier and more reliable.

2. Why Use Guzzle for Scraping?

• Efficiency: Guzzle is lightweight and fast, allowing you to make multiple HTTP requests concurrently.
• Flexibility: You can customize headers, cookies, and user agents to make your scraper behave like a real browser.
• Error Handling: Guzzle provides robust error handling, which is essential when dealing with web scraping.

3. Important Considerations

Before we dive into coding, it’s important to understand that scraping Google search results directly can violate their terms of service. Google also has anti-scraping mechanisms such as CAPTCHA challenges. For an ethical and reliable solution, consider using APIs like SerpAPI that provide search result data. If you’re scraping other public websites, always comply with their terms of service.

4. Getting Started with Guzzle

To follow along with this tutorial, you need to have Guzzle installed. If you don’t have Guzzle in your project, you can install it via Composer:

composer require guzzlehttp/guzzle

5. Step-by-Step Guide to Scraping Emails Using Guzzle

Step 1: Set Up the Guzzle Client

First, initialize a Guzzle client that will handle your HTTP requests.

use GuzzleHttp\Client;

$client = new Client([
    'headers' => [
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3',
    ]
]);

This user agent helps your requests appear like they are coming from a browser rather than a bot.

Step 2: Perform Google Search and Fetch HTML

In this example, we’ll perform a Google search to find websites containing the keyword “contact” along with a specific domain, and then extract the HTML of the results.

$searchQuery = "site:example.com contact";
$url = "https://www.google.com/search?q=" . urlencode($searchQuery);

$response = $client->request('GET', $url);
$htmlContent = $response->getBody()->getContents();

You can modify the search query based on your needs. Here, we’re searching for websites related to “example.com” that contain a contact page.

Step 3: Parse HTML and Extract URLs

After receiving the HTML response from Google, you need to extract the URLs from the search results. You can use PHP’s DOMDocument to parse the HTML and fetch the URLs.

$dom = new \DOMDocument();
@$dom->loadHTML($htmlContent);

$xpath = new \DOMXPath($dom);
$nodes = $xpath->query("//a[@href]");

$urls = [];
foreach ($nodes as $node) {
    $href = $node->getAttribute('href');
    if (strpos($href, '/url?q=') === 0) {
        // Extract the actual URL and decode it
        $parsedUrl = explode('&', str_replace('/url?q=', '', $href))[0];
        $urls[] = urldecode($parsedUrl);
    }
}

Here, we use XPath to identify all anchor (<a>) tags and extract the URLs associated with the search results.

Step 4: Visit Each URL and Scrape Emails

Once you have a list of URLs, you can visit each website and scrape emails using regular expressions (regex).

$emailPattern = '/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/';

foreach ($urls as $url) {
    try {
        $response = $client->request('GET', $url);
        $webContent = $response->getBody()->getContents();

        preg_match_all($emailPattern, $webContent, $matches);
        $emails = $matches[0];

        if (!empty($emails)) {
            echo "Emails found on $url: \n";
            print_r($emails);
        } else {
            echo "No emails found on $url \n";
        }
    } catch (\Exception $e) {
        echo "Failed to fetch content from $url: " . $e->getMessage() . "\n";
    }
}

This code uses Guzzle to visit each URL and then applies a regex pattern to extract all email addresses present on the page.

Step 5: Store the Extracted Emails

You can store the extracted emails in a file or database. Here’s an example of how to store them in a CSV file:

$csvFile = fopen('emails.csv', 'w');

foreach ($emails as $email) {
    fputcsv($csvFile, [$email]);
}

fclose($csvFile);

6. Handling CAPTCHA and Rate Limiting

Google employs CAPTCHA challenges and rate limits to prevent automated scraping. If you encounter these, you can:

• Implement delays between requests to avoid detection.
• Rotate user agents or proxy IP addresses.
• Consider using APIs like SerpAPI or web scraping services that handle CAPTCHA for you.

7. Ethical Scraping

Web scraping has its ethical and legal challenges. Always ensure that:

• You respect a website’s robots.txt file.
• You have permission to scrape the data.
• You comply with the website’s terms of service.

Conclusion

Scraping emails from Google search results using Guzzle in PHP is a powerful method for collecting contact information from public websites. Guzzle’s ease of use and flexibility make it an excellent tool for scraping tasks, but it’s essential to ensure that your scraper is designed ethically and within legal limits. As scraping can be blocked by Google, consider alternatives like official APIs for smoother data extraction.

As your email scraping efforts grow in scope, performance optimization becomes crucial. Extracting emails from large sets of web pages or handling heavy traffic can significantly slow down your PHP scraper if not properly optimized. In this blog, we’ll explore key strategies for improving the performance and scalability of your email extractor, ensuring it can handle large datasets efficiently.

We’ll cover:

• Choosing the right scraping technique for performance
• Parallel processing and multi-threading
• Database optimization for email storage
• Handling timeouts and retries
• Example code to optimize your scraper

Step 1: Choosing the Right Scraping Technique

The scraping technique you use can greatly impact the performance of your email extraction process. When working with large-scale scraping operations, it’s important to carefully select tools and strategies that balance speed and accuracy.

Using cURL for Static Websites

For simple, static websites, cURL remains a reliable and fast option. If the website doesn’t rely on JavaScript for content rendering, using cURL allows you to fetch the page source quickly and process it for emails.

function fetchEmailsFromStaticSite($url) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $html = curl_exec($ch);
    curl_close($ch);

    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $html, $matches);
    return array_unique($matches[0]);
}

For websites using JavaScript to load content, consider using Selenium, as discussed in the previous blog.

Step 2: Parallel Processing and Multi-threading

Scraping a single website at a time can be slow, especially when dealing with large numbers of pages. PHP’s pcntl_fork() function allows you to run processes in parallel, which can speed up your scraping.

Example: Multi-threading with pcntl_fork()

$urls = ['https://example1.com', 'https://example2.com', 'https://example3.com'];

foreach ($urls as $url) {
    $pid = pcntl_fork();
    
    if ($pid == -1) {
        die('Could not fork');
    } elseif ($pid) {
        // Parent process: wait for child
        pcntl_wait($status);
    } else {
        // Child process: execute the scraper for each URL
        scrapeEmailsFromURL($url);
        exit(0);
    }
}

function scrapeEmailsFromURL($url) {
    // Your scraping logic here
}

By running multiple scraping processes simultaneously, you can drastically reduce the time needed to process large datasets.

Step 3: Database Optimization for Storing Emails

If you are scraping and storing large amounts of email data, database optimization is key. Using MySQL or a similar relational database allows you to store, search, and query email addresses efficiently. However, optimizing your database is essential to ensure performance at scale.

Indexing for Faster Queries

When storing emails, always create an index on the email column. This makes searching for duplicate emails faster and improves query performance overall.

CREATE INDEX email_index ON emails (email);

Batch Inserts

Instead of inserting each email one by one, consider using batch inserts to improve the speed of data insertion.

function insertEmailsBatch($emails) {
    $values = [];
    foreach ($emails as $email) {
        $values[] = "('" . mysqli_real_escape_string($email) . "')";
    }

    $sql = "INSERT INTO emails (email) VALUES " . implode(',', $values);
    // Execute the query
}

Batch inserts reduce the number of individual queries sent to the database, improving performance.

Step 4: Handling Timeouts and Retries

When scraping websites, you may encounter timeouts or connection failures. To handle this gracefully, implement retries and set time limits on your cURL or Selenium requests.

Example: Implementing Timeouts with cURL

function fetchPageWithTimeout($url, $timeout = 10) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);  // Set timeout
    $html = curl_exec($ch);

    if (curl_errno($ch)) {
        // Retry the request if it failed
        return fetchPageWithTimeout($url, $timeout);
    }

    curl_close($ch);
    return $html;
}

This method ensures that your scraper won’t hang indefinitely if a website becomes unresponsive.

Step 5: Load Balancing for Large-Scale Scraping

As your scraping needs grow, you may reach a point where a single server is not enough. Load balancing allows you to distribute the scraping load across multiple servers, reducing the risk of being throttled or blocked by websites.

There are several approaches to load balancing:

• Round-Robin DNS: Distribute requests evenly across multiple servers using DNS records.
• Proxy Pools: Rotate proxies to avoid being blocked.
• Distributed Scraping Tools: Consider using distributed scraping tools like Scrapy or tools built on top of Apache Kafka for large-scale operations.

Step 6: Example: Optimizing Your PHP Scraper

Here’s an optimized PHP email scraper that incorporates the techniques discussed above:

function scrapeEmailsOptimized($url) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_TIMEOUT, 10);

    $html = curl_exec($ch);
    if (curl_errno($ch)) {
        curl_close($ch);
        return false;  // Handle failed requests
    }

    curl_close($ch);

    // Extract emails using regex
    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $html, $matches);
    return array_unique($matches[0]);
}

// Batch process URLs
$urls = ['https://example1.com', 'https://example2.com', 'https://example3.com'];
foreach ($urls as $url) {
    $emails = scrapeEmailsOptimized($url);
    if ($emails) {
        insertEmailsBatch($emails);  // Batch insert into database
    }
}

Conclusion

Optimizing your email extraction process is critical when scaling up. By using parallel processing, optimizing database interactions, and implementing timeouts and retries, you can improve the performance of your scraper while maintaining accuracy. As your scraping operations grow, these optimizations will allow you to handle larger datasets, reduce processing time, and ensure smooth operation.

As modern websites increasingly use JavaScript to load dynamic content, traditional scraping techniques using PHP and cURL may fall short. This is especially true when extracting emails from JavaScript-heavy websites. In this blog, we’ll focus on scraping emails from websites that render content via JavaScript using PHP in combination with headless browser tools like Selenium.

In this guide, we will cover:

• Why JavaScript rendering complicates email extraction
• Using PHP and Selenium to scrape JavaScript-rendered content
• Handling dynamic elements and AJAX requests
• Example code to extract emails from such websites

Step 1: Understanding JavaScript Rendering Challenges

Many modern websites, particularly single-page applications (SPAs), load content dynamically through JavaScript after the initial page load. This means that when you use tools like PHP cURL to fetch a website’s HTML, you may only receive a skeleton page without the actual content—such as email addresses—because they are populated after JavaScript execution.

Here’s where headless browsers like Selenium come in. These tools render the entire webpage, including JavaScript, allowing us to scrape the dynamically loaded content.

Step 2: Setting Up PHP with Selenium for Email Scraping

To scrape JavaScript-rendered websites, you’ll need to use Selenium, a powerful browser automation tool that can be controlled via PHP. Selenium enables you to load and interact with JavaScript-rendered web pages, making it ideal for scraping emails from such websites.

Installing Selenium and WebDriver

First, install Selenium for PHP using Composer:

composer require php-webdriver/webdriver

Then, make sure you have the ChromeDriver or GeckoDriver (for Firefox) installed on your machine. You can download them from the following links:

• ChromeDriver
• GeckoDriver

Next, set up Selenium:

1. Download the Selenium standalone server.
2. Run the Selenium server using Java:

java -jar selenium-server-standalone.jar

Step 3: Writing PHP Code to Scrape JavaScript-Rendered Emails

Now that Selenium is set up, let’s dive into the PHP code to scrape emails from a JavaScript-heavy website.

Example: Extracting Emails from a JavaScript-Rendered Website

Here’s a basic PHP script that uses Selenium and ChromeDriver to scrape emails from a page rendered using JavaScript:

require 'vendor/autoload.php';

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\WebDriverBy;

function scrapeEmailsFromJSRenderedSite($url) {
    // Connect to the Selenium server running on localhost
    $serverUrl = 'http://localhost:4444/wd/hub';
    $driver = RemoteWebDriver::create($serverUrl, DesiredCapabilities::chrome());

    // Navigate to the target URL
    $driver->get($url);

    // Wait for the JavaScript content to load (adjust as needed for the site)
    sleep(5);

    // Get the page source (fully rendered)
    $pageSource = $driver->getPageSource();

    // Use regex to extract email addresses from the page source
    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $pageSource, $matches);

    // Output the extracted emails
    if (!empty($matches[0])) {
        echo "Emails found on the website:\n";
        foreach (array_unique($matches[0]) as $email) {
            echo $email . "\n";
        }
    } else {
        echo "No email found on the website.\n";
    }

    // Close the browser session
    $driver->quit();
}

// Example usage
$target_url = 'https://example.com';
scrapeEmailsFromJSRenderedSite($target_url);

Step 4: Handling Dynamic Elements and AJAX Requests

Many JavaScript-heavy websites use AJAX requests to load specific parts of the content. These requests can be triggered upon scrolling or clicking, making scraping more challenging.

Here’s how you can handle dynamic content:

• Wait for Elements: Use Selenium’s built-in WebDriverWait or sleep() functions to give the page time to load fully before scraping.
• Scroll Down: If content is loaded upon scrolling, you can simulate scrolling in the page to trigger the loading of more content.
• Interact with Elements: If content is loaded via clicking a button or link, you can automate this action using Selenium.

Example: Clicking and Extracting Emails

use Facebook\WebDriver\WebDriverExpectedCondition;

// Navigate to the page
$driver->get($url);

// Wait for the element to be clickable and click it
$element = $driver->wait()->until(
    WebDriverExpectedCondition::elementToBeClickable(WebDriverBy::cssSelector('.load-more-button'))
);
$element->click();

// Wait for the new content to load
sleep(3);

// Extract emails from the new content
$pageSource = $driver->getPageSource();
preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $pageSource, $matches);

Step 5: Best Practices for Email Scraping

1. Politeness: Slow down the rate of requests and avoid overloading the server. Use random delays between requests.
2. Proxies: If you’re scraping many websites, use proxies to avoid being blocked.
3. Legal Considerations: Always check a website’s terms of service before scraping and ensure compliance with data privacy laws like GDPR.

Conclusion

Scraping emails from JavaScript-rendered websites can be challenging, but with the right tools like Selenium, it’s certainly achievable. By integrating Selenium with PHP, you can extract emails from even the most dynamic web pages, opening up new possibilities for lead generation and data gathering.

In the evolving landscape of digital marketing and lead generation, social media profiles often serve as a rich source of business information, including contact emails. This blog will focus on how to scrape emails from social media profiles using PHP, which can help you expand your email extraction toolset beyond just websites and PDFs.

In this blog, we will cover:

• Popular social media platforms for email extraction.
• Techniques to extract emails from platforms like Facebook, LinkedIn, Twitter, and Instagram.
• PHP tools and libraries to automate this process.

Step 1: Target Social Media Platforms for Email Scraping

Social media platforms are treasure troves of contact information, but they each present different challenges for scraping. Here are the most commonly targeted platforms:

• Facebook: Often includes emails on business pages or personal profiles under the “Contact Info” section.
• LinkedIn: Primarily used for professional networking, LinkedIn users may list email addresses in their profiles.
• Twitter: While not all profiles share emails directly, you can often find them in the bio section.
• Instagram: Many business accounts provide contact details, including email, under the profile description.

Before diving into the scraping process, remember that scraping social media profiles comes with ethical and legal concerns. Make sure you respect user privacy and abide by platform rules.

Step 2: Using PHP and cURL for Scraping Social Media Profiles

We’ll use PHP’s cURL library to fetch HTML content from the social media pages and regular expressions to extract email addresses. Let’s start by scraping Facebook pages.

Example: Scraping Emails from Facebook Pages

function scrapeEmailsFromFacebook($facebookUrl) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $facebookUrl);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    $html = curl_exec($ch);
    curl_close($ch);

    // Use regex to find email addresses
    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $html, $matches);

    if (!empty($matches[0])) {
        echo "Emails found on Facebook page:\n";
        foreach (array_unique($matches[0]) as $email) {
            echo $email . "\n";
        }
    } else {
        echo "No email found on Facebook page.\n";
    }
}

// Example usage
$facebook_url = "https://www.facebook.com/ExampleBusiness";
scrapeEmailsFromFacebook($facebook_url);

In this script, we make a request to the Facebook page and scan the resulting HTML content for email addresses. The preg_match_all() function is used to find all the emails on the page.

Example: Scraping LinkedIn Profiles

LinkedIn is one of the most challenging platforms to scrape because it uses dynamic content and strict anti-scraping measures. However, emails can often be found in the “Contact Info” section of LinkedIn profiles if users choose to share them.

For scraping LinkedIn, you’ll likely need a headless browser tool like Selenium to load dynamic content:

require 'vendor/autoload.php';  // Include Composer autoloader for Selenium

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\WebDriverBy;

$serverUrl = 'http://localhost:4444/wd/hub';
$driver = RemoteWebDriver::create($serverUrl, DesiredCapabilities::chrome());

$driver->get('https://www.linkedin.com/in/username/');

$contactInfo = $driver->findElement(WebDriverBy::cssSelector('.ci-email'));
$email = $contactInfo->getText();

echo "Email found on LinkedIn: $email\n";

$driver->quit();

In this example, Selenium is used to load the LinkedIn profile page and extract the email address from the “Contact Info” section.

Step 3: Extract Emails from Twitter Profiles

While Twitter users don’t typically display their email addresses, some may include them in their bio or tweets. You can use a similar scraping technique as Facebook to check for email addresses on the page.

function scrapeEmailsFromTwitter($twitterUrl) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $twitterUrl);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $html = curl_exec($ch);
    curl_close($ch);

    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $html, $matches);

    if (!empty($matches[0])) {
        echo "Emails found on Twitter profile:\n";
        foreach (array_unique($matches[0]) as $email) {
            echo $email . "\n";
        }
    } else {
        echo "No email found on Twitter profile.\n";
    }
}

// Example usage
$twitter_url = "https://twitter.com/ExampleUser";
scrapeEmailsFromTwitter($twitter_url);

Step 4: Scraping Instagram Business Profiles for Emails

Instagram business profiles often list an email in their contact button or profile description. You can extract this email by scraping the profile page.

function scrapeEmailsFromInstagram($instagramUrl) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $instagramUrl);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $html = curl_exec($ch);
    curl_close($ch);

    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $html, $matches);

    if (!empty($matches[0])) {
        echo "Emails found on Instagram profile:\n";
        foreach (array_unique($matches[0]) as $email) {
            echo $email . "\n";
        }
    } else {
        echo "No email found on Instagram profile.\n";
    }
}

// Example usage
$instagram_url = "https://www.instagram.com/ExampleBusiness";
scrapeEmailsFromInstagram($instagram_url);

Step 5: Handling Rate Limits and Captchas

Social media platforms are notorious for rate-limiting scrapers and employing CAPTCHA challenges to prevent bots. Here are some strategies for handling these issues:

• Slow Down Requests: Avoid making requests too quickly by adding random delays between each request.
• Use Proxies: To avoid getting your IP banned, rotate through different proxy servers.
• CAPTCHA Solvers: If CAPTCHA challenges are frequent, you may need to integrate third-party CAPTCHA-solving services.

Conclusion

Scraping emails from social media platforms using PHP is a powerful way to gather contact information for marketing, outreach, or lead generation. By targeting platforms like Facebook, LinkedIn, Twitter, and Instagram, you can extend your email extraction capabilities and collect valuable business data. Just remember to comply with each platform’s terms of service and follow best practices to respect privacy and avoid legal issues.

In today’s digital landscape, protecting email addresses from bots and spammers is a common practice. Many websites employ obfuscation techniques to hide their email addresses, making it challenging for automated tools to extract them. In this blog, we will explore various methods for decoding obfuscated emails, helping you effectively retrieve contact information while respecting ethical boundaries.

Understanding Email Obfuscation

Email obfuscation refers to the techniques used to protect email addresses from web scrapers and spammers. Common methods include:

• Encoding: Transforming the email into a different format (e.g., Base64, hexadecimal).
• JavaScript: Using JavaScript to generate or display email addresses dynamically.
• HTML Entities: Replacing characters in the email address with HTML entities.
• Cloudflare and Other Services: Using services like Cloudflare to obscure emails through protective measures.

By understanding these techniques, you can develop effective methods to decode these obfuscated emails.

Cloudflare

function decodeCloudflareEmail($encoded) {
    $r = hexdec(substr($encoded, 0, 2));  // Extract the first two characters for XOR operation
    $email = '';
    for ($i = 2; $i < strlen($encoded); $i += 2) {
        $email .= chr(hexdec(substr($encoded, $i, 2)) ^ $r);  // Decode each byte
    }
    return $email;
}

Akamai

function decodeAkamaiEmail($encoded) {
    // Example XOR decoding for Akamai
    $key = 0x5A;  // Example XOR key
    $email = '';
    for ($i = 0; $i < strlen($encoded); $i++) {
        $email .= chr(ord($encoded[$i]) ^ $key);  // Decode each character
    }
    return $email;
}

Incapsula

function decodeIncapsulaEmail($encoded) {
    // Assuming it's Base64 encoded for Incapsula
    return base64_decode($encoded);
}

JavaScript-based Encoding:

function decodeJavaScriptEmail($encoded) {
    return str_replace(['[at]', '[dot]'], ['@', '.'], $encoded);  // Common decoding
}

Conclusion

These functions cover the most commonly used methods for decoding obfuscated emails, especially from popular protection services. Each function is tailored to handle specific encoding techniques, ensuring you can effectively retrieve hidden email addresses.