Category: Scraping using PHP

Introduction

Email extraction has become increasingly complex due to various protection mechanisms employed by websites. To build a robust email extraction tool using PHP and MySQL, it’s crucial to implement advanced techniques that address these challenges. In this blog, we’ll explore 20 advanced methods for email extraction, focusing on decoding obfuscated emails, handling modern web technologies, and overcoming common obstacles.

Let’s dive into these techniques!

1. Decoding Cloudflare-Obfuscated Emails

Websites using Cloudflare often obfuscate email addresses to protect against bots. The obfuscation typically involves encoding email addresses into hexadecimal strings.

function decodeCloudflareEmail($encoded) {
    $r = hexdec(substr($encoded, 0, 2));  // Extract the first two characters for XOR operation
    $email = '';
    for ($i = 2; $i < strlen($encoded); $i += 2) {
        $email .= chr(hexdec(substr($encoded, $i, 2)) ^ $r);  // Decode each byte
    }
    return $email;
}

// Usage
$encoded_email = 'data-cfemail="...";';  // Sample input
$decoded_email = decodeCloudflareEmail($encoded_email);

Ensure you correctly extract the data-cfemail attribute and handle various encoding formats.

2. Extracting Emails from HTML Comments

Some websites hide emails in HTML comments, making them invisible to regular scraping methods.

$content = file_get_contents('http://example.com');  // Fetch webpage content
preg_match_all('/<!--(.*?)-->/s', $content, $matches);  // Find all HTML comments
$emails = [];
foreach ($matches[1] as $comment) {
    preg_match_all($pattern, $comment, $emailMatches);  // Extract emails using regex
    $emails = array_merge($emails, $emailMatches[0]);
}

// Usage
print_r($emails);

Ensure your regex pattern is robust enough to capture various email formats.

3. Handling Base64-Encoded Emails

Websites may encode email addresses in Base64 to obscure them.

$encoded_email = 'dGVzdEBleGFtcGxlLmNvbQ==';  // Base64-encoded email
$decoded_email = base64_decode($encoded_email);
echo $decoded_email;  // Outputs: [email protected]

Be cautious of the different encoding schemes (like URL encoding) and ensure you decode them appropriately.

4. Decoding Hexadecimal Emails

Some email addresses are represented in hexadecimal format.

$hex_email = '74 65 73 74 40 65 78 61 6d 70 6c 65 2e 63 6f 6d';  // Hex representation
$decoded_email = implode('', array_map('chr', array_map('hexdec', explode(' ', $hex_email))));
echo $decoded_email;  // Outputs: [email protected]

Validate the input format to ensure it’s a proper hexadecimal string.

5. Extracting Emails from JavaScript Variables

Some websites assign email addresses to JavaScript variables, making them less accessible through standard scraping.

preg_match_all('/var\s+email\s*=\s*[\'"]([^\'"]+)[\'"]/', $content, $matches);  // Regex to find email
$emails = $matches[1];  // Store the extracted emails

// Usage
print_r($emails);

6. Bypassing CAPTCHA with OCR

CAPTCHAs can block automated bots. Optical Character Recognition (OCR) tools can be used to read these images.

exec('tesseract captcha_image.png output', $output);  // Use Tesseract to extract text
$email = trim(file_get_contents('output.txt'));  // Extract email from the OCR output

OCR accuracy may vary; consider using pre-processing techniques to enhance image quality before passing it to Tesseract.

7. Decoding JavaScript-Obfuscated Emails

Some sites hide emails using JavaScript functions, requiring reverse engineering of the script.

preg_match('/var encoded = "(.*?)"/', $content, $matches);  // Capture the encoded variable
$encoded = $matches[1];
$decoded = str_replace(['[at]', '[dot]'], ['@', '.'], $encoded);  // Decode the email

You may need to analyze the JavaScript to understand the obfuscation method used.

8. Handling UTF-16-Encoded Emails

Some emails might be encoded in UTF-16 format.

$utf16_email = '\u0074\u0065\u0073\u0074\u0040\u0065\u0078\u0061\u006d\u0070\u006c\u0065\u002e\u0063\u006f\u006d';  // Sample UTF-16 email
$decoded_email = json_decode('"' . $utf16_email . '"');
echo $decoded_email;  // Outputs: [email protected]

Ensure the string is formatted correctly for decoding.

9. Extracting Emails from PDFs with PHP

Emails may be hidden in PDF documents, which can be parsed to extract text.

$pdf_content = shell_exec('pdftotext file.pdf -');  // Use pdftotext to extract text from PDF
preg_match_all($pattern, $pdf_content, $matches);  // Extract emails using regex
$emails = $matches[0];

// Usage
print_r($emails);

Ensure you have the necessary tools installed (like Poppler or FPDF) and manage PDF parsing errors.

10. Scraping Emails from Image Files with OCR

Emails can also be present as images. Using OCR can help extract text from these images.

exec('tesseract email_image.png output', $output);  // Use Tesseract to read the image
$email = trim(file_get_contents('output.txt'));  // Extract email from the OCR output

The quality of the image can greatly affect OCR accuracy, consider pre-processing images to improve readability.

11. Using Anti-Scraping Services for CAPTCHA Solving

CAPTCHA services like 2Captcha can help automate solving CAPTCHAs.

$response = file_get_contents('http://2captcha.com/in.php?key=YOUR_API_KEY&method=userrecaptcha&googlekey=SITE_KEY&url=http://example.com');
$captcha_id = explode('|', $response)[1];

// Polling for result
do {
    sleep(5);  // Wait before requesting results
    $result = file_get_contents('http://2captcha.com/res.php?key=YOUR_API_KEY&action=get&id=' . $captcha_id);
} while ($result == 'CAPCHA_NOT_READY');

Using such services might incur costs, consider the balance between efficiency and budget.

12. Handling Emails in SVG Elements

Emails can sometimes be embedded within SVG graphics.

preg_match_all('/<text.*?>(.*?)<\/text>/', $content, $matches);  // Extract SVG text
$emails = [];
foreach ($matches[1] as $text) {
    preg_match_all($pattern, $text, $emailMatches);  // Extract emails using regex
    $emails = array_merge($emails, $emailMatches[0]);
}

// Usage
print_r($emails);

Ensure you are familiar with SVG structure, as it may vary between websites.

13. Processing Emails with Multiple Layers of Obfuscation

Some emails might undergo multiple encoding processes, requiring a systematic approach to decode.

$complex_email = '...';  // Your encoded email
$decoded_base64 = base64_decode($complex_email);
$decoded_hex = implode('', array_map('chr', array_map('hexdec', str_split($decoded_base64, 2))));
echo $decoded_hex;

Always validate the output after each decoding step to ensure correctness.

14. Extracting Emails from Social Media

Emails may be publicly listed in social media profiles, accessible through their APIs or scraping.

• Use API calls where available (like LinkedIn or Twitter) to fetch user profile data.
• Alternatively, scrape public profiles while respecting the platform’s terms of service.

Ensure compliance with social media policies regarding data scraping and respect user privacy.

15. Scraping Emails with Headless Browsers

Headless browsers like Puppeteer or Selenium can render JavaScript-heavy pages and extract visible emails.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('http://example.com')
emails = driver.find_elements_by_xpath('//a[contains(@href, "mailto:")]')
email_list = [email.get_attribute('href').replace('mailto:', '') for email in emails]
driver.quit()

Ensure you have the appropriate web driver and manage resources properly to avoid memory leaks.

16. Using SQL Queries for Email Validation

After extraction, validate emails using SQL queries to ensure they are correctly formatted.

SELECT email FROM users WHERE email REGEXP '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Z|a-z]{2,}';

Be mindful of potential SQL injection, use prepared statements to enhance security.

17. Monitoring Email Extraction Processes

Implement monitoring systems to track the performance of email extraction.

• Log every extraction attempt and result for auditing.
• Use analytics to understand user behavior and improve extraction methods.

Ensure your logging system does not compromise user privacy.

18. Integrating Email Extraction with Other Systems

Connect your email extraction tool with CRM systems for automated lead generation.

• Use APIs to send extracted emails directly to your CRM.
• Schedule regular extraction tasks for continuous data flow.

Ensure data consistency and manage API rate limits.

19. Testing Email Extraction with Unit Tests

Implement unit tests to ensure your email extraction logic works as intended.

public function testEmailExtraction() {
    $this->assertEquals('[email protected]', extractEmail('Contact us at [email protected]'));
}

Ensure you cover various edge cases and possible encoding scenarios in your tests.

20. Utilizing Machine Learning for Email Detection

Employ machine learning algorithms to enhance email detection accuracy, especially in complex content.

• Train a model on labeled data containing emails.
• Use libraries like TensorFlow or Scikit-learn for implementing your model.

Data preparation can be time-consuming; ensure you have a balanced dataset for effective training.

Conclusion

Incorporating these 20 advanced techniques into your email extraction strategy will enhance its effectiveness and adaptability to various challenges. By leveraging these methods, you can create a more resilient email extraction tool that can handle different obfuscation techniques, ensuring accurate and comprehensive email data collection.

Introduction

Welcome back to our email extraction series! Throughout our journey, we’ve covered various techniques for extracting emails from diverse content types, including static HTML pages, JavaScript-rendered content, and documents like PDFs and images. In this final installment, we will synthesize these techniques into a comprehensive email extraction tool using PHP and MySQL. This tool will empower you to efficiently extract email addresses from multiple input sources and store them systematically for further use.

Project Overview

Our objective is to create a PHP application that:

1. Accepts URLs for email extraction.
2. Identifies the content type (static HTML, JavaScript-rendered, PDF, etc.).
3. Extracts email addresses based on the content type.
4. Stores the extracted emails in a MySQL database for easy access and management.

By the end of this post, you will have a fully functional email extraction tool that can be further customized to suit your needs.

Setting Up the Environment

Before we dive into coding, ensure that you have the following set up in your development environment:

1. PHP: Make sure you have PHP installed on your local server. You can check this by running php -v in your terminal.
2. Composer: Composer is a dependency manager for PHP that helps us manage libraries easily. Install it by following the instructions on the Composer website.
3. MySQL: Set up a MySQL database to store the extracted emails. If you don’t have MySQL installed, consider using tools like XAMPP or MAMP, which bundle Apache, PHP, and MySQL together.
4. Selenium: If you plan to extract emails from JavaScript-rendered content, ensure you have Selenium WebDriver set up as discussed in our previous blog. This will allow us to automate browser actions.

Database Setup

To store the extracted email addresses, create a database and a corresponding table. Here’s how you can do this using SQL commands:

CREATE DATABASE email_extractor;
USE email_extractor;

CREATE TABLE emails (
    id INT AUTO_INCREMENT PRIMARY KEY,
    email_address VARCHAR(255) UNIQUE NOT NULL
);

This structure allows us to store unique email addresses, ensuring that duplicates are not recorded.

Building the Email Extraction Tool

1. Define the Directory Structure

Organizing your project files properly will help you manage and maintain the code efficiently. Here’s a recommended directory structure:

email_extractor/
├── composer.json
├── index.php
├── extractors/
│   ├── PdfExtractor.php
│   ├── HtmlExtractor.php
│   └── JsExtractor.php
└── db.php

2. Create Database Connection

Create a db.php file for MySQL connection to centralize database operations. Here’s a sample implementation:

<?php
$host = 'localhost'; // or your host
$username = 'your_username'; // your MySQL username
$password = 'your_password'; // your MySQL password
$dbname = 'email_extractor'; // your database name

$mysqli = new mysqli($host, $username, $password, $dbname);

if ($mysqli->connect_error) {
    die("Connection failed: " . $mysqli->connect_error);
}
?>

3. Create Extractors

In the extractors folder, we will create three classes, each responsible for extracting emails from a specific content type.

HtmlExtractor.php: Handles static HTML extraction.

<?php
class HtmlExtractor {
    public function extract($html) {
        // Regular expression to match email addresses
        preg_match_all("/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/", $html, $matches);
        return $matches[0]; // Returns an array of email addresses
    }
}
?>

This class utilizes a regular expression to find and return all email addresses in the provided HTML content.

JsExtractor.php: Handles JavaScript-rendered content extraction using Selenium.

<?php
require 'vendor/autoload.php'; // Load Composer's autoloader
use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Remote\DesiredCapabilities;

class JsExtractor {
    private $driver;

    public function __construct() {
        $host = 'http://localhost:4444'; // Selenium Server URL
        $this->driver = RemoteWebDriver::create($host, DesiredCapabilities::chrome());
    }

    public function extract($url) {
        $this->driver->get($url); // Navigate to the URL
        $this->driver->wait()->until(
            WebDriverExpectedCondition::presenceOfElementLocated(WebDriverBy::cssSelector('body')) // Wait for body to load
        );
        $html = $this->driver->getPageSource(); // Get the page source
        $this->driver->quit(); // Close the browser

        // Extract email addresses from the HTML
        preg_match_all("/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/", $html, $matches);
        return $matches[0]; // Returns an array of email addresses
    }
}
?>

In this class, we initiate a Selenium WebDriver instance, navigate to the specified URL, wait for the page to load, and then extract the HTML content for email extraction.

PdfExtractor.php: Handles PDF email extraction.

<?php
require 'vendor/autoload.php'; // Load Composer's autoloader
use Smalot\PdfParser\Parser;

class PdfExtractor {
    public function extract($filePath) {
        $parser = new Parser(); // Initialize PDF parser
        $pdf = $parser->parseFile($filePath); // Parse the PDF file
        $text = $pdf->getText(); // Extract text from the PDF

        // Regular expression to match email addresses
        preg_match_all("/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/", $text, $matches);
        return $matches[0]; // Returns an array of email addresses
    }
}
?>

This class uses the Smalot/PdfParser library to extract text from PDF files, allowing us to find email addresses within.

4. Create the Main Extraction Logic in index.php

The index.php file will serve as the main interface for user input and processing. Here’s the complete implementation:

<?php
require 'db.php'; // Include database connection
require 'extractors/HtmlExtractor.php'; // Include HTML extractor
require 'extractors/JsExtractor.php'; // Include JS extractor
require 'extractors/PdfExtractor.php'; // Include PDF extractor

if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    $url = $_POST['url'];
    $contentType = $_POST['content_type']; // Get the selected content type
    $emails = []; // Initialize an array to store extracted emails

    // Determine the appropriate extractor based on content type
    switch ($contentType) {
        case 'static_html':
            $html = file_get_contents($url); // Fetch HTML content
            $extractor = new HtmlExtractor(); // Create an instance of HtmlExtractor
            $emails = $extractor->extract($html); // Extract emails
            break;

        case 'js_rendered':
            $extractor = new JsExtractor(); // Create an instance of JsExtractor
            $emails = $extractor->extract($url); // Extract emails from JS-rendered content
            break;

        case 'pdf':
            // Assuming the PDF file is accessible via URL, we can download it first
            $tempFile = tempnam(sys_get_temp_dir(), 'pdf_'); // Create a temporary file
            file_put_contents($tempFile, file_get_contents($url)); // Download the PDF
            $extractor = new PdfExtractor(); // Create an instance of PdfExtractor
            $emails = $extractor->extract($tempFile); // Extract emails
            unlink($tempFile); // Delete the temporary file
            break;

        default:
            echo "Unsupported content type.";
            exit;
    }

    // Insert emails into database
    foreach ($emails as $email) {
        $stmt = $mysqli->prepare("INSERT IGNORE INTO emails (email_address) VALUES (?)"); // Prepare SQL statement
        $stmt->bind_param("s", $email); // Bind the email parameter
        $stmt->execute(); // Execute the statement
    }

    echo "Extracted emails: " . implode(", ", $emails); // Display extracted emails
}
?>

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Email Extractor</title>
    <style>
        body { font-family: Arial, sans-serif; }
        form { margin: 20px; }
        input, select { margin-bottom: 10px; }
    </style>
</head>
<body>
    <h1>Email Extraction Tool</h1>
    <form method="post" action="">
        <label for="url">Enter URL:</label>
        <input type="text" name="url" required>
        
        <label for="content_type">Select Content Type:</label>
        <select name="content_type">
            <option value="static_html">Static HTML</option>
            <option value="js_rendered">JavaScript Rendered</option>
            <option value="pdf">PDF</option>
        </select>
        
        <button type="submit">Extract Emails</button>
    </form>
</body>
</html>

Breakdown of index.php

• Form Handling: The form captures user input for the URL and content type. When submitted, it triggers a POST request to extract emails.
• Content Type Logic: Based on the selected content type, the appropriate extractor class is instantiated. For PDFs, we download the file temporarily to process it.
• Database Insertion: Extracted emails are inserted into the database using a prepared statement, which helps prevent SQL injection.
• User Feedback: The tool displays the extracted email addresses to the user.

Conclusion

In this blog post, we successfully built a comprehensive email extraction tool that integrates multiple techniques for extracting email addresses from various content types. By using PHP and MySQL, we created a flexible and efficient application capable of handling static HTML, JavaScript-rendered content, and PDF files seamlessly.

Introduction

In our previous blogs, we focused on building a basic email extractor using PHP and MySQL and discussed advanced techniques to enhance its functionality. In this blog, we will explore how to extract email addresses from different content types, specifically PDFs and images. This will provide you with a comprehensive understanding of how to broaden your email extraction capabilities beyond just web pages.

1. Understanding the Challenges

Before diving into the extraction process, it’s essential to understand the challenges associated with different content types:

• PDF Files: PDFs can contain text in various formats, including images, tables, and other complex layouts, making extraction tricky.
• Images: Email addresses in images require Optical Character Recognition (OCR) technology to convert the visual text into machine-readable format.

2. Extracting Emails from PDF Files

To extract email addresses from PDFs, you can use libraries like TCPDF or FPDF for PHP. However, a more straightforward approach is to use the pdfparser library or pdftotext command-line utility.

Using pdftotext Command-Line Utility

Installation: Ensure you have pdftotext installed on your server. For most Linux distributions, you can install it using:

sudo apt-get install poppler-utils

Extracting Text from PDF: Use the following PHP code to extract text from a PDF file:

function extractEmailsFromPDF($filePath) {
    $text = shell_exec("pdftotext " . escapeshellarg($filePath) . " -");
    return extractEmailsFromText($text);
}

function extractEmailsFromText($text) {
    preg_match_all('/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}/i', $text, $matches);
    return array_unique($matches[0]);
}

Storing Extracted Emails: Once you have extracted the emails, you can store them in your MySQL database using the techniques discussed in previous blogs.

3. Extracting Emails from Images

Extracting email addresses from images involves using OCR technology. One of the popular libraries for this purpose is Tesseract OCR.

Using Tesseract OCR

Installation: Install Tesseract OCR on your server:

For Linux:

sudo apt-get install tesseract-ocr

For Windows, download the installer from Tesseract at UB Mannheim.

Extracting Text from Images: Use the following PHP code to process images and extract text:

function extractEmailsFromImage($imagePath) {
    // Run Tesseract command to extract text
    $text = shell_exec("tesseract " . escapeshellarg($imagePath) . " stdout");
    return extractEmailsFromText($text);
}

Integration into Your Project: Similar to PDF extraction, you can now integrate this functionality into your email extractor. Combine it with your existing email extraction logic to handle various input formats.

4. Combining Multiple Extraction Methods

To create a robust email extractor that can handle PDFs, images, and web pages, consider the following:

• File Upload Handling: Allow users to upload multiple file types (PDFs, images) in addition to providing URLs. Use an HTML form to facilitate this.
• Dynamic Extraction Logic: Implement logic to determine the file type and call the appropriate extraction function based on the content type.

if (isset($_FILES['file'])) {
    $fileType = $_FILES['file']['type'];
    $filePath = $_FILES['file']['tmp_name'];

    if ($fileType === 'application/pdf') {
        $emails = extractEmailsFromPDF($filePath);
    } elseif (strpos($fileType, 'image/') === 0) {
        $emails = extractEmailsFromImage($filePath);
    }
    // Handle URLs as well...
}

5. Data Quality and Cleanup

Once you extract emails from different sources, it’s essential to clean up the data. Here are some steps to consider:

• Remove Duplicates: Implement checks to prevent duplicate entries across all extracted emails.
• Sanitize Emails: Ensure that the extracted emails conform to the correct format before storing them in the database.
• Log Extraction Results: Maintain a log of successful and failed extractions for better troubleshooting.

Conclusion

In this blog, we explored advanced methods for extracting emails from PDFs and images, broadening the scope of your email extraction capabilities. By integrating these techniques into your existing email extractor, you can create a versatile tool that efficiently gathers email addresses from various content types.

In the next blog, we will discuss how to implement data scraping ethically and comply with legal guidelines. Stay tuned!

Introduction

In our last blog, we built a simple email extractor using PHP and MySQL. While that project provided a foundational understanding of email extraction, there are several advanced techniques that can enhance the efficiency, accuracy, and reliability of your email extraction process. In this blog, we will explore these techniques in detail.

1. Improving Email Validation

While basic email validation checks the syntax, it’s essential to implement more robust validation. Consider using the following strategies:

• Domain Validation: Verify that the domain of the email address actually exists. You can use DNS lookup functions in PHP to check for valid MX (Mail Exchange) records.

function domainExists($domain) {
    return checkdnsrr($domain, 'MX');
}

function isValidEmail($email) {
    if (filter_var($email, FILTER_VALIDATE_EMAIL)) {
        $domain = substr(strrchr($email, "@"), 1);
        return domainExists($domain);
    }
    return false;
}

• Third-party APIs: Consider integrating third-party email validation services like Hunter.io or NeverBounce. These services provide comprehensive checks on whether the email address is deliverable.

2. Handling Rate Limiting and Timeouts

When scraping multiple websites, it’s crucial to respect the target server’s resources. Implement rate limiting to avoid being blocked:

• Sleep Between Requests: Introduce a delay between requests.

sleep(1); // Sleep for 1 second between requests

• Handle Timeouts: Use cURL options to set timeouts, preventing your script from hanging indefinitely.

curl_setopt($ch, CURLOPT_TIMEOUT, 10); // Set a timeout of 10 seconds

3. Managing Duplicate Entries

To prevent duplicate entries in your database, you can implement checks before inserting new emails. Here’s how:

• Modify the SQL Query: Use INSERT IGNORE or INSERT ... ON DUPLICATE KEY UPDATE in your SQL query.

$sql = "INSERT INTO emails (email_address, source) VALUES ('$email', '$url') ON DUPLICATE KEY UPDATE email_address=email_address";

• Check Before Inserting: Alternatively, you can check if the email already exists before inserting it:

$checkEmailQuery = "SELECT * FROM emails WHERE email_address = '$email'";
$result = $conn->query($checkEmailQuery);
if ($result->num_rows == 0) {
    $conn->query($sql);
}

4. Multi-threading for Faster Extraction

Using multi-threading can significantly speed up the extraction process, especially when dealing with multiple URLs. Libraries like cURL Multi in PHP can help achieve this.

$multiHandle = curl_multi_init();
// Add multiple cURL handles to the multi-handle
foreach ($urls as $url) {
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_multi_add_handle($multiHandle, $ch);
}

// Execute all queries simultaneously
do {
    $status = curl_multi_exec($multiHandle, $active);
    curl_multi_select($multiHandle);
} while ($active && $status == CURLM_CALL_MULTI_PERFORM);

// Fetch results
foreach ($handles as $ch) {
    $html = curl_multi_getcontent($ch);
    // Process the HTML as needed
    curl_multi_remove_handle($multiHandle, $ch);
}
curl_multi_close($multiHandle);

5. Storing Emails in a More Structured Way

Instead of just storing emails in a flat table, consider creating a more structured database design:

• Create a separate table for domains to avoid redundancy:

CREATE TABLE domains (
    id INT AUTO_INCREMENT PRIMARY KEY,
    domain_name VARCHAR(255) NOT NULL UNIQUE
);

CREATE TABLE emails (
    id INT AUTO_INCREMENT PRIMARY KEY,
    email_address VARCHAR(255) NOT NULL,
    domain_id INT,
    source VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (domain_id) REFERENCES domains(id)
);

• Normalize Data: Link the emails to their respective domains, reducing redundancy and improving query efficiency

6. Implementing Email Extraction with User-Agent Rotation

Some websites block requests that do not originate from a browser. To avoid this, you can rotate User-Agent strings for your cURL requests:

$userAgents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36",
    // Add more User-Agents
];
curl_setopt($ch, CURLOPT_USERAGENT, $userAgents[array_rand($userAgents)]);

7. Error Logging and Monitoring

Implementing error logging can help you identify issues during the scraping process. You can log errors to a file or a database:

function logError($message) {
    file_put_contents('error_log.txt', date('Y-m-d H:i:s') . " - " . $message . PHP_EOL, FILE_APPEND);
}

Conclusion

In this blog, we explored advanced techniques to enhance the email extraction process using PHP and MySQL. By implementing better validation, handling rate limits, and optimizing your database structure, you can significantly improve your email extractor’s performance and reliability.

Introduction

In our previous blogs, we covered the fundamentals of email extraction using PHP and MySQL. We explored how to scrape websites for emails and the tools and techniques involved. In this blog, we will take a practical approach and build a simple email extractor from scratch using PHP and MySQL. By the end of this guide, you’ll have a fully functional email extractor that can extract emails from websites and store them in a database.

1. Overview of the Project

We will create a PHP script that:

• Takes a URL input from the user.
• Scrapes the webpage for email addresses.
• Validates the extracted emails.
• Stores the emails in a MySQL database.

This project will provide a hands-on experience in using PHP for web scraping and data storage.

2. Setting Up the Environment

Before we start coding, ensure you have the following set up:

• Web Server: Use WAMP or XAMPP to create a local server environment.
• Database: Create a MySQL database named email_extractor with a table for storing emails.

CREATE TABLE emails (
    id INT AUTO_INCREMENT PRIMARY KEY,
    email_address VARCHAR(255) NOT NULL,
    source VARCHAR(255),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

3. Creating the PHP Script

Now, let’s create a PHP script named email_extractor.php that will perform the email extraction. We will utilize cURL for fetching webpage content and DOMDocument for parsing HTML.

<?php
// Database connection
$servername = "localhost";
$username = "root";
$password = "";
$dbname = "email_extractor";

$conn = new mysqli($servername, $username, $password, $dbname);
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

// Function to extract emails from the given URL
function extractEmails($url) {
    // Initialize cURL session
    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $html = curl_exec($ch);
    curl_close($ch);

    // Load HTML into DOMDocument
    $dom = new DOMDocument();
    @$dom->loadHTML($html);
    $emails = [];

    // Extract emails using regex
    $pattern = '/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}\b/i';
    preg_match_all($pattern, $html, $matches);
    
    foreach ($matches[0] as $email) {
        // Remove duplicates and store valid emails
        if (!in_array($email, $emails)) {
            $emails[] = $email;
        }
    }

    return $emails;
}

// Check if URL is provided
if (isset($_POST['url'])) {
    $url = $_POST['url'];
    $emails = extractEmails($url);

    // Insert extracted emails into database
    foreach ($emails as $email) {
        $sql = "INSERT INTO emails (email_address, source) VALUES ('$email', '$url')";
        $conn->query($sql);
    }

    echo "Emails extracted and stored successfully!";
}

// Close database connection
$conn->close();
?>

4. Validating Extracted Emails

It’s essential to validate the emails you extract to ensure they are legitimate. You can implement a basic email validation function in your PHP script:

function isValidEmail($email) {
    return filter_var($email, FILTER_VALIDATE_EMAIL) !== false;
}

// Modify the insertion loop to validate emails
foreach ($emails as $email) {
    if (isValidEmail($email)) {
        $sql = "INSERT INTO emails (email_address, source) VALUES ('$email', '$url')";
        $conn->query($sql);
    }
}

5. Enhancements and Features to Consider

You can enhance your email extractor by adding the following features:

• Error Handling: Implement error handling to manage exceptions and invalid URLs gracefully.
• Rate Limiting: Introduce delays between requests to avoid overwhelming target servers.
• User Interface Improvements: Make the form more user-friendly with validation messages and loading indicators.
• Email Verification: Integrate third-party APIs for verifying the existence of extracted email addresses.

Conclusion

In this blog, we built a simple email extractor using PHP and MySQL. We learned how to scrape emails from a webpage, validate them, and store them in a MySQL database. This project serves as a practical introduction to web scraping and data handling with PHP.

Introduction

In the previous blog, we covered the basics of extracting emails from text using PHP and MySQL. Now, we’ll take it a step further by learning how to scrape websites for email addresses. Web scraping is a powerful technique used to extract data from websites, and with the right approach, you can gather email addresses from web pages effectively.

In this blog, we will walk you through setting up a basic web scraper using PHP to extract emails from a given website and store them in a MySQL database.

1. What Is Web Scraping?

Web scraping involves using a program to automatically extract data from websites. It’s useful for tasks like gathering contact information, tracking prices, or collecting large datasets. Email extraction is one of the common use cases for web scraping.

2. Legal Considerations for Scraping

Before you begin, it’s important to understand the legal aspects of web scraping. Many websites have terms of service that restrict scraping. Be sure to:

• Check the Website’s Terms: Make sure you have permission to scrape.
• Respect Robots.txt: This file tells scrapers what is and isn’t allowed to be scraped on a website.
• Ethical Scraping: Avoid overwhelming a server with too many requests in a short time.

3. Tools Needed for Scraping with PHP

For scraping, you can use the following tools and libraries:

• cURL: A popular PHP library used for making HTTP requests.
• PHP DOMDocument: A library that allows you to parse HTML and XML documents.
• MySQL: To store extracted emails.

4. Setting Up Your Environment

Make sure your environment is set up with PHP, cURL, and MySQL. You can use WAMP or XAMPP as your local server.

1. Install WAMP/XAMPP.
2. Verify that cURL is enabled in your PHP configuration (php.ini).
3. Create a MySQL database email_extractor with the emails table as we did in the previous blog.

CREATE TABLE emails (
    id INT AUTO_INCREMENT PRIMARY KEY,
    email_address VARCHAR(255) NOT NULL,
    source VARCHAR(255)
);

5. Writing the Scraper Using cURL

cURL is a powerful PHP library that allows you to make HTTP requests to websites. Here’s how to use it to get the HTML of a webpage:

<?php
// URL of the website to scrape
$url = "https://example.com";

// Initialize cURL session
$ch = curl_init($url);

// Set cURL options
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// Execute cURL and store the result
$html = curl_exec($ch);

// Close cURL session
curl_close($ch);

// Display the HTML content
echo $html;
?>

6. Parsing the HTML with PHP DOMDocument

Once you have the HTML content of the page, you need to extract the email addresses from it. PHP’s DOMDocument can be used to parse the HTML.

<?php
// Load HTML content into DOMDocument
$dom = new DOMDocument();
@$dom->loadHTML($html);

// Get all anchor tags
$links = $dom->getElementsByTagName('a');

foreach ($links as $link) {
    // Check if the link contains 'mailto:' (common for email links)
    $href = $link->getAttribute('href');
    if (strpos($href, 'mailto:') !== false) {
        $email = str_replace('mailto:', '', $href);
        echo "Found email: $email\n";
    }
}
?>

This script will find any mailto: links in the page and extract the email addresses.

7. Extracting Emails Using Regex

Not all email addresses are in mailto: links. Some are embedded in the text, so we can use regular expressions to capture them.

<?php
$pattern = '/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}\b/i';
preg_match_all($pattern, $html, $matches);

foreach ($matches[0] as $email) {
    echo "Found email: $email\n";
}
?>

This will scan the entire HTML content for any email patterns and display them.

8. Storing Extracted Emails in MySQL

Once you’ve extracted the emails, you can store them in the MySQL database as shown in the previous blog. Here’s a quick recap of how to insert the emails into the emails table.

<?php
$servername = "localhost";
$username = "root";
$password = "";
$dbname = "email_extractor";

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);

// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

foreach ($matches[0] as $email) {
    $sql = "INSERT INTO emails (email_address, source) VALUES ('$email', '$url')";
    $conn->query($sql);
}

$conn->close();
?>

This will save all the found email addresses into the emails table with the source URL.

9. Handling Multiple Pages

Many websites have multiple pages of content. To scrape them all, you can loop through the pages by dynamically modifying the URL or following pagination links.

Example:

for ($i = 1; $i <= 5; $i++) {
    $url = "https://example.com/page=$i";
    // Fetch and process each page
}

Conclusion

In this blog, we covered the process of scraping websites for email extraction using PHP. We learned how to:

• Use cURL to fetch webpage HTML.
• Parse HTML using PHP DOMDocument.
• Extract emails using mailto: links and regex.
• Store the results in a MySQL database.

Introduction

Email extraction is an essential technique in data collection, allowing businesses to gather email addresses for various purposes like marketing, lead generation, and customer outreach. In this blog, we will introduce the basics of extracting emails using PHP and MySQL, and walk through a simple example to get you started.

1. Importance of Email Extraction

Email extraction helps businesses build contact lists, target potential customers, and analyze communication patterns. It is especially useful for:

• Marketing: Gathering emails for email marketing campaigns.
• Lead Generation: Extracting emails from websites or documents to create leads.
• Customer Analysis: Storing customer emails for future reference or outreach.

2. Setting Up PHP and MySQL Environment

To start with email extraction, you need to set up your local PHP environment. You can use either WAMP for Windows or XAMPP for Mac. Here’s how to get started:

• Install WAMP/XAMPP.
• Ensure PHP and MySQL are configured correctly.
• Create a database in MySQL for storing extracted email addresses.

For this series, we will create a database email_extractor and a table emails to store the extracted data.

CREATE DATABASE email_extractor;

CREATE TABLE emails (
    id INT AUTO_INCREMENT PRIMARY KEY,
    email_address VARCHAR(255) NOT NULL,
    source VARCHAR(255)
);

3. Writing a Simple PHP Email Extractor

PHP makes it easy to extract emails from any text using regular expressions. A common function used for this is preg_match_all(), which can scan a block of text for email patterns.

4. Regular Expression for Email Extraction

The heart of email extraction lies in using the correct regular expression (regex) to match email patterns. A simple regex for matching emails looks like this:

$pattern = '/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}\b/i';

This regex matches most standard email formats and is case-insensitive.

5. Example Code for Extracting Emails

Here is a basic PHP script to extract emails from a given text:

<?php
$text = "Contact [email protected] or [email protected]";
$pattern = '/[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,4}\b/i';
preg_match_all($pattern, $text, $matches);
print_r($matches[0]);
?>

This script will output the found emails in the text.

6. Storing Emails in MySQL

After extracting emails, it’s important to store them in a MySQL database for future use. You can connect PHP to your MySQL database and insert the extracted emails into a table.

7. Inserting Extracted Emails into the Database

Here’s how you can modify the above script to store the emails in a MySQL database:

<?php
$servername = "localhost";
$username = "root";
$password = "";
$dbname = "email_extractor";

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);

// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

foreach ($matches[0] as $email) {
    $sql = "INSERT INTO emails (email_address, source) VALUES ('$email', 'sample text')";
    $conn->query($sql);
}

$conn->close();
?>

This will save each extracted email into the emails table.

8. Understanding How the Code Works

• preg_match_all(): This PHP function searches the input text for matches to the email pattern defined by the regex.
• MySQL Insertion: After matching emails, the script inserts each one into the MySQL database along with its source.

9. Testing and Verifying Your Script

Once the script is running, you can test it by inputting text that contains email addresses. After running the script, you should check your MySQL database to ensure that the email addresses have been correctly inserted.

been correctly inserted.

Conclusion

In this first blog, we’ve introduced the basic concepts of email extraction using PHP and MySQL. We set up the environment, wrote a simple script to extract emails using regex, and stored the results in a MySQL database. This serves as a foundation for more advanced techniques we’ll explore in future blogs.