Selenium Scraping With Node.js

Jan Wiśniewski 04 Oct 2024 15 min read

Let’s learn how to harness the power of Selenium and Node.js to automate web scraping, navigating through dynamic content! We'll guide you through every step – from setting up your environment to handling errors and extracting valuable data. Whether you're gathering insights or scaling your business operations, this comprehensive  tutorial will show you how to maximize efficiency of Selenium scraping with Node.js – and with Infatica proxies for seamless scraping performance!

Required Tools

1. Node.js is a JavaScript runtime built on Chrome's V8 engine, allowing you to run JavaScript code, which is a popular programming language, server-side. It's essential for executing your scraping scripts and handling asynchronous operations effectively. Its non-blocking I/O model makes it ideal for network-heavy tasks like Selenium scraping with Node.js.

2. npm: (Node Package Manager) is a free tool that acts as the default package manager for Node.js. It allows you to easily install and manage third-party libraries and dependencies required for your scraping project. For instance, you can use npm for installing Selenium, libraries for handling CSV files, and other useful packages that enhance your scraping capabilities.

3. Selenium WebDriver is a powerful tool for browser automation. It provides APIs for interacting with web pages programmatically, enabling you to navigate, extract data, and simulate user interactions (like clicking buttons or filling out forms). Using Selenium is crucial for scraping dynamic content that relies on JavaScript, making it an integral part of your scraping toolkit.

4. ChromeDriver is a separate executable that Selenium WebDriver uses to control the Chrome browser. It acts as a bridge between your scraping scripts and the Chrome browser, allowing your Node.js application to send commands to Chrome for tasks like navigating to URLs, retrieving elements, and executing JavaScript. Having ChromeDriver set up is necessary to run Selenium tests in a Chrome environment.

5. IDE/Text Editor are essential for writing and managing your scraping code. Tools like Visual Studio Code, Atom, or Sublime Text offer features like syntax highlighting, code completion, and debugging, which enhance your coding efficiency. An IDE helps organize your files and provides a comfortable environment to develop, test, and run your scripts.

6. CSV/Excel Reader: Once you extract public data from websites, you may want to analyze or manipulate it. A CSV/Excel reader allows you to easily work with required data in a tabular format. Libraries like papaparse or fast-csv in Node.js make it simple to parse and generate CSV files, while Excel readers help you visualize and analyze following data in spreadsheet applications. These tools are vital for managing the output of your scraping tasks effectively.

Prerequisites

Step 1: Set Up Your Project

1. Initialize a new Node.js project: Start by creating the project directory and initializing a new Node.js project using the following commands in your terminal:

mkdir selenium-scraping
cd selenium-scraping
npm init -y

This creates a new folder, changes the directory to it, and initializes a package.json file with default settings (-y skips the setup prompts).

Step 2: Install Dependencies

1. Install necessary dependencies: In the terminal, run the following command – this will install the Selenium-webdriver package, which is the main library you’ll use to interact with Selenium.

npm install selenium-webdriver

2. Install additional utilities: Depending on your scraping task, you may want to install other helpful packages. For example:

• Axios or node-fetch: For making HTTP requests.
• Cheerio: For parsing HTML code in case you want to extract specific data from the page after Selenium loads it.

Install these packages if needed, which set up the basic dependencies required to run Selenium and optionally include web scraping tools that can also handle data from web pages.

npm install axios cheerio

Step 3: Set Up Selenium

1. Download the WebDriver for your browser: Selenium requires a browser-specific WebDriver to control the browser. Depending on which browser you plan to use, download the relevant WebDriver – and ensure the WebDriver is compatible with the version of the browser you're using.

2. Add WebDriver to system PATH: After downloading the WebDriver, add its location to your system’s PATH so Selenium can easily find and use it. For most operating systems:

• Windows: Edit the system environment variables to include the path to the WebDriver executable.
• macOS/Linux: Add the WebDriver’s location to your .bashrc or .zshrc file:

export PATH=$PATH:/path/to/webdriver

3. Verify the WebDriver setup: To ensure everything is working, run a simple test. Create a test.js file in your project folder and add the following code that contains this basic script:

const { Builder } = require('selenium-webdriver');

(async function test() {
    let driver = await new Builder().forBrowser('chrome').build();
    await driver.get('https://www.google.com');
    console.log('Selenium is working!');
    await driver.quit();
})();

Run it with:

node test.js

If the browser opens and navigates to Google, you’ve successfully set up Selenium.

Step 4: Use Selenium to Launch a Browser

2. Create a simple script to launch a browser: You’ll need to use a new builder class from the Selenium-webdriver package to specify which browser to launch. Here’s an example script:

const { Builder } = require('selenium-webdriver');

(async function launchBrowser() {
    // Set up the driver for Chrome (you can replace 'chrome' with 'firefox', 'edge', etc.)
    let driver = await new Builder().forBrowser('chrome').build();

    try {
        // Navigate to a website (e.g., Google)
        await driver.get('https://www.google.com');

        // You can add additional actions here, such as finding elements or interacting with the page
    } finally {
        // Close the browser after finishing your tasks
        await driver.quit();
    }
})();

Here’s how this code works:

• Builder(): The Builder() class allows you to configure which browser to launch (chrome, firefox, edge, etc.).
• .forBrowser('chrome'): Specifies which browser to use. You can change 'chrome' to 'firefox', 'edge', or other supported browsers.
• .get(): The get() method opens a URL in the browser.
• .quit(): Closes the browser when the task is complete.

4. Run the script: Save the file as launch.js and run it in your terminal. This will open the specified browser, navigate to the chosen URL, and then close the browser after performing any actions you define.

node launch.js

Web Scraping with Selenium and Node.js

Step 1: Define the URL to Scrape

Step 1: Define the target URL: In web scraping, the first step is to specify the URL of the website or webpage you want to web-scrape. This URL is typically the target page where the data you need resides. Here’s a simple code snippet to define the URL in a Node.js script:

const url = 'https://www.example.com/products';

// Log the URL to ensure it's defined correctly
console.log(`Scraping data from: ${url}`);

Here’s how this code works:

• const url: The URL of the target page you want to scrape. In this case, it points to a sample product details page, https://www.example.com/products.
• console.log(): Logs the URL to confirm that the correct page is being targeted before you proceed to the next steps, like making an HTTP request or interacting with the page.

Step 2: Making an HTTP Request

Once you’ve defined the URL to scrape, the next step is to make an HTTP request to that URL to retrieve the web page's HTML content. In Selenium, the browser itself loads the page, so an additional HTTP request isn’t necessary, but sometimes using an HTTP request is more efficient if you don’t need to interact with the page.

For Node.js, you can use libraries like axios or node-fetch to make an HTTP request. First, install axios if you haven’t already:

npm install axios

Then, use this code snippet to make the request:

const axios = require('axios');

// Define the URL to scrape
const url = 'https://www.example.com/products';

// Make an HTTP request to the defined URL
async function fetchPage() {
    try {
        const response = await axios.get(url);
        const html = response.data;
        
        console.log('Page fetched successfully!');
        // Output part of the HTML response
        console.log(html.substring(0, 500)); // print the first 500 characters
    } catch (error) {
        console.error(`Error fetching the page: ${error}`);
    }
}

// Run the function
fetchPage();

Here’s how this code works:

• axios.get(url): Makes an HTTP `GET` request to the specified URL.
• response.data: Contains the HTML document with the page’s contents.
• html.substring(0, 500): Outputs the first 500 characters of the HTML content for verification. This is just to ensure the page is loaded properly.
• Error handling: Catches and logs any errors during the request.

Step 3: Parse the HTML Response

Once you’ve made the HTTP request and retrieved the HTML content of the webpage, the next step is to parse the HTML so that you can extract the data you need. In Node.js, the cheerio library is commonly used for parsing and manipulating HTML because it provides a jQuery-like interface.

Here’s how to parse the HTML response:

const axios = require('axios');
const cheerio = require('cheerio');

// Define the URL to scrape
const url = 'https://www.example.com/products';

// Fetch the page and parse the HTML
async function fetchAndParsePage() {
    try {
        const response = await axios.get(url);
        const html = response.data;

        // Load the HTML into cheerio for parsing
        const $ = cheerio.load(html);

        console.log('HTML parsed successfully!');
        
        // Example: select and log the title of the page
        const pageTitle = $('title').text();
        console.log(`Page title: ${pageTitle}`);
    } catch (error) {
        console.error(`Error fetching or parsing the page: ${error}`);
    }
}

// Run the function
fetchAndParsePage();

Here’s how this code works:

• cheerio.load(html): Loads the HTML into Cheerio, which gives you a jQuery-like interface to select and manipulate elements.
• $('title').text(): This selects the <title> element from the page and retrieves the text data (the title of the page).

Step 4: Wait for Page Elements to Load

When scraping dynamic websites that use JavaScript to load content after the page initially loads (e.g., using AJAX), you’ll need to wait for certain elements to appear before extracting data. Selenium is helpful here since it can control the browser and wait for elements to be fully loaded.

In Selenium with Node.js, you can use WebDriver’s built-in wait functions, such as until.elementLocated(), to pause the execution until an element appears on the page.

const { Builder, By, until } = require('selenium-webdriver');

// Define the URL to scrape
const url = 'https://www.example.com/products';

(async function waitForElements() {
    // Set up the driver for Chrome (or another browser)
    let driver = await new Builder().forBrowser('chrome').build();

    try {
        // Open the URL
        await driver.get(url);

        console.log('Page loaded! Waiting for elements to load...');

        // Wait for a specific element to load (e.g., a product list)
        const element = await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

        console.log('Elements loaded successfully! Now ready for data extraction.');

        // You can now proceed with data extraction...
    } catch (error) {
        console.error(`Error waiting for elements: ${error}`);
    } finally {
        // Close the browser
        await driver.quit();
    }
})();

Here’s how this code works:

• until.elementLocated(By.css('.product-list')): Waits until an element matching the CSS selector `.product-list` is located on the page. You can replace the CSS selector with any specific element you need to wait for.
• Timeout (10000): Specifies a 10-second timeout. If the element doesn’t appear within this time, an error is thrown.
• await driver.get(url): Loads the page, and the script waits for the element before continuing.

This ensures that the page is fully loaded and the required elements are ready before moving forward.

Step 5: Select Data

After ensuring that the required page elements have loaded, the next step is to select the data you want to scrape. In Selenium, you can use selectors like By.id(), By.className(), or By.css() to target specific elements on the page.

const { Builder, By, until } = require('selenium-webdriver');

// Define the URL to scrape
const url = 'https://www.example.com/products';

(async function selectData() {
    let driver = await new Builder().forBrowser('chrome').build();

    try {
        // Open the URL
        await driver.get(url);

        // Wait for the product list to load
        await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

        // Select all product names from the page using a CSS selector
        let productElements = await driver.findElements(By.css('.product-name'));

        console.log('Products found:');
        for (let productElement of productElements) {
            let productName = await productElement.getText();
            console.log(productName);
        }
    } catch (error) {
        console.error(`Error selecting data: ${error}`);
    } finally {
        await driver.quit();
    }
})();

Here’s how this code works:

• findElements(By.css('.product-name')): Selects all elements with the class `.product-name`. This method returns an array of WebElement objects.
• getText(): Retrieves the text content of each element. In this case, it will extract the product names.
• for...of loop: Iterates over each product element and logs the name to the console.

This allows you to select and log the desired data from the webpage, which you can later use in further steps like data extraction.

Step 6: Extracting Data

Once you've selected the elements you want to scrape, the next step is to extract the data from those elements. In most cases, you’ll extract text or attributes like `href` from links, src from images, etc. Selenium provides build methods like getText() and getAttribute() to help with this.

const { Builder, By, until } = require('selenium-webdriver');

// Define the URL to scrape
const url = 'https://www.example.com/products';

(async function extractData() {
    let driver = await new Builder().forBrowser('chrome').build();

    try {
        // Open the URL
        await driver.get(url);

        // Wait for the product list to load
        await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

        // Select all product names and extract their text
        let productElements = await driver.findElements(By.css('.product-name'));
        
        console.log('Extracted product data:');
        for (let productElement of productElements) {
            let productName = await productElement.getText();
            console.log(`Product: ${productName}`);
        }

        // Example of extracting additional attributes (e.g., prices)
        let priceElements = await driver.findElements(By.css('.product-price'));
        for (let priceElement of priceElements) {
            let price = await priceElement.getText();
            console.log(`Price: ${price}`);
        }
    } catch (error) {
        console.error(`Error extracting data: ${error}`);
    } finally {
        await driver.quit();
    }
})();

Here’s how this code works:

• getText(): Extracts the text content of each selected element (e.g., product names and prices).
• findElements(By.css('.product-price')): You can repeat the selection and extraction process for other elements, such as prices, images, or links.

Step 6.1: Select Elements Using CSS Selectors

CSS selectors are essential when selecting elements from a webpage. You can target elements by their tag name, class, ID, class attributes, or hierarchy. Selenium supports CSS selectors, and you can use them to efficiently pinpoint the data you need. Here are some common CSS selector examples:

Tag name: Selects all elements by their tag name.

let elements = await driver.findElements(By.css('h1'));

Class name: Selects all elements with a specific class.

let elements = await driver.findElements(By.css('.product-name'));

ID: Selects a unique element by its ID.

let element = await driver.findElement(By.css('#main-header'));

Attributes: Selects elements based on their attributes.

let elements = await driver.findElements(By.css('img[src*="product-image"]'));

And here’s an example code snippet that uses CSS Selectors:

// Select elements using CSS selectors
let productNameElements = await driver.findElements(By.css('.product-name'));
let headerElement = await driver.findElement(By.css('#main-header'));
let imageElements = await driver.findElements(By.css('img[src*="product-image"]'));

Step 7: Save the Extracted Data to a CSV File

After extracting the data from the webpage, you can save it into a CSV file for easy analysis or use. In Node.js, you can use the built-in fs (File System) module to write data to a file and the papaparse or fast-csv library to format the data as CSV.

First, install the papaparse library to convert your data to CSV format:

npm install papaparse

Then, here’s how you can extract the data and save it to a CSV file:

const { Builder, By, until } = require('selenium-webdriver');
const fs = require('fs');
const Papa = require('papaparse');

// Define the URL to scrape
const url = 'https://www.example.com/products';

(async function saveDataToCSV() {
    let driver = await new Builder().forBrowser('chrome').build();

    try {
        // Open the URL
        await driver.get(url);

        // Wait for the product list to load
        await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

        // Extract product names and prices
        let products = [];
        let productElements = await driver.findElements(By.css('.product-name'));
        let priceElements = await driver.findElements(By.css('.product-price'));

        for (let i = 0; i < productElements.length; i++) {
            let productName = await productElements[i].getText();
            let productPrice = await priceElements[i].getText();
            products.push({ name: productName, price: productPrice });
        }

        // Convert the data to CSV format
        const csv = Papa.unparse(products);

        // Save the CSV to a file
        fs.writeFileSync('products.csv', csv);
        console.log('Data saved to products.csv');
    } catch (error) {
        console.error(`Error saving data to CSV: ${error}`);
    } finally {
        await driver.quit();
    }
})();

Here’s how this code works:

• products[]: An array to hold the product data, where each item is an object containing a `name` and price.
• Papa.unparse(): Converts the products array into CSV format.
• fs.writeFileSync(): Writes the CSV data to a file named products.csv.
• products.csv: The generated CSV file will have two columns: name and `price`, listing all products scraped from the webpage.

This will save your extracted data into a CSV file that can be opened in tools like Excel, Google Sheets, or any CSV viewer.

Step 8: Handle Pagination (Optional)

Many websites display data across multiple pages, such as product listings. To scrape data, you need to handle pagination, navigating to the next page and continuing the scraping process. In Selenium, you can click on the "Next" button (or similar) to load the next page, and then continue scraping.

const { Builder, By, until } = require('selenium-webdriver');
const fs = require('fs');
const Papa = require('papaparse');

// Define the base URL for the scraping
const baseURL = 'https://www.example.com/products?page=';

(async function scrapeWithPagination() {
    let driver = await new Builder().forBrowser('chrome').build();
    let products = [];
    let page = 1;

    try {
        while (true) {
            // Load the page
            await driver.get(`${baseURL}${page}`);
            console.log(`Scraping page ${page}...`);

            // Wait for the product list to load
            await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

            // Extract product names and prices
            let productElements = await driver.findElements(By.css('.product-name'));
            let priceElements = await driver.findElements(By.css('.product-price'));

            if (productElements.length === 0) {
                console.log('No more products found. Exiting...');
                break; // Exit the loop if no more products are found
            }

            for (let i = 0; i < productElements.length; i++) {
                let productName = await productElements[i].getText();
                let productPrice = await priceElements[i].getText();
                products.push({ name: productName, price: productPrice });
            }

            // Check if a "Next" button exists for the next page
            let nextButton = await driver.findElements(By.css('.pagination-next'));
            if (nextButton.length === 0) {
                console.log('No more pages to scrape. Exiting...');
                break; // Exit the loop if no "Next" button is found
            }

            page++; // Move to the next page
        }

        // Convert the data to CSV format
        const csv = Papa.unparse(products);

        // Save the CSV to a file
        fs.writeFileSync('products.csv', csv);
        console.log('Data saved to products.csv');
    } catch (error) {
        console.error(`Error handling pagination: ${error}`);
    } finally {
        await driver.quit();
    }
})();

Here’s how this code works:

1. Pagination URL (baseURL + page): The script dynamically updates the URL to load each page of the product listing.
2. Looping through pages: The while (true) loop continues until no more products or a "Next" button is found. It increments the page variable to load the next page.
3. Check for the "Next" button: The findElements(By.css('.pagination-next')) checks for the presence of a "Next" button to determine if there are more pages to scrape. If the button is missing, the loop breaks.
4. Scraping data from multiple pages: Each page is scraped in the same way as before, and the product data is collected into the products array.
5. Exit conditions: The loop breaks when there are no more products or no more pages.

Step 9: Handle Errors and Rate Limiting

Web scraping can sometimes run into issues like server errors, missing elements, or rate limiting (when a website blocks or throttles requests due to high traffic). To handle these problems, you need to implement error handling, retries, and delays between requests.

const { Builder, By, until } = require('selenium-webdriver');
const fs = require('fs');
const Papa = require('papaparse');

// Define the URL for scraping
const url = 'https://www.example.com/products';
const MAX_RETRIES = 3; // Maximum number of retries in case of failure
const DELAY_BETWEEN_REQUESTS = 5000; // Delay in milliseconds (5 seconds)

async function delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

async function scrapeDataWithRetries(driver, retries = 0) {
    try {
        // Open the URL
        await driver.get(url);
        
        // Wait for the product list to load
        await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

        // Extract product names and prices
        let products = [];
        let productElements = await driver.findElements(By.css('.product-name'));
        let priceElements = await driver.findElements(By.css('.product-price'));

        for (let i = 0; i < productElements.length; i++) {
            let productName = await productElements[i].getText();
            let productPrice = await priceElements[i].getText();
            products.push({ name: productName, price: productPrice });
        }

        // Convert the data to CSV format
        const csv = Papa.unparse(products);

        // Save the CSV to a file
        fs.writeFileSync('products.csv', csv);
        console.log('Data saved to products.csv');
    } catch (error) {
        console.error(`Error scraping data: ${error.message}`);
        
        // Retry logic for errors
        if (retries < MAX_RETRIES) {
            console.log(`Retrying... (Attempt ${retries + 1})`);
            await delay(DELAY_BETWEEN_REQUESTS); // Wait before retrying
            await scrapeDataWithRetries(driver, retries + 1);
        } else {
            console.error('Max retries reached. Exiting...');
        }
    }
}

(async function scrapeWithErrorHandling() {
    let driver = await new Builder().forBrowser('chrome').build();

    try {
        await scrapeDataWithRetries(driver);
    } catch (error) {
        console.error(`Critical error: ${error.message}`);
    } finally {
        await driver.quit();
    }
})();

Here’s how this code works:

1. Error handling (try-catch): The try block contains the main scraping logic, while the catch block handles any errors that occur, such as network timeouts or missing elements.
2. Retries (MAX_RETRIES): The script attempts to scrape the data up to three times if an error occurs. The scrapeDataWithRetries() function is recursive, and it keeps retrying until the maximum number of retries is reached.
3. Delay between retries (delay(ms)): Implements a delay between retries to avoid overloading the server and triggering rate limiting.
4. Rate limiting: Many websites impose rate limits to prevent excessive scraping. By introducing delays (e.g., 5 seconds between retries), you can avoid triggering these limits.

How Can I Avoid Being Blocked While Scraping a Website?

Proxies are a powerful tool for bypassing scraping-related challenges. Let’s take a closer look at Infatica proxies’ benefits – and setting up your Selenium scraper to work seamlessly with Infatica proxies:

1. Bypassing IP blocks: Many websites limit the number of requests from a single IP address to prevent scraping. With Infatica's rotating proxies, you can change IP addresses to avoid being blocked or flagged.

2. Improved anonymity and privacy: Infatica’s proxies offer high levels of anonymity by masking your real IP address. This makes it harder for websites to trace your activity back to a single source, enhancing privacy.

3. Global reach: Infatica provides proxies from multiple countries and regions. This enables you to scrape geo-restricted content by using IPs from the desired location.

4. Fast and reliable connections: Infatica proxies ensure high-speed connections, which are crucial for scraping large amounts of data without delays.

5. Bypassing CAPTCHA and other anti-bot measures: By rotating IP addresses and using residential proxies, you can reduce the chances of triggering CAPTCHA or anti-bot mechanisms, ensuring smoother scraping sessions.

How to Add Infatica Proxies to Your Web Scraping Pipeline

To use Infatica proxies with Selenium, you'll need to configure the Selenium WebDriver to route requests through the proxy.

const { Builder, By, until } = require('selenium-webdriver');
const chrome = require('selenium-webdriver/chrome');

// Infatica proxy credentials
const PROXY_HOST = 'proxy.infatica.io';
const PROXY_PORT = '8888';
const PROXY_USERNAME = 'your-username';
const PROXY_PASSWORD = 'your-password';

(async function scrapeWithProxy() {
    // Set up Chrome options to use a proxy
    const chromeOptions = new chrome.Options();
    chromeOptions.addArguments(`--proxy-server=http://${PROXY_USERNAME}:${PROXY_PASSWORD}@${PROXY_HOST}:${PROXY_PORT}`);

    let driver = await new Builder()
        .forBrowser('chrome')
        .setChromeOptions(chromeOptions)
        .build();

    try {
        // Open the target URL
        await driver.get('https://www.example.com/products');
        
        // Wait for the product list to load
        await driver.wait(until.elementLocated(By.css('.product-list')), 10000);

        // Scraping logic here (e.g., extracting product names and prices)
        let productElements = await driver.findElements(By.css('.product-name'));
        for (let productElement of productElements) {
            let productName = await productElement.getText();
            console.log(`Product: ${productName}`);
        }
    } catch (error) {
        console.error(`Error using proxy: ${error}`);
    } finally {
        await driver.quit();
    }
})();

Here’s how this code works:

• chromeOptions.addArguments(): Configures the WebDriver to use a proxy. The proxy server’s credentials (username and password) are passed directly in the URL format.
• http://${PROXY_USERNAME}:${PROXY_PASSWORD}@${PROXY_HOST}:${PROXY_PORT}: This string sets the proxy server’s address, along with your Infatica account credentials.
• Global IP rotation: Infatica proxies will rotate IP addresses automatically as needed, allowing you to scrape multiple pages while avoiding detection.