Selenium Web Scraping with Python: A Step-by-Step Guide (2026)

Jan Wiśniewski 17 Jul 2026 15 min read

Selenium is one of the most established ways to scrape dynamic websites with Python, but it is not a universal default. In this guide, you’ll build a Selenium scraper step by step, from setup and element selection to waits, interactions, and pagination. Then, you'll learn when a lighter tool or managed API is the better choice.

What Is Selenium and When Should You Use It for Scraping?

Selenium is a browser-automation tool that controls a real browser through code: clicking, scrolling, filling forms, and running JavaScript exactly like a human user. For web scraping, that means it can reach data that only appears after a page's JavaScript runs, which simpler tools miss.

Selenium earns its weight when a site renders content with JavaScript, requires interaction (logins, clicks, infinite scroll), or needs a real browser to behave normally. It's the wrong tool when the data is already in the page's HTML (requests + BeautifulSoup is far faster) or when you're starting a fresh JS-heavy project, where Playwright is usually the better modern choice. We'll come back to those alternatives at the end; first, let's scrape with Selenium properly.

Step 1: Setting Up Selenium with Python

Goal: Install Selenium, launch a headless Chrome browser, and confirm that your script can load our running example site.

Selenium controls a real browser, so you need two things: the Python package and a compatible browser driver. With Selenium 4.6 or later, Selenium Manager resolves the driver automatically in most cases: there is no need to download ChromeDriver manually.

Install Selenium:

python -m pip install selenium

Now create a file named step_01_setup.py and run it with python step_01_setup.py:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

START_URL = "https://quotes.toscrape.com/"


def build_driver() -> webdriver.Chrome:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,1200")

    return webdriver.Chrome(options=options)


driver = build_driver()

try:
    driver.get(START_URL)
    print(driver.title)
finally:
    driver.quit()

How it works

Options() configures Chrome before Selenium launches it. The --headless=new flag runs the browser without opening a visible window, while --window-size gives it a predictable desktop viewport.

webdriver.Chrome() starts Chrome and lets Selenium Manager find or download the matching driver if needed. driver.get() then opens the Quotes to Scrape sandbox, which we will use throughout this tutorial. Finally, driver.quit() closes the browser cleanly, even if the script fails midway through.

Watch out for

Do not make webdriver-manager a required installation step. It was useful before Selenium Manager, but current Selenium versions handle driver resolution by default. It can still be useful for older Selenium versions, strict version pinning, or custom caching setups.

If the first run fails on a work device, the usual cause is restricted network access: Selenium Manager may need to download a browser driver before it can launch Chrome. Also, remove --headless=new temporarily while debugging: you will be able to see what the browser is doing.

Result

The script should print:

Quotes to Scrape

You now have a reusable build_driver() function. The following steps will keep the same setup and gradually turn it into a scraper that extracts, waits for, interacts with, and paginates through quote data.

Step 2: Your First Scrape

Goal: Load the Quotes to Scrape homepage, extract the first quote and its author, and print both values.

With the browser setup working, you can start reading data from the page. We will begin with one quote card, then expand the same scraper to collect multiple records in the next step.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

START_URL = "https://quotes.toscrape.com/"


def build_driver() -> webdriver.Chrome:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,1200")

    return webdriver.Chrome(options=options)


driver = build_driver()

try:
    driver.get(START_URL)

    first_quote = driver.find_element(By.CSS_SELECTOR, ".quote")
    quote_text = first_quote.find_element(By.CSS_SELECTOR, ".text").text
    author = first_quote.find_element(By.CSS_SELECTOR, ".author").text

    print(f"{quote_text}\n— {author}")
finally:
    driver.quit()

How it works

driver.get() loads the page in the browser, including any JavaScript it needs to render. find_element() returns the first matching element: here, the first container with the quote class.

Rather than searching the entire page for the text and author separately, the script searches inside that one quote card. This keeps related data together and makes the selector less likely to pick up the wrong author elsewhere on the page.

Watch out for

find_element() raises a NoSuchElementException when Selenium cannot find a match. This static page should load quickly, but JavaScript-heavy pages often need an explicit wait before you look for an element (covered in Step 4).

Also, avoid selectors based on a page element’s position, such as div:nth-child(3). A selector based on a meaningful class like .quote is usually more resilient when the page layout changes.

Result

The script should print the first quote on the page:

“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”
— Albert Einstein

You have now extracted one complete record. Next, we will use the same quote-card structure to locate multiple elements and collect more than one quote at a time.

Step 3: Locating Elements with By.ID, CSS, and XPath

Goal: Use Selenium’s three most common locator strategies to find a unique form field, repeated quote cards, and a specific nested element.

Selenium needs a reliable way to identify the elements you want to read or interact with. The best choice depends on the page: an ID is ideal for a unique element, CSS selectors work well for repeated page structures, and XPath helps when you need to describe an element’s relationship to another one.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

START_URL = "https://quotes.toscrape.com/"
LOGIN_URL = "https://quotes.toscrape.com/login"


def build_driver() -> webdriver.Chrome:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,1200")

    return webdriver.Chrome(options=options)


driver = build_driver()

try:
    # CSS selector: find every quote card on the homepage.
    driver.get(START_URL)

    quote_cards = driver.find_elements(By.CSS_SELECTOR, ".quote")
    quotes = [
        {
            "text": card.find_element(By.CSS_SELECTOR, ".text").text,
            "author": card.find_element(By.CSS_SELECTOR, ".author").text,
        }
        for card in quote_cards
    ]

    print(f"Found {len(quotes)} quote cards")
    print(quotes[0])

    # XPath: select the author inside the first quote card.
    first_author = driver.find_element(
        By.XPATH,
        "(//div[contains(@class, 'quote')]//small[contains(@class, 'author')])[1]",
    )
    print(f"First author selected with XPath: {first_author.text}")

    # ID: find one unique form field on the site's demo login page.
    driver.get(LOGIN_URL)

    username_field = driver.find_element(By.ID, "username")
    print(f"Username field ID: {username_field.get_attribute('id')}")
finally:
    driver.quit()

How it works

find_elements() returns every matching element, so it is the right choice for the repeated .quote cards. The list comprehension then extracts the text and author from each card, keeping every quote as one record.

The XPath expression selects the author nested inside the first quote card. Finally, By.ID finds the unique username field on the sandbox’s login page, which is a good example of using an ID where the page provides one.

Choosing a locator

Watch out for

Prefer an ID when a meaningful, stable one exists. For scraped data, CSS selectors are usually the clearest default because they mirror the page’s HTML structure without tying your code to a particular position.

XPath is powerful, but long, absolute paths such as /html/body/div[2]/div[1]/… are fragile: a small layout change can break them. Keep XPath expressions short and based on meaningful attributes or relationships.

Result

The script should print output similar to this:

Found 10 quote cards
{'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”', 'author': 'Albert Einstein'}
First author selected with XPath: Albert Einstein
Username field ID: username

You now have a list of quote records rather than one isolated value. In the next step, we will make that same extractor reliable on a page where the quote cards do not appear immediately.

Step 4: Handling Dynamic Content with Waits

Goal: Wait for JavaScript-generated quote cards to appear before extracting them.

On dynamic websites, Selenium can finish loading the initial HTML before the data you need has appeared in the page. The Quotes to Scrape sandbox includes a deliberately delayed JavaScript version of the same quote list, making it a useful way to see why waits matter.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

DELAYED_URL = "https://quotes.toscrape.com/js-delayed/?delay=10000"


def build_driver() -> webdriver.Chrome:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,1200")

    return webdriver.Chrome(options=options)


def extract_quotes(driver: webdriver.Chrome) -> list[dict[str, str]]:
    quote_cards = driver.find_elements(By.CSS_SELECTOR, ".quote")

    return [
        {
            "text": card.find_element(By.CSS_SELECTOR, ".text").text,
            "author": card.find_element(By.CSS_SELECTOR, ".author").text,
        }
        for card in quote_cards
    ]


driver = build_driver()

try:
    driver.get(DELAYED_URL)

    wait = WebDriverWait(driver, 15)
    wait.until(
        EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, ".quote")
        )
    )

    quotes = extract_quotes(driver)

    print(f"Found {len(quotes)} delayed quotes")
    print(f"First author: {quotes[0]['author']}")
finally:
    driver.quit()

How it works

WebDriverWait(driver, 15) gives Selenium up to 15 seconds to meet a condition. presence_of_all_elements_located() repeatedly checks whether quote cards exist in the page’s DOM and continues as soon as they do.

Only after that condition succeeds does the script call the extract_quotes() function from Step 3. The scraper now works with the same quote-card selectors as before, but it no longer assumes the data is available the moment driver.get() returns.

Why not use time.sleep()?

A fixed pause such as ime.sleep(2) does not check whether the page is ready. Two seconds may be too short on a slow connection, causing an intermittent error; on a fast connection, it simply makes every request slower than necessary.

An explicit wait targets the condition that actually matters: quote cards are present. It proceeds as soon as that is true and raises a useful timeout error if it never becomes true.

Implicit vs. explicit waits

An implicit wait sets one global timeout for every find_element() call. It can be convenient for simple scripts, but it does not describe what your page is waiting for and can make debugging harder.

An explicit wait is more precise: you use it for the particular element or state required at that point in the workflow. For scraping dynamic pages, it is usually the better default. Do not mix implicit and explicit waits in the same scraper, as their timeouts can interact in confusing ways.

Watch out for

Wait for the right condition. presence_of_element_located() is enough when you only need to read an element’s text, as in this example. Before clicking a button, use a condition such as element_to_be_clickable() instead.

Also, do not collect element references before a page finishes updating. JavaScript frameworks can replace old DOM nodes during a re-render, producing a StaleElementReferenceException. Waiting first and then locating the quote cards avoids that problem.

Result

After roughly 10 seconds (the intentional delay built into this sandbox) the script should print:

Found 10 delayed quotes
First author: Albert Einstein

Your scraper can now handle quote data that appears after the page initially loads. Next, we will make Selenium interact with the page by filling a form, clicking controls, and scrolling for additional content.

Step 5: Interacting with Pages

Goal: Fill and submit a form, click a page control, and scroll until an infinite-scroll page loads more quote cards.

Selenium is useful when collecting data requires an action before the page reveals it. On the Quotes to Scrape sandbox, we can use its demo login form and then move to its scrolling endpoint, where more quotes load as the browser reaches the bottom of the page.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

LOGIN_URL = "https://quotes.toscrape.com/login"
SCROLL_URL = "https://quotes.toscrape.com/scroll"


def build_driver() -> webdriver.Chrome:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,1200")

    return webdriver.Chrome(options=options)


driver = build_driver()
wait = WebDriverWait(driver, 10)

try:
    # Fill and submit the sandbox's demo login form.
    driver.get(LOGIN_URL)

    username = wait.until(
        EC.visibility_of_element_located((By.ID, "username"))
    )
    password = driver.find_element(By.ID, "password")

    username.send_keys("demo_user")
    password.send_keys("demo_password")

    login_button = wait.until(
        EC.element_to_be_clickable(
            (By.CSS_SELECTOR, "input[type='submit']")
        )
    )
    login_button.click()

    wait.until(
        EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, ".quote")
        )
    )
    print("Login form submitted")

    # Scroll until more quote cards appear.
    driver.get(SCROLL_URL)

    wait.until(
        EC.presence_of_all_elements_located(
            (By.CSS_SELECTOR, ".quote")
        )
    )

    quote_count = len(driver.find_elements(By.CSS_SELECTOR, ".quote"))
    print(f"Initial quote cards: {quote_count}")

    for scroll_number in range(1, 4):
        driver.execute_script(
            "window.scrollTo(0, document.body.scrollHeight);"
        )

        wait.until(
            lambda browser: len(
                browser.find_elements(By.CSS_SELECTOR, ".quote")
            ) > quote_count
        )

        quote_count = len(driver.find_elements(By.CSS_SELECTOR, ".quote"))
        print(f"After scroll {scroll_number}: {quote_count} quote cards")
finally:
    driver.quit()

How it works

send_keys() enters text into each form field, while click() activates the login button once Selenium considers it clickable. The sandbox accepts the demonstration credentials, then returns to a page containing quote cards.

For the scrolling example, execute_script() runs JavaScript in the page and moves the browser to the bottom. The custom wait.until() condition checks whether the number of .quote cards has increased before the next scroll begins.

Watch out for

Do not use a fixed number of scrolls in a production scraper without a stopping rule. An infinite-scroll feed may continue indefinitely, return duplicate records, or stop loading without warning. In a real workflow, stop after a target number of records, a known number of empty scroll attempts, or a date or URL boundary.

Use form automation only where you are authorized to access the account and data. Selenium can submit a form, but it does not make private or access-controlled data permissible to collect.

Result

The exact count can vary as the page loads, but the output should show the number of quote cards increasing after each scroll:

Login form submitted
Initial quote cards: 10
After scroll 1: 20 quote cards
After scroll 2: 30 quote cards
After scroll 3: 40 quote cards

You can now make Selenium act on a page rather than only read it. Next, we will apply the same quote extraction logic across multiple pages and cover the real-world constraints that appear as a scraper grows.

Step 6: Handling Common Challenges: Blocks, CAPTCHAs, and Pagination

Goal: Follow the quote site’s pagination until there are no more pages, while treating missing content and access restrictions as signals to stop and investigate.

A scraper that works on one page still needs to handle the next-page link, repeated records, and unexpected page states. The Quotes to Scrape sandbox has ten pages of quotes, so we can extend the extractor from the previous steps into a complete multi-page scraper.

from urllib.parse import urljoin

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

START_URL = "https://quotes.toscrape.com/"


def build_driver() -> webdriver.Chrome:
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1440,1200")

    return webdriver.Chrome(options=options)


def extract_quotes(driver: webdriver.Chrome) -> list[dict[str, str]]:
    quote_cards = driver.find_elements(By.CSS_SELECTOR, ".quote")

    return [
        {
            "text": card.find_element(By.CSS_SELECTOR, ".text").text,
            "author": card.find_element(By.CSS_SELECTOR, ".author").text,
        }
        for card in quote_cards
    ]


driver = build_driver()
wait = WebDriverWait(driver, 10)

all_quotes = []
seen_quotes = set()
page_number = 1

try:
    driver.get(START_URL)

    while True:
        try:
            wait.until(
                EC.presence_of_all_elements_located(
                    (By.CSS_SELECTOR, ".quote")
                )
            )
        except TimeoutException as error:
            raise RuntimeError(
                "Quote cards did not appear. Stop and inspect the page "
                "before retrying."
            ) from error

        for quote in extract_quotes(driver):
            quote_key = (quote["text"], quote["author"])

            if quote_key not in seen_quotes:
                all_quotes.append(quote)
                seen_quotes.add(quote_key)

        print(
            f"Page {page_number}: collected "
            f"{len(all_quotes)} unique quotes"
        )

        next_links = driver.find_elements(
            By.CSS_SELECTOR,
            "li.next > a",
        )

        if not next_links:
            break

        next_url = urljoin(
            driver.current_url,
            next_links[0].get_attribute("href"),
        )
        driver.get(next_url)
        page_number += 1

    print(f"Total unique quotes: {len(all_quotes)}")
finally:
    driver.quit()

How it works

The script extracts the quote cards on the current page, then looks for a Next link. When it finds one, urljoin() turns the relative link into a complete URL, Selenium opens it, and the loop repeats.

seen_quotes prevents duplicate records from entering the final list if a page changes or unexpectedly repeats an item. When no next-page link remains, the loop ends naturally.

Watch out for

A pagination loop needs a stopping rule. This example stops when the site has no next-page link, but other sites may use numbered pages, a “Load more” button, or infinite scroll. Always add a clear boundary (final page, a maximum number of records, date range, etc.) so the scraper cannot run indefinitely.

If the expected quote cards never appear, the script raises an error instead of retrying forever. Selenium does not expose every HTTP response code directly, so a blocked, redirected, or changed page may first show up as missing elements or an unexpected page title.

What to do about blocks and CAPTCHAs

A CAPTCHA or access-restriction page is a signal to pause. Do not keep increasing request volume of web scraping proxies, blindly retrying, or attempting to automate around an access control. Check the site’s terms, robots guidance, available API, your authorization, and whether the data collection is lawful for the data and jurisdiction involved.

For authorized workflows, you may still need reliable routing across approved locations or managed IP rotation. In that case, use a provider with documented sourcing and operational controls. Infatica’s 35M+ ethically sourced residential IPs can provide that infrastructure layer, but they do not override a site’s restrictions or replace compliance review. 

Once a project requires browser instances, geographic routing, retries, and access-review processes across many URLs, maintaining Selenium yourself may no longer be the simplest option. We will compare Selenium with lighter tools and managed APIs in the next section.

Result

The sandbox contains ten pages of ten quotes each, so the final lines should read:

Page 10: collected 100 unique quotes
Total unique quotes: 100

You now have a complete Selenium scraper that loads a page, identifies structured data, waits for dynamic content, interacts with a page, and follows pagination.

When Not to Use Selenium (and What to Use Instead)

Selenium is a capable browser-automation tool, but it is not the automatic best choice for every web scraping job. Launching and controlling a full browser adds time, memory use, and maintenance work, so the right tool depends on how the target site delivers its data and how much infrastructure the project requires.

When the operational work around browser instances, retries, routing, and monitoring outweighs the extraction logic, a managed API may be the better fit. Infatica’s Web Scraper API returns page output through one endpoint, so teams can focus on using data from permitted targets rather than maintaining browser infrastructure.

Selenium Web Scraping FAQ

Jan Wiśniewski

Jan is a content manager at Infatica. He is curious to see how technology can be used to help people and explores how proxies can help to address the problem of internet freedom and online safety.