Web Scraping Techniques in 2026: From Basic to Advanced

Pavlo Zinkovski 27 Jul 2026 26 min read

Web scraping can be as simple as downloading one static page or as complex as running a distributed pipeline across thousands of dynamic pages. The difference lies in the technique you apply at each stage. This guide breaks down the main web scraping techniques by workflow stage, from basic fetching and parsing to anti-blocking and production-scale reliability.

Techniques at a Glance

What Is Web Scraping?

Web scraping is the automated extraction of data from websites. However, a "technique" covers a lot of ground: from a one-line HTTP request to distributed, fingerprint-evading scraping at millions of pages.

Every scraping job, however simple or advanced, moves through the same five stages: fetching the page's raw HTML, parsing that HTML to extract the data you want, handling dynamic content when JavaScript loads data after the page does, avoiding blocks so sites don't shut you out, and scaling when one page becomes a million. The techniques below are organized by stage, so you can jump to whichever one you're stuck on.

Stage 1: Fetching the Page

Fetching is the first step in any scraping pipeline: your scraper sends a request to a URL and receives a response, usually raw HTML. For simple static pages, this stage may be enough to collect the page source and move straight to parsing. For larger or more sensitive projects, fetching also includes session handling, request headers, and faster request patterns.

Basic HTTP requests

What it is: Basic HTTP fetching means sending a request to a web page and downloading the HTML response your parser will work with.

When to use it: Use basic requests when the page is static, publicly available, and does not require login, JavaScript rendering, or complex session state. This is usually the fastest and simplest technique for small scraping jobs, one-off scripts, and early prototypes.

How it works: A scraper sends an HTTP GET request to a URL, waits for the server response, checks the status code, and reads the returned HTML. In Python, this is often done with the requests library:

import requests

url = "https://example.com/products"

response = requests.get(url, timeout=10)
response.raise_for_status()

html = response.text
print(html[:500])

The important part is not just getting the response, but handling it safely: set a timeout, check for errors, and avoid assuming every request will return valid HTML.

Watch out for: A basic request only downloads what the server sends in the initial response. If the data is loaded later by JavaScript, stored behind an API call, or shown only after a user action, a plain HTTP request will not capture it.

Level: Basic

Sessions and cookies

What it is: Sessions and cookies let your scraper preserve state across multiple requests, similar to how a browser remembers a user as they move between pages.

When to use it: Use sessions when the website relies on cookies for pagination, location settings, language preferences, shopping carts, consent banners, or multi-step navigation. They become important as soon as one request depends on what happened in a previous request.

How it works: Instead of sending isolated requests, you create a session object that stores cookies and reuses connection settings automatically. If the first request sets a cookie, the next request can send it back without you copying it manually:

import requests

session = requests.Session()

# First request may set cookies or preferences
session.get("https://example.com", timeout=10)

# Later requests reuse the same session state
response = session.get("https://example.com/products?page=2", timeout=10)
response.raise_for_status()

html = response.text

This is useful for scraping flows where the site expects continuity. For example, a product listing may return different results after a region selector, currency preference, or consent choice has been stored in a cookie.

Watch out for: Sessions are not a substitute for authorization. Do not use them to access content you are not allowed to access, and avoid scraping behind logins unless you have clear permission. Also, cookies can expire, change format, or become tied to other request signals, so session-based scrapers need error handling.

Level: Basic to Intermediate

Custom headers and user-agents

What it is: Custom headers let your scraper send browser-like request metadata, such as the user-agent, accepted content types, language, and referrer.

When to use it: Use custom headers when a site returns different content depending on the client, rejects obviously incomplete requests, or serves localized results based on language and region headers. They are also useful for making your scraper’s requests more explicit and predictable.

How it works: Browsers send a set of HTTP headers with every request. A minimal script may send only a few, which can lead to unusual responses or missing content. Adding realistic headers can make the server return the same version of the page that a normal browser would receive:

import requests

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}

response = requests.get("https://example.com/products", headers=headers, timeout=10)
response.raise_for_status()

html = response.text

The user-agent tells the server what kind of client is making the request. Other headers help define what formats the scraper accepts and which language version it expects.

Watch out for: Headers should be consistent. Sending a Chrome user-agent with odd or missing browser headers can look unnatural on stricter sites. Headers alone do not solve JavaScript rendering, rate limits, IP reputation, or advanced fingerprinting checks.

Level: Intermediate

Async fetching

What it is: Async fetching sends many HTTP requests concurrently instead of waiting for each page to finish before starting the next one.

When to use it: Use async fetching when you need to collect many static pages quickly and the bottleneck is network waiting time, not parsing or rendering. It becomes worth it when a scraper moves from a few URLs to hundreds or thousands of pages.

How it works: In a normal synchronous scraper, each request blocks the script until the server responds. Async libraries such as httpx or aiohttp let the scraper start other requests while earlier ones are still waiting. This can dramatically improve throughput on static pages:

import asyncio
import httpx

urls = [
    "https://example.com/products?page=1",
    "https://example.com/products?page=2",
    "https://example.com/products?page=3",
]

async def fetch(client, url):
    response = await client.get(url, timeout=10)
    response.raise_for_status()
    return response.text

async def main():
    async with httpx.AsyncClient() as client:
        pages = await asyncio.gather(*(fetch(client, url) for url in urls))

    print(f"Fetched {len(pages)} pages")

asyncio.run(main())

This is one of the fastest techniques for static websites because it avoids the overhead of browser rendering. The scraper still receives raw HTML, but it can collect many pages in the time a synchronous script would spend waiting.

Watch out for: Faster is not always better. Too much concurrency can overload the target site, trigger rate limits, or create unstable results. Use sensible limits, add retries, respect crawl rules, and monitor response codes so speed does not come at the cost of reliability.

Level: Advanced

Stage 2: Parsing the HTML

Once you have fetched the page, the next step is parsing: turning raw HTML into the specific fields your scraper needs. This might be a product name, price, article title, review count, job description, or any other piece of structured data hidden inside the page. The right parsing technique depends on how predictable the HTML is and how much the page layout changes over time.

BeautifulSoup and CSS selectors

What it is: BeautifulSoup with CSS selectors is a simple way to search HTML elements by tag, class, ID, attribute, or document structure.

When to use it: Use this technique for static pages with reasonably clean HTML, especially when you are extracting common fields like titles, links, prices, dates, or product names. It is usually the first parsing method beginners learn because it is readable and quick to prototype.

How it works: BeautifulSoup turns raw HTML into a searchable document tree. CSS selectors then let you target elements in a familiar way, similar to how front-end developers style pages with CSS.

from bs4 import BeautifulSoup
import requests

url = "https://example.com/products"
response = requests.get(url, timeout=10)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

products = []

for card in soup.select(".product-card"):
    name = card.select_one(".product-name")
    price = card.select_one(".product-price")

    products.append({
        "name": name.get_text(strip=True) if name else None,
        "price": price.get_text(strip=True) if price else None,
    })

print(products)

CSS selectors work well when the page has stable class names or predictable HTML blocks. They are also easy to read later, which matters when someone has to maintain the scraper.

Watch out for: CSS selectors can break when class names change, when the site redesigns its layout, or when the data is loaded by JavaScript after the initial HTML response. Always handle missing elements instead of assuming every selector will match.

Level: Basic

XPath

What it is: XPath is a query language for selecting elements from an HTML or XML document using their position, attributes, text, or relationship to other elements.

When to use it: Use XPath when CSS selectors are not expressive enough, especially when you need to select elements based on nearby text, parent-child relationships, sibling elements, or partial attribute matches.

How it works: XPath can navigate the document tree more precisely than CSS. For example, instead of selecting “the element with this class,” you can select “the price next to a label that says Price” or “all links inside a specific table row.”

import requests
from lxml import html

url = "https://example.com/products"
response = requests.get(url, timeout=10)
response.raise_for_status()

tree = html.fromstring(response.text)

names = tree.xpath("//div[contains(@class, 'product-card')]//h2/text()")
prices = tree.xpath("//div[contains(@class, 'product-card')]//span[contains(@class, 'price')]/text()")

products = [
    {"name": name.strip(), "price": price.strip()}
    for name, price in zip(names, prices)
]

print(products)

XPath is especially useful on pages where the target element does not have a clean class or ID, but its position in the document is reliable.

Watch out for: Avoid long absolute XPath expressions copied directly from browser DevTools, such as /html/body/div[2]/main/div[3]/.... They are extremely brittle and often break after small layout changes. Prefer shorter relative XPath expressions based on stable attributes, text, or structure.

Level: Intermediate

Regex

What it is: Regex is a pattern-matching technique for extracting text that follows a predictable format.

When to use it: Use regex for small, well-defined text extraction tasks: pulling an SKU from a product description, normalizing a price, extracting an ID from a URL, or finding a value inside a script tag. It should support parsing, not replace an HTML parser.

How it works: Regex searches a string for a pattern. For example, after you have already selected a product details block, you might use regex to extract an SKU or numeric price from the text.

import re

text = "Product: Wireless Mouse | SKU: WM-2048 | Price: $24.99"

sku_match = re.search(r"SKU:\s*([A-Z0-9-]+)", text)
price_match = re.search(r"Price:\s*\$([0-9.]+)", text)

sku = sku_match.group(1) if sku_match else None
price = price_match.group(1) if price_match else None

print({"sku": sku, "price": price})

Regex is also useful for cleaning extracted values after parsing. For example, you can remove extra whitespace, strip currency symbols, or extract numbers from mixed text.

Watch out for: Do not use regex to parse full HTML pages. HTML is nested, inconsistent, and often malformed, which makes regex fragile for document-level extraction. Use BeautifulSoup, lxml, or another parser to locate the right part of the page first, then use regex only for predictable text patterns.

Level: Basic to Intermediate

Resilient selectors

What it is: Resilient selectors are selectors designed to survive small layout changes by targeting stable attributes, semantic structure, and fallback patterns instead of fragile class names alone.

When to use it: Use resilient selectors when a scraper needs to run continuously, especially on websites that change layout, test new designs, or use generated class names. They become important when broken selectors mean missing data, failed jobs, or manual maintenance.

How it works: Instead of relying on one brittle selector, build a selector strategy. Prefer stable attributes such as data-*, itemprop, aria-label, canonical URLs, or schema markup. Add fallbacks when the same field can appear in multiple layouts.

from bs4 import BeautifulSoup

def first_text(soup, selectors):
    for selector in selectors:
        element = soup.select_one(selector)
        if element:
            return element.get_text(strip=True)
    return None

html = """
<div class="product">
  <h1 data-testid="product-title">Wireless Mouse</h1>
  <span itemprop="price">$24.99</span>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

product_name = first_text(soup, [
    "[data-testid='product-title']",
    "h1.product-title",
    "h1"
])

price = first_text(soup, [
    "[itemprop='price']",
    ".product-price",
    "[data-testid='price']"
])

print({"name": product_name, "price": price})

This approach makes the scraper more tolerant of small HTML changes. It also makes failures easier to debug because you can see which fallback matched and which ones stopped working.

Watch out for: Resilient does not mean unbreakable: a major redesign can still invalidate every selector. Production scrapers should log missing fields, track extraction rates, and alert you when a selector starts returning empty or suspicious values.

Level: Intermediate

Structured extraction: JSON-LD and microdata

What it is: Structured extraction means pulling data from machine-readable markup embedded in the page, such as JSON-LD, microdata, or schema.org fields.

When to use it: Use structured extraction when scraping product pages, articles, recipes, job postings, reviews, events, or local business pages. These pages often include structured data for search engines, which can be cleaner and more stable than visible HTML.

How it works: Many websites include JSON-LD inside <script type="application/ld+json"> tags. Instead of scraping the visible product title and price from the layout, you can parse the JSON object directly.

import json
import requests
from bs4 import BeautifulSoup

url = "https://example.com/product"
response = requests.get(url, timeout=10)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")

structured_data = []

for script in soup.select("script[type='application/ld+json']"):
    try:
        data = json.loads(script.string)
        structured_data.append(data)
    except (TypeError, json.JSONDecodeError):
        continue

print(structured_data)

Structured data can contain fields such as name, description, brand, price, availability, ratingValue, or reviewCount. When it is available, it often reduces the amount of fragile selector logic you need to write.

Watch out for: Structured data is not always complete, accurate, or consistent with what users see on the page. Some pages include multiple JSON-LD blocks, nested graphs, or outdated values. Treat structured data as a strong extraction source, but validate important fields against the visible page when accuracy matters.

Level: Intermediate

Stage 3: Handling Dynamic Content

Some websites do not send all their data in the first HTML response. Instead, the browser loads the page shell, runs JavaScript, and then fetches products, prices, reviews, job listings, or search results afterward. When that happens, basic HTTP requests and HTML parsing are not enough. You either need to render the page like a browser or find the underlying data request the browser is making.

Why static fetching fails on JavaScript-heavy sites

What happens: Static fetching fails when the initial HTML response does not contain the data you see in the browser because JavaScript loads it after the page opens.

Where it happens: Diagnose this first whenever your scraper returns empty fields, missing product cards, or a page that looks complete in the browser but incomplete in requests. It is especially common with single-page apps, infinite-scroll pages, modern e-commerce sites, dashboards, and search interfaces.

How it happens: A basic scraper downloads the server’s first response. A browser does more: it parses the HTML, downloads JavaScript files, executes them, and may call one or more APIs to fill the page with data. You can quickly check the difference by searching the raw HTML for text that is visible in the browser.

import requests

url = "https://example.com/products"
response = requests.get(url, timeout=10)
response.raise_for_status()

html = response.text

if "Wireless Mouse" in html:
    print("The data is present in the initial HTML.")
else:
    print("The data is probably loaded later by JavaScript.")

If the target text is missing from the raw HTML, parsing harder will not help. At that point, the solution is usually either browser rendering or API interception.

Watch out for: Do not assume missing data always means JavaScript. The page may also return different content because of headers, cookies, region settings, redirects, or blocking. Check the status code, response length, and returned HTML before switching to a heavier tool.

Level: Intermediate

Headless browsers

What it is: A headless browser is a real browser controlled by code, usually without a visible window, that can execute JavaScript and return the fully rendered page.

When to use it: Use a headless browser when the data only appears after JavaScript runs, when the page requires user interactions, or when you need browser-only behavior such as clicking, scrolling, form input, or waiting for client-side routing.

How it works: Tools like Playwright and Selenium open a browser session, navigate to a URL, execute the page’s JavaScript, and let you inspect the rendered HTML or interact with elements. In Python, Playwright can load a page and extract the rendered content like this:

from playwright.sync_api import sync_playwright

url = "https://example.com/products"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto(url, wait_until="networkidle")
    html = page.content()

    print(html[:500])

    browser.close()

The main advantage is that the scraper sees the page closer to how a user’s browser sees it. This makes headless browsers useful for JavaScript-heavy pages that do not expose clean data in the initial HTML.

Watch out for: Headless browsers are much slower and more resource-heavy than plain HTTP requests. They also add operational complexity: browser versions, memory usage, crashes, timeouts, and anti-bot fingerprinting can all become maintenance issues. Use them when you need rendering, not as the default for every page.

Level: Intermediate

Waiting for elements

What it is: Waiting for elements means pausing the scraper until a specific element, text block, or network state appears instead of assuming the page is ready immediately after navigation.

When to use it: Use waits when a dynamic page loads content gradually: product grids, search results, prices, filters, reviews, or infinite-scroll sections. It becomes necessary when your scraper sometimes works and sometimes extracts empty data because it reads the page too early.

How it works: Dynamic pages often render in stages. The browser may finish the initial navigation before the data you need appears. Instead of using a fixed sleep timer, wait for a reliable selector that indicates the target content is loaded.

from playwright.sync_api import sync_playwright

url = "https://example.com/products"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto(url)
    page.wait_for_selector(".product-card", timeout=10000)

    products = page.locator(".product-card").all_text_contents()
    print(products)

    browser.close()

Selector-based waits are usually better than hardcoded delays because they adapt to faster and slower page loads. If the element appears quickly, the scraper continues. If it never appears, the scraper can fail clearly instead of silently returning empty data.

Watch out for: Waiting for the wrong selector can make the scraper unreliable. A generic container may appear before the actual data is loaded, while an overly specific selector may break after a small layout change. Prefer a selector that proves the data itself is present.

Level: Intermediate

API/XHR interception

What it is: API or XHR interception means finding the data request a web page makes in the background and calling that endpoint directly instead of rendering the full page.

When to use it: Use API interception when a JavaScript-heavy page loads structured data from a backend endpoint. It is often the best technique for infinite scroll, search results, product listings, maps, and pages that fetch JSON after the initial load.

How it works: Open the browser’s developer tools, go to the Network tab, and filter by Fetch/XHR. Then reload the page or perform the action that loads the data, such as searching, scrolling, or changing filters. If you see a request returning JSON with the fields you need, you may be able to call that endpoint directly.

import requests

api_url = "https://example.com/api/products"

params = {
    "page": 1,
    "limit": 50,
    "category": "electronics",
}

headers = {
    "Accept": "application/json",
    "User-Agent": "Mozilla/5.0",
}

response = requests.get(api_url, params=params, headers=headers, timeout=10)
response.raise_for_status()

data = response.json()

for item in data["products"]:
    print(item["name"], item["price"])

This is often faster and cleaner than browser rendering because the scraper receives structured JSON instead of a large rendered page. It can also reduce parsing complexity: instead of selecting text from HTML, you extract fields directly from a response object.

Watch out for: Background APIs are not always public or stable. Some require session cookies, tokens, signed parameters, or request headers generated by the page. Do not use intercepted endpoints to bypass access controls, and expect undocumented APIs to change without notice. Always verify that your use case is permitted and that the endpoint returns the same data users are allowed to access.

Level: Advanced

Stage 4: Avoiding Blocks

Once a scraper moves beyond a few simple requests, access limits become part of the job. Websites may throttle repeated requests, return different responses, challenge suspicious sessions, or block traffic that looks automated. The goal at this stage is not to force access where it is not allowed, but to scrape responsibly: respect crawl rules, control request volume, use reliable infrastructure, and make your scraper behave predictably.

Robots.txt and rate limiting

What it is: Robots.txt checks and rate limiting help your scraper respect a site’s crawl preferences and avoid sending too many requests too quickly.

When to use it: Use this technique on every scraping project, even small ones. It becomes especially important when you scrape multiple pages, run scheduled jobs, or collect data from the same domain repeatedly.

How it works: The robots.txt file tells crawlers which paths the site owner prefers automated agents not to access. Rate limiting controls how often your scraper sends requests, reducing load on the target site and lowering the chance of throttling.

import time
import requests
import urllib.robotparser

base_url = "https://example.com"
target_url = "https://example.com/products"

rp = urllib.robotparser.RobotFileParser()
rp.set_url(f"{base_url}/robots.txt")
rp.read()

user_agent = "MyScraperBot/1.0"

if rp.can_fetch(user_agent, target_url):
    response = requests.get(target_url, headers={"User-Agent": user_agent}, timeout=10)
    response.raise_for_status()
    html = response.text

    # Simple delay before the next request
    time.sleep(2)
else:
    print("This URL is disallowed by robots.txt")

For production jobs, rate limiting is usually handled with queues, per-domain request limits, and backoff rules when the site returns errors like 429 Too Many Requests.

Watch out for: Robots.txt is not the same as legal permission, and ignoring it is a bad signal for both ethics and reliability. Also, a fixed delay is only a starting point. Larger crawlers need domain-level limits, logging, retries, and clear stop conditions when a site starts rejecting requests.

Level: Basic

Rotating proxies

What it is: Proxy rotation sends requests through different IP addresses instead of routing every request from the same machine or server.

When to use it: Use proxy rotation when you scrape at meaningful volume, collect region-specific data, or need more reliable access to public pages that may throttle repeated traffic from one IP. It becomes important when request volume, geolocation, or IP reputation starts affecting success rates.

How it works: A scraper connects to a proxy endpoint, and the proxy provider routes each request through an IP from its pool. Datacenter proxies are fast and cost-efficient, but easier to identify as server traffic. Residential proxies use IPs associated with real consumer networks, which can be useful for public web data collection where location and normal-looking network origin matter. Mobile proxies route through mobile carrier networks and are usually reserved for harder, mobile-specific targets.

import requests

proxy_url = "http://username:password@proxy.example.com:10000"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get(
    "https://example.com/products",
    proxies=proxies,
    timeout=15,
)

response.raise_for_status()
print(response.text[:500])

Watch out for: More IPs do not automatically mean better scraping. Poor proxy quality, aggressive concurrency, or inconsistent session handling can still trigger blocks. Match the proxy type to the job: datacenter for speed and cost, residential for broader public-web reliability, and mobile only when the target or use case genuinely requires it.

Level: Intermediate

Realistic headers and TLS fingerprints

What it is: Realistic headers and TLS fingerprints make your scraper’s network requests more consistent with the type of client it claims to be.

When to use it: Use this technique when a site responds differently to scripts than to browsers, when headers affect localization or content format, or when stricter sites evaluate more than just the IP address. It matters most at scale, where small inconsistencies become easy to detect.

How it works: Basic scrapers often send sparse request headers. Real browsers send a fuller, internally consistent set of signals: user-agent, accepted content types, language, encoding, connection behavior, and TLS handshake characteristics. At the header level, you can at least avoid obviously incomplete requests:

import requests

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
}

response = requests.get(
    "https://example.com/products",
    headers=headers,
    timeout=10,
)

response.raise_for_status()

TLS fingerprinting is the deeper version of the same problem: even before headers are sent, the server can inspect how the client negotiates the HTTPS connection. A Python script, a real Chrome browser, and an outdated HTTP client may all produce different handshake patterns.

Watch out for: Do not treat headers as a magic anti-blocking layer. A Chrome user-agent with non-Chrome-like behavior can be more suspicious than a simple, honest crawler identity. TLS fingerprinting is also difficult to manage correctly in DIY scrapers, so many teams use browser automation or managed scraping infrastructure rather than trying to tune low-level network fingerprints themselves.

Level: Advanced

CAPTCHA handling

What it is: CAPTCHA handling means detecting when a site presents a challenge and deciding how your scraper should respond.

When to use it: Use CAPTCHA handling when your scraper may encounter challenge pages, blocked responses, or verification screens during legitimate public data collection. At production scale, this is less about “solving CAPTCHAs” and more about preventing your pipeline from silently treating a challenge page as valid data.

How it works: The first step is detection. Your scraper should recognize common signs of a challenge: unusual status codes, changed page titles, missing expected elements, or text such as “verify you are human.” Then it can stop, slow down, retry later, switch to a permitted access method, or route the case for manual review.

import requests

response = requests.get("https://example.com/products", timeout=10)
html = response.text.lower()

challenge_signals = [
    "captcha",
    "verify you are human",
    "unusual traffic",
    "access denied",
]

if any(signal in html for signal in challenge_signals):
    print("Challenge detected. Do not parse this as normal page data.")
else:
    print("Page looks normal. Continue parsing.")

In a reliable scraper, CAPTCHA detection should be part of quality control. A blocked page saved as if it were a product page can corrupt your dataset just as badly as a parsing error.

Watch out for: CAPTCHA is often an explicit signal that the site does not want the current traffic pattern to continue. Trying to bypass it can create legal, ethical, and account-risk issues. A better response is to reduce request volume, improve compliance, use an official API where available, or reassess whether the target and method are appropriate.

Level: Intermediate to Advanced

Human-like behavior

What it is: Human-like behavior means pacing browser actions in a way that matches normal user interaction instead of firing every click, scroll, and request instantly.

When to use it: Use this technique when you are working with browser automation for legitimate use cases, such as testing, QA, or scraping public pages that require scrolling or interaction before content loads. It becomes relevant on pages with infinite scroll, lazy-loaded results, filters, or multi-step navigation.

How it works: Real users do not load a page, click five controls, scroll to the bottom, and leave within 200 milliseconds. Browser automation can add reasonable waits, scroll gradually, and interact only when elements are ready.

import random
from playwright.sync_api import sync_playwright

url = "https://example.com/products"

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    page.goto(url)
    page.wait_for_selector(".product-card", timeout=10000)

    for _ in range(3):
        page.mouse.wheel(0, random.randint(500, 900))
        page.wait_for_timeout(random.randint(800, 1600))

    products = page.locator(".product-card").all_text_contents()
    print(products)

    browser.close()

Some advanced teams go further by modeling cursor movement paths, scroll rhythm, and interaction timing. In most cases, though, the bigger reliability wins come from simpler choices: do not overload the site, wait for real elements, keep sessions consistent, and stop when the site pushes back.

Watch out for: Human-like behavior should not be used as a cover for abusive scraping. It also adds complexity and slows collection, so it is not worth using on simple static pages. If a page can be fetched cleanly with HTTP requests or an underlying API, browser interaction is usually unnecessary overhead.

Level: Advanced

Stage 5: Scaling and Reliability

This stage is where scraping becomes an engineering system: concurrency, queues, retries, monitoring, distributed workers, and sometimes managed APIs.

Concurrency and parallelism

What it is: Concurrency and parallelism let a scraper process multiple pages at the same time instead of handling one URL after another.

When to use it: Use this technique when you need to scrape hundreds, thousands, or millions of pages and the main bottleneck is waiting for network responses. It becomes important when a sequential scraper is too slow to finish within your required time window.

How it works: A sequential scraper sends one request, waits for the response, parses it, and only then moves to the next URL. A concurrent scraper keeps multiple requests in progress at once. For static pages, asynchronous HTTP clients are often the fastest approach:

import asyncio
import httpx

urls = [
    "https://example.com/products?page=1",
    "https://example.com/products?page=2",
    "https://example.com/products?page=3",
]

async def fetch(client, url):
    response = await client.get(url, timeout=10)
    response.raise_for_status()
    return response.text

async def main():
    limits = httpx.Limits(max_connections=10)

    async with httpx.AsyncClient(limits=limits) as client:
        pages = await asyncio.gather(*(fetch(client, url) for url in urls))

    print(f"Fetched {len(pages)} pages")

asyncio.run(main())

The max_connections limit matters because unlimited concurrency can create more problems than it solves. A good scraper increases throughput while still controlling load, errors, and retry behavior.

Watch out for: High concurrency can trigger rate limits, increase block rates, overload your own infrastructure, or produce partial data if failures are not handled properly. Scale gradually, monitor status codes, and set per-domain limits instead of treating every website the same.

Level: Advanced

Separating fetch from parse

What it is: Separating fetch from parse means splitting the scraping pipeline into two steps: one that downloads pages and another that extracts data from them.

When to use it: Use this technique when scraping jobs become large, slow, or failure-prone. It is especially useful when you need to retry failed requests, reprocess saved HTML, debug extraction logic, or scale fetching and parsing independently.

How it works: In a small script, fetching and parsing often happen in the same loop. At scale, this makes debugging harder: if extraction fails, you may need to re-fetch the page just to test a selector. A more reliable setup stores the raw response first, then parses it separately.

import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"

# Step 1: Fetch and store raw HTML
response = requests.get(url, timeout=10)
response.raise_for_status()
html = response.text

with open("products.html", "w", encoding="utf-8") as file:
    file.write(html)

# Step 2: Parse saved HTML
with open("products.html", "r", encoding="utf-8") as file:
    soup = BeautifulSoup(file.read(), "html.parser")

titles = [item.get_text(strip=True) for item in soup.select(".product-title")]
print(titles)

In production, the “storage” layer might be object storage, a database, or a message queue. The key idea is that fetching and parsing become independent stages, each with its own logs, retries, and performance metrics.

Watch out for: Storing raw pages adds cost and data governance responsibilities. Avoid saving unnecessary personal data, define retention rules, and make sure stored HTML is protected appropriately. Also, keep track of when each page was fetched so downstream teams do not treat stale HTML as fresh data.

Level: Advanced

Distributed scraping

What it is: Distributed scraping spreads work across multiple machines, workers, or containers instead of running the entire job on one process.

When to use it: Use distributed scraping when a single machine cannot finish the job fast enough, when you need fault tolerance, or when scraping workloads must run continuously in production. It becomes relevant for large catalogs, frequent monitoring, or multi-market data collection.

How it works: A distributed scraper usually has a queue of URLs and multiple workers. Each worker takes a URL, fetches it, stores the result, and marks the task as complete. If one worker fails, the queue can reassign the job instead of losing it.

from queue import Queue
from threading import Thread
import requests

urls = Queue()

for page in range(1, 101):
    urls.put(f"https://example.com/products?page={page}")

def worker():
    while not urls.empty():
        url = urls.get()

        try:
            response = requests.get(url, timeout=10)
            response.raise_for_status()
            print(f"Fetched {url}: {len(response.text)} bytes")
        except requests.RequestException as error:
            print(f"Failed {url}: {error}")
        finally:
            urls.task_done()

threads = [Thread(target=worker) for _ in range(5)]

for thread in threads:
    thread.start()

urls.join()

This simplified example uses threads on one machine, but the same pattern scales to real distributed systems with Redis, Kafka, Celery, cloud queues, containers, and worker fleets.

Watch out for: Distribution multiplies complexity. You need deduplication, queue visibility timeouts, worker health checks, centralized logs, rate limits across all workers, and a way to prevent duplicate writes. Without coordination, ten workers can accidentally behave like ten separate aggressive scrapers.

Level: Advanced

Retry logic and error handling

What it is: Retry logic and error handling define what your scraper should do when requests fail, responses are incomplete, or the target site returns temporary errors.

When to use it: Use this technique in any scraper that needs reliable output. It becomes essential for scheduled jobs, production pipelines, and datasets where missing pages or corrupted rows can affect downstream decisions.

How it works: Not every failed request should be handled the same way. A timeout may be worth retrying. A 429 Too Many Requests response should usually trigger backoff. A 404 Not Found may be a permanent failure. A challenge page should be detected and excluded from normal parsing.

import time
import requests

def fetch_with_retries(url, max_attempts=3):
    for attempt in range(1, max_attempts + 1):
        try:
            response = requests.get(url, timeout=10)

            if response.status_code == 429:
                wait_time = attempt * 5
                print(f"Rate limited. Waiting {wait_time} seconds.")
                time.sleep(wait_time)
                continue

            response.raise_for_status()
            return response.text

        except requests.RequestException as error:
            if attempt == max_attempts:
                raise
            wait_time = attempt * 2
            print(f"Attempt {attempt} failed: {error}. Retrying in {wait_time}s.")
            time.sleep(wait_time)

html = fetch_with_retries("https://example.com/products")

Good retry logic should also record what happened: URL, status code, error type, attempt count, proxy used, timestamp, and whether the page was eventually recovered.

Watch out for: Blind retries can make blocking worse. If a site is returning 429, challenge pages, or repeated 403 responses, retrying faster is the wrong move. Back off, reduce concurrency, rotate responsibly, or stop the job until you understand the cause.

Level: Advanced

Switching to a managed API

What it is: A managed scraping API handles the fetching, rendering, proxy rotation, retries, and anti-blocking infrastructure behind a single endpoint.

When to use it: Use a managed API when the engineering cost of maintaining your own scraping infrastructure becomes higher than the value of building it in-house. This usually happens when you need production reliability, JavaScript rendering, proxy management, CAPTCHA handling, retries, and clean outputs across many domains.

How it works: Instead of building every scraping layer yourself, your system sends the target URL and configuration to an API. The provider handles the infrastructure-heavy parts and returns structured output such as HTML, JSON, markdown, or a rendered result.

curl -X POST "https://scrape.infatica.io/serp" \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.google.com/search?q=python+scraping"}'

Watch out for: A managed API does not remove the need for clear requirements. You still need to define target fields, freshness, allowed sources, output format, compliance boundaries, and data validation rules. It also may not be the right fit for a tiny one-off scrape where a simple HTTP request and parser are enough.

Level: Advanced

Choosing the Right Technique for Your Project

Know the techniques, but don't want to maintain them?

Proxies, fingerprints, retries, and rendering are the techniques that eat the most engineering time. Infatica's Web Scraper API handles them behind one endpoint: send a URL, receive clean JSON or markdown.

Frequently Asked Questions

Pavlo Zinkovski

As Infatica's CTO & CEO, Pavlo shares the knowledge on the technical fundamentals of proxies.