Efficient and Easy Web Scraping with Scrapy

Pavlo Zinkovski 27 Aug 2024 21 min read

Data collection often comes with challenges as web scrapers often need to navigate complex HTML structures, handle dynamic content, and respect ethical boundaries like the terms of service of the websites being scraped. This is where specialized tools like Scrapy come into play, offering a robust framework for developing efficient and ethical web scrapers. In this article, we’ll take a deep dive into Scrapy, starting with the basics and gradually moving towards more advanced features. By the end of this guide, you’ll have a solid understanding of how to build and deploy your own web scrapers using Scrapy.

Introduction to Scrapy

Scrapy is an open-source web scraping framework written in Python. It’s designed to be both powerful and flexible, enabling you to scrape data from websites with minimal code. Scrapy is particularly well-suited for large-scale scraping projects, where speed and efficiency are crucial. Unlike basic web scraping libraries like Beautiful Soup or requests, Scrapy provides a full-fledged framework that handles everything from making HTTP requests to processing and storing scraped data.

One of the key advantages of Scrapy is its ability to manage multiple requests simultaneously, making it much faster than many other scraping tools. It also comes with a rich set of features, such as built-in support for handling requests, parsing responses, following links, and exporting data in various formats. Moreover, Scrapy is highly extensible, allowing you to customize its behavior through middlewares, pipelines, and extensions.

Getting started with Scrapy

Scrapy requires Python, so make sure you have Python installed on your system. It’s also a good practice to use a virtual environment to manage dependencies for your projects, ensuring that your main Python environment remains clean and uncluttered.

Installing Scrapy

1. Install Python: If you don’t have Python installed, download and install the latest version from the official Python website. During installation, ensure you check the option to add Python to your system’s PATH.

2. Set up a virtual environment: Open a terminal or command prompt and navigate to the directory where you want to create your Scrapy project. Run the following commands to create and activate a virtual environment:

python -m venv scrapy-env
source scrapy-env/bin/activate  # On Windows, use: scrapy-env\Scripts\activate

This command creates a new directory named scrapy-env, where the virtual environment is stored.

3. Install Scrapy: With the virtual environment activated, install Scrapy using pip:

pip install scrapy

This command installs Scrapy and its dependencies within your virtual environment, ensuring that they won’t interfere with other Python projects on your system.

4. Verify the installation: To confirm that Scrapy has been installed correctly, run the following command:

scrapy

If the installation was successful, you’ll see a list of Scrapy commands and options.

Creating a scrapy project

Now that you have Scrapy installed, the next step is to create a new Scrapy project. A Scrapy project is a collection of code and settings that define how your web scraper will behave. Scrapy projects are organized into a specific directory structure, making it easy to manage even complex scraping tasks. Here’s how to create a Scrapy project:

1. Navigate to your working directory: Use the terminal or command prompt to navigate to the directory where you want to store your Scrapy project.

2. Create a new project: Run the following command to create a new Scrapy project:

scrapy startproject myproject

Replace myproject with your preferred project name. This command creates a new directory with the same name as your project, containing the initial files and folders needed for a Scrapy project.

3. Understand the project structure: Let’s take a look at the files and directories that Scrapy creates:

• myproject/: The root directory of your project.
• myproject/spiders/: This folder is where you’ll define your spiders—individual scripts that contain the logic for scraping specific websites.
• myproject/settings.py: This file contains configuration settings for your project, such as user agents, download delays, and item pipelines.
• myproject/items.py: This file defines the structure of the data you want to scrape.
• myproject/middlewares.py: Middlewares allow you to process requests and responses globally across all spiders.
• myproject/pipelines.py: Pipelines process items that are extracted by your spiders, such as saving them to a database or performing data validation.

Building your first spider

In Scrapy, a spider is a class that defines how to scrape information from a website. Each spider you create is responsible for scraping specific data from one or more websites. The spider specifies where to start scraping (the initial URL), how to follow links, and how to extract the desired information from the pages it visits.

Spiders are at the heart of any Scrapy project, and they can range from very simple to highly complex, depending on the task. In this chapter, you’ll learn how to create a basic spider to scrape data from a simple website.

Creating a simple spider

Let’s start by creating a simple spider that scrapes quotes from the Quotes to Scrape website, which is designed for learning and practicing web scraping.

1. Navigate to the spiders directory: In your project directory, navigate to the spiders folder:

cd myproject/spiders

2. Create a new spider file: Create a new Python file for your spider. Let’s name it quotes_spider.py:

touch quotes_spider.py

Open this file in your preferred code editor.

3. Define the spider class: In the quotes_spider.py file, start by importing the necessary Scrapy modules and defining the spider class:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

Let’s break down this code:

• name: Identifies the spider. It must be unique within a Scrapy project and is used when running the spider.
• start_urls: This list contains the initial URLs that the spider will start crawling from.
• parse() method: Is called to handle the response downloaded from the initial URLs. It defines how to extract the data you want.
• yield: Is used to return scraped data (as a dictionary) or to follow links to additional pages.
• CSS selectors: The response.css() method is used to select HTML elements based on CSS selectors. In this example, it’s used to extract the text of quotes, authors, and tags.

4. Running the spider: With the spider defined, it’s time to run it. Make sure you’re in the root directory of your Scrapy project, and then run the following command:

scrapy crawl quotes

This command starts the quotes spider. You’ll see Scrapy fetch the first page, extract the data, follow the next page link, and repeat the process until it reaches the last page.

5. Viewing the output: By default, Scrapy prints the scraped data to the console. You can also save the output to a file in various formats (e.g., JSON, CSV) by using the following command:

scrapy crawl quotes -o quotes.json

This command saves the scraped data to quotes.json in your project directory.

Running the spider

After you’ve successfully written your first spider, you can run it using the scrapy crawl command followed by the name of the spider. Running the spider will initiate the crawling process, starting from the URLs specified in start_urls and following the rules defined in the parse() method.

For example, to run the spider you just created, use:

scrapy crawl quotes

Scrapy will handle all the requests, follow links, and extract data based on your spider’s instructions. The results will be displayed in the terminal or saved to a file if you specified an output format.

Troubleshooting common issues

While creating and running spiders, you may encounter some common issues. Here are a few tips to help you troubleshoot:

• Spider doesn’t run: Ensure that you are in the correct directory and that the spider’s name is unique within the project.
• No data extracted: Double-check your CSS selectors or XPath expressions to make sure they correctly target the elements you want to scrape.
• Spider stops prematurely: This may occur if the parse() method does not correctly follow pagination links or if there’s a problem with the website’s structure. Make sure the next_page logic is functioning correctly.

Extracting data

In Scrapy, extracting data from web pages is done using selectors. Selectors are powerful tools that allow you to locate and extract specific elements from the HTML of a web page. Scrapy supports two main types of selectors: CSS selectors and XPath expressions.

CSS selectors are commonly used because of their simplicity and readability. They work similarly to how you might target elements in a stylesheet. XPath expressions offer more flexibility and are particularly useful when dealing with complex HTML structures or when you need to perform more advanced queries.

Let’s explore both methods using the example of scraping quotes from a website. CSS selectors are straightforward and familiar if you’ve worked with web design. Here’s how you can use them in Scrapy:

for quote in response.css('div.quote'):
    text = quote.css('span.text::text').get()
    author = quote.css('small.author::text').get()
    tags = quote.css('div.tags a.tag::text').getall()

In this example:

• div.quote selects all div elements with the class quote.
• span.text::text extracts the text inside a span element with the class text.
• small.author::text extracts the text from the small element with the class author.
• div.tags a.tag::text extracts all the tags associated with a quote, as a list.

XPath expressions provide more control and precision. Here’s how you can achieve the same extraction using XPath:

for quote in response.xpath('//div[@class="quote"]'):
    text = quote.xpath('span[@class="text"]/text()').get()
    author = quote.xpath('small[@class="author"]/text()').get()
    tags = quote.xpath('div[@class="tags"]/a[@class="tag"]/text()').getall()

In this example:

• //div[@class="quote"] selects all div elements with the class quote from anywhere in the document.
• span[@class="text"]/text() extracts the text inside the span element with the class text.
• small[@class="author"]/text() extracts the text from the small element with the class author.
• div[@class="tags"]/a[@class="tag"]/text() extracts all tag texts, similar to the CSS example.

XPath is particularly powerful when you need to navigate complex HTML or extract elements based on more advanced criteria.

Handling different data formats

Web pages contain a variety of data formats, from simple text to images, links, and structured data like tables. Scrapy provides tools to handle these different types of data.

Text extraction is the most common task in web scraping. Using either CSS or XPath selectors, you can extract text content from HTML elements as shown in the examples above. However, sometimes, you need to extract not just the text content but also the attributes of an HTML element, such as the href attribute of a link or the src attribute of an image.

Here’s how you can extract the href attribute of a link using CSS selectors:

for link in response.css('a::attr(href)').getall():
    yield {'link': link}

And using XPath:

for link in response.xpath('//a/@href').getall():
    yield {'link': link}

To extract images, you typically need the src attribute, which points to the image file. Here’s how to extract image URLs:

for img in response.css('img::attr(src)').getall():
    yield {'image_url': img}

Or with XPath:

for img in response.xpath('//img/@src').getall():
    yield {'image_url': img}

Web pages often contain structured data like tables, which require more careful extraction. You can use nested selectors or XPath expressions to navigate through table rows and cells:

for row in response.css('table tr'):
    yield {
        'column1': row.css('td:nth-child(1)::text').get(),
        'column2': row.css('td:nth-child(2)::text').get(),
    }

Or using XPath:

for row in response.xpath('//table//tr'):
    yield {
        'column1': row.xpath('td[1]/text()').get(),
        'column2': row.xpath('td[2]/text()').get(),
    }

Once you’ve scraped the data, you’ll likely want to save it in a format that’s easy to work with. Scrapy supports exporting data in several formats, including JSON, CSV, and XML.

Exporting to JSON: To export scraped data to a JSON file, you can run your spider with the following command:

scrapy crawl quotes -o quotes.json

This command will save all the scraped data into quotes.json.

Exporting to CSV: To export to CSV, use the following command:

scrapy crawl quotes -o quotes.csv

Scrapy will save the scraped data as quotes.csv, with each dictionary item from the spider’s yield statements becoming a row in the CSV file.

Exporting to XML: For XML export, use:

scrapy crawl quotes -o quotes.xml

This will output your scraped data as an XML file.

Scrapy automatically handles the conversion of the data into the chosen format, making it easy to integrate with other tools or workflows.

Advanced Scrapy features

Handling pagination

Websites often spread content across multiple pages, requiring your spider to navigate through these pages to scrape all the data. Scrapy makes it easy to handle pagination by following links from one page to the next. Let’s continue with the example of scraping quotes from the Quotes to Scrape website, which uses pagination.

Basic pagination handling: In the spider we created earlier, we included logic to follow the “Next” page link. Let’s take a closer look at how this works:

def parse(self, response):
    for quote in response.css('div.quote'):
        yield {
            'text': quote.css('span.text::text').get(),
            'author': quote.css('small.author::text').get(),
            'tags': quote.css('div.tags a.tag::text').getall(),
        }

    next_page = response.css('li.next a::attr(href)').get()
    if next_page is not None:
        yield response.follow(next_page, self.parse)

Here’s how this code works:

• next_page = response.css('li.next a::attr(href)').get(): This line selects the URL of the “Next” page by finding the href attribute of the link inside the li element with the class next.
• yield response.follow(next_page, self.parse): If a “Next” page exists, this line instructs Scrapy to follow the link and call the parse method on the new page, continuing the scraping process.

This approach works well for sites with straightforward pagination links.

Handling complex pagination: Some websites might have more complex pagination mechanisms, such as AJAX-based loading or pagination parameters in the URL. In such cases, you may need to adjust your approach.

For example, if pagination is controlled by a URL parameter (e.g., http://example.com/page=2), you can generate the URLs dynamically:

def start_requests(self):
    base_url = 'http://example.com/page='
    for page_num in range(1, 11):  # Scraping the first 10 pages
        yield scrapy.Request(url=f'{base_url}{page_num}', callback=self.parse)

In this case, the start_requests() method generates requests for each page URL by iterating through a range of page numbers.

Dealing with dynamic content

Some websites rely on JavaScript to load content dynamically, which can be challenging for traditional scraping techniques since Scrapy, by default, doesn’t execute JavaScript. However, Scrapy offers several ways to deal with dynamic content.

Using scrapy-splash: One common approach is to use Scrapy with Splash, a headless browser designed for rendering JavaScript. You’ll need to install and set up Splash, and then integrate it with Scrapy.

First, install the scrapy-splash package:

pip install scrapy-splash

Next, add Splash middleware to your Scrapy project’s settings.py:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

SPLASH_URL = 'http://localhost:8050'

In your spider, you can now use Splash to render JavaScript and retrieve the content:

import scrapy
from scrapy_splash import SplashRequest

class QuotesSpider(scrapy.Spider):
    name = 'quotes'

    def start_requests(self):
        url = 'http://quotes.toscrape.com/js/'
        yield SplashRequest(url, self.parse, args={'wait': 1})

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

In this example:

• SplashRequest: This is used instead of the standard Request to make a request via Splash, allowing the page to be rendered with JavaScript.
• args={'wait': 1}: This argument instructs Splash to wait for 1 second before returning the rendered page, ensuring that all dynamic content has loaded.

Using scrapy with Selenium

Another option for handling dynamic content is using Selenium, a web automation tool that can control a real browser. While Selenium is more resource-intensive than Scrapy, it’s useful for scraping sites with complex JavaScript. First, install Selenium:

pip install selenium

Then, in your Scrapy spider, you can use Selenium to interact with the page and extract content:

from selenium import webdriver
from scrapy import signals
from scrapy.spiders import Spider

class SeleniumSpider(Spider):
    name = 'selenium_spider'

    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(SeleniumSpider, cls).from_crawler(crawler, *args, **kwargs)
        spider.driver = webdriver.Chrome()  # Or Firefox, depending on your browser
        crawler.signals.connect(spider.spider_closed, signals.spider_closed)
        return spider

    def start_requests(self):
        self.driver.get('http://quotes.toscrape.com/js/')
        # Extract content with Selenium's methods
        quotes = self.driver.find_elements_by_class_name('quote')
        for quote in quotes:
            yield {
                'text': quote.find_element_by_class_name('text').text,
                'author': quote.find_element_by_class_name('author').text,
                'tags': [tag.text for tag in quote.find_elements_by_class_name('tag')],
            }

    def spider_closed(self, spider):
        self.driver.quit()

In this example:

• webdriver.Chrome(): Initializes the Chrome browser (you can use Firefox or another supported browser).
• self.driver.get(url): Opens the web page in the browser.
• find_elements_by_class_name(): Extracts elements using Selenium’s methods.

Using Selenium allows you to interact with the page as a user would, including clicking buttons or scrolling, which is sometimes necessary for loading additional content.

Using scrapy shell: Scrapy Shell is an interactive tool that allows you to test your scraping logic in real-time. It’s extremely useful for debugging spiders, testing selectors, and exploring the structure of a website. To launch Scrapy Shell for a specific URL, use the following command:

scrapy shell 'http://quotes.toscrape.com/'

This opens an interactive shell where you can explore the page content.

Testing selectors: In the shell, you can test CSS selectors or XPath expressions:

response.css('span.text::text').get()
response.xpath('//span[@class="text"]/text()').get()

This immediately shows the results, helping you refine your selectors before implementing them in your spider.

Data pipelines and item processing

In Scrapy, once data is scraped by a spider, it can be processed further before being stored. This processing is handled by item pipelines. Pipelines allow you to clean, validate, and modify the data, as well as to save it in a database, file, or another storage system. Pipelines are crucial for ensuring that the data you scrape is in the correct format and ready for use.

Setting up an item pipeline

To set up an item pipeline in Scrapy, you’ll need to follow these steps:

1. Define an item pipeline class: You define a pipeline as a Python class that processes the items. This class must implement a method called process_item() which receives each item as it’s scraped.
2. Enable the pipeline in settings: Once you’ve defined your pipeline, you need to enable it in your Scrapy project’s settings.py file by adding it to the ITEM_PIPELINES setting.

Let’s create a simple pipeline that cleans up and processes the scraped data. Suppose we’re scraping quotes from a website and we want to convert all author names to lowercase and remove any leading or trailing whitespace.

1. Create the pipeline class: In your Scrapy project, create a new file named pipelines.py (if it doesn’t already exist) and add the following code:

class CleanQuotesPipeline:
    def process_item(self, item, spider):
        item['author'] = item['author'].strip().lower()
        return item

In this example:

• process_item(self, item, spider): This method is called for every item that passes through the pipeline. The item is the data scraped by the spider, and `spider` is the spider that scraped it.
• strip().lower(): We use the strip() method to remove any leading or trailing whitespace and lower() to convert the text to lowercase.

2. Enable the pipeline: Next, enable the pipeline by adding it to your settings.py file:

ITEM_PIPELINES = {
    'myproject.pipelines.CleanQuotesPipeline': 300,
}

The number 300 indicates the order in which this pipeline will run if there are multiple pipelines (lower numbers run earlier).

You can define and use multiple pipelines to handle different aspects of item processing. For example, you might have one pipeline to clean data, another to validate it, and another to save it to a database.

Here’s how you can set up multiple pipelines in settings.py:

ITEM_PIPELINES = {
    'myproject.pipelines.CleanQuotesPipeline': 300,
    'myproject.pipelines.ValidateQuotesPipeline': 400,
    'myproject.pipelines.SaveToDatabasePipeline': 500,
}

Each pipeline processes the item sequentially in the order specified by the numbers.

Common pipeline tasks

Cleaning and transforming data: For example, you might want to ensure that dates are in a consistent format, numeric values are converted to integers or floats, or HTML tags are removed from text.

Here’s an example of a pipeline that removes HTML tags from the text field of a quote:

import re

class CleanHTMLPipeline:
    def process_item(self, item, spider):
        item['text'] = re.sub(r'<.*?>', '', item['text'])
        return item

This pipeline uses a regular expression to remove any HTML tags from the text field of the item.

Validating data: Validation ensures that the data scraped meets certain criteria before it’s stored or used. For instance, you might want to ensure that no fields are empty or that certain fields contain only specific types of data.

Here’s a simple validation pipeline that checks if the author field is not empty:

class ValidateQuotesPipeline:
    def process_item(self, item, spider):
        if not item.get('author'):
            raise DropItem("Missing author in %s" % item)
        return item

If the author field is missing or empty, this pipeline raises a DropItem exception, which tells Scrapy to discard the item.

Saving data to a database: One of the final steps in many pipelines is saving the processed data to a database. Scrapy can integrate with various databases like MySQL, PostgreSQL, MongoDB, or even a simple SQLite database.

Here’s an example of a pipeline that saves items to a MongoDB database:

import pymongo

class MongoPipeline:
    def open_spider(self, spider):
        self.client = pymongo.MongoClient("mongodb://localhost:27017/")
        self.db = self.client["quotes_db"]
        self.collection = self.db["quotes"]

    def close_spider(self, spider):
        self.client.close()

    def process_item(self, item, spider):
        self.collection.insert_one(dict(item))
        return item

In this example:

• open_spider(self, spider): This method is called when the spider is opened, allowing you to establish a connection to the database.
• close_spider(self, spider): This method is called when the spider closes, so you can clean up resources like closing the database connection.
• process_item(self, item, spider): This method inserts each item into the MongoDB collection.

Exporting data to a file: If you prefer to save your scraped data to a file, you can write a pipeline that exports data to a JSON, CSV, or XML file.

Here’s an example of a pipeline that writes items to a CSV file:

import csv

class CsvExportPipeline:
    def open_spider(self, spider):
        self.file = open('quotes.csv', 'w', newline='')
        self.writer = csv.writer(self.file)
        self.writer.writerow(['text', 'author', 'tags'])

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        self.writer.writerow([item['text'], item['author'], ','.join(item['tags'])])
        return item

Conditional pipelines

Sometimes you might want to apply a pipeline only to certain items or under specific conditions. For example, you might only want to save quotes by a particular author.

Here’s how you could modify a pipeline to process only certain items:

class AuthorFilterPipeline:
    def process_item(self, item, spider):
        if item['author'] == 'Albert Einstein':
            return item
        else:
            raise DropItem("Non-Einstein quote dropped: %s" % item)

In this example, only quotes by Albert Einstein are kept; all others are discarded.

Chaining pipelines with shared state: If you have multiple pipelines that need to share data, you can use the item itself or the spider object to pass information between pipelines.

For example, you might have one pipeline that processes an item and adds some metadata to it, which a later pipeline then uses:

class MetadataPipeline:
    def process_item(self, item, spider):
        item['processed_at'] = datetime.utcnow()
        return item

class SaveWithMetadataPipeline:
    def process_item(self, item, spider):
        # Use the 'processed_at' field added by the previous pipeline
        save_to_db(item)
        return item

Here, the MetadataPipeline adds a processed_at timestamp to each item, and the SaveWithMetadataPipeline can then use this information when saving the item to a database.

Handling challenges in web scraping

Websites often implement anti-scraping mechanisms to protect their data, and web scrapers must be equipped to handle various errors and exceptions that may arise during the scraping process. In this chapter, we’ll explore these challenges and provide strategies for overcoming them, including dealing with anti-scraping mechanisms and handling errors and exceptions effectively.

Dealing with anti-scraping mechanisms

Rotating user agents: One of the simplest anti-scraping measures is checking the user agent string in HTTP requests. The user agent identifies the browser or tool making the request, and many websites block requests that come from known web scraping tools.

To bypass this, you can rotate user agents with each request, making it look like the requests are coming from different browsers and devices. Scrapy allows you to customize the user agent by setting the USER_AGENT in the settings.py file:

# settings.py
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'

To rotate user agents dynamically, you can create a middleware that selects a random user agent for each request:

import random

class RotateUserAgentMiddleware:
    user_agents = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3',
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
        # Add more user agents here
    ]

    def process_request(self, request, spider):
        request.headers['User-Agent'] = random.choice(self.user_agents)

Enable this middleware in your settings.py file:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RotateUserAgentMiddleware': 400,
}

Using proxies: Websites may block requests based on IP address if they detect suspicious activity, such as multiple requests from the same IP within a short period. Using proxies allows you to distribute requests across different IP addresses, making it harder for the website to detect and block your scraper.

The Ultimate Guide to Using Proxies with Scrapy: Setup, Benefits, and Best Practices | Infatica

Learn how to use proxies with Scrapy to optimize your web scraping projects. Discover the best types of proxies, how to set them up, and why they’re essential for data collection.

Denis KryukovInfatica Blog: Learn How to Harness the Power of Data

To use proxies in Scrapy, you can specify a proxy in the Request object:

yield scrapy.Request(url, callback=self.parse, meta={'proxy': 'http://proxyserver:port'})

For rotating proxies, you can use a middleware like scrapy-proxies or scrapy-rotating-proxies, which automatically assigns a different proxy to each request:

# settings.py
ROTATING_PROXY_LIST = [
    'http://proxy1.example.com:8000',
    'http://proxy2.example.com:8031',
    'http://proxy3.example.com:8052',
]

Implementing delays and throttling: Websites often monitor the frequency of requests and block IPs that send too many requests in a short period. To avoid detection, you can implement delays between requests or use Scrapy’s AutoThrottle feature.

To manually set a delay between requests, use the DOWNLOAD_DELAY setting:

# settings.py
DOWNLOAD_DELAY = 2  # Delay of 2 seconds between requests

For automatic throttling based on the server's response times, enable AutoThrottle:

# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1  # Initial download delay
AUTOTHROTTLE_MAX_DELAY = 10  # Maximum delay
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0  # Target concurrency

AutoThrottle dynamically adjusts the request rate to balance speed and reduce the likelihood of being blocked.

Handling errors and exceptions

Web scraping projects often encounter errors and exceptions that can disrupt the scraping process. Handling these errors effectively is crucial for ensuring that your scraper remains robust and reliable.

Handling HTTP errors: Websites may return various HTTP status codes that indicate errors, such as 404 (Not Found), 500 (Internal Server Error), or 403 (Forbidden). Scrapy provides several ways to handle these HTTP errors.

You can handle HTTP errors by defining custom error-handling callbacks:

def start_requests(self):
    urls = [
        'http://example.com/page1',
        'http://example.com/page2',
    ]
    for url in urls:
        yield scrapy.Request(url, callback=self.parse, errback=self.error_handler)

def error_handler(self, failure):
    # Log the error
    self.logger.error(repr(failure))

    # Retry the request if necessary
    if failure.check(HttpError):
        response = failure.value.response
        if response.status in [500, 503]:
            self.logger.info("Retrying %s", response.url)
            yield scrapy.Request(response.url, callback=self.parse, errback=self.error_handler)

In this example:

• errback=self.error_handler: Assigns an error-handling method to each request.
• failure.check(HttpError): Checks if the error is an HTTP error, and retries the request if the error is a server-side issue (e.g., 500 or 503).

Handling missing data and parsing errors: Data extraction errors can occur if the structure of the target web page changes or if certain elements are missing. You can handle these errors by validating the data before processing it.

Here’s an example:

def parse(self, response):
    quotes = response.css('div.quote')
    for quote in quotes:
        text = quote.css('span.text::text').get()
        author = quote.css('small.author::text').get()

        if not text or not author:
            self.logger.warning("Missing data in %s", response.url)
            continue

        yield {
            'text': text,
            'author': author,
        }

In this example, the spider checks if the text or author field is missing and logs a warning instead of raising an exception.

Managing timeouts and connection errors: Scrapy allows you to set timeouts for requests, ensuring that your spider doesn’t hang indefinitely if a server is slow to respond.

Set timeouts in settings.py:

# settings.py
DOWNLOAD_TIMEOUT = 15  # Timeout after 15 seconds

To handle timeouts or connection errors, you can also implement retries:

# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3  # Number of retries
RETRY_HTTP_CODES = [500, 502, 503, 504, 408]  # HTTP codes to retry

These settings ensure that your spider retries failed requests up to three times before giving up.

Dealing with rate limiting and bans: If your scraper is sending too many requests too quickly, it may get rate-limited or banned by the target website. To handle this, you can implement automatic retries with exponential backoff, which increases the delay between retries.

Scrapy doesn’t have built-in support for exponential backoff, but you can implement it manually:

import time

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def start_requests(self):
        urls = ['http://example.com/page1', 'http://example.com/page2']
        for url in urls:
            yield scrapy.Request(url, callback=self.parse, errback=self.error_handler)

    def error_handler(self, failure):
        if failure.check(HttpError):
            response = failure.value.response
            if response.status in [429]:  # Too Many Requests
                retry_after = int(response.headers.get('Retry-After', 10))
                self.logger.info("Rate limited. Retrying in %s seconds", retry_after)
                time.sleep(retry_after)
                yield scrapy.Request(response.url, callback=self.parse, errback=self.error_handler)

In this example, if the spider encounters a 429 Too Many Requests response, it waits for the time specified in the Retry-After header before retrying the request.

Frequently Asked Questions