HTTP Headers Guide

Pavlo Zinkovski 13 Mar 2023 5 min read

HTTP headers are an important component of every web scraping pipeline: If set up correctly, they can have a significant impact on the performance and security of a website, as they can affect how fast the page loads, how well it is protected from attacks, and how it interacts with other web services. In this article, we will explore the different HTTP header types, how to use them in your web development projects, and why they are important for creating a better web experience.

Understanding HTTP headers

At its core, everything on the web (opening a text page, viewing media, sending a message, etc.) is a communication between the client and the target server. This exchange of data is realized via HTTP requests and HTTP responses.

HTTP headers provide additional information about the request being made, beyond what is included in the request line (e.g., the requested resource, the HTTP method being used). Common HTTP headers include HTTP request headers and HTTP response headers.

HTTP request headers

HTTP request headers are an important tool for web developers and server administrators. By providing additional information about the request being made, they can improve the performance, security, and functionality of web applications. Their typical usage scenarios include:

• Authentication: Request headers can be used to authenticate the client making the request. For example, the Authorization header can be used to send authentication credentials with the request.
• Caching: Headers like Cache-Control and Expires can be used to control how intermediaries cache the response, which can improve performance by reducing the number of requests that need to be made.
• Content negotiation: HTTP_Accept-* headers can be used to specify the format that the client prefers to receive the response in. This can allow servers to send the appropriate format (e.g., HTML, XML, JSON) to the client.
• Security: Headers like Referrer-Policy and X-Content-Type-Options can be used to improve the security of the request and response. For example, the Referrer-Policy header can be used to control what information is sent in the Referer header, which can help prevent certain types of attacks.

HTTP request headers typically consist of a name-value pair and are sent as part of the request message. Here are some HTTP header examples for the request subtype:

• User-Agent: This header specifies the user agent (i.e., the web browser) that is making the request. This information is often used by web servers to tailor the response to the specific browser being used.
• Accept-*: This header specifies the type of data that the client can handle, such as HTML, XML, or JSON. Servers can use this information to send the appropriate type of data in the response.
• Authorization: This header is used to authenticate the client making the request. For example, if the client is accessing a protected resource, they may need to provide a username and password in this header.
• Cache-Control: This header specifies caching directives that tell intermediaries (such as proxies) how to handle the response. For example, it can specify that the response should not be cached or that it can be cached but only for a certain period of time.

Here’s a HTTP request header example:

Request URL: https://infatica.io
Request Method: GET
Status Code: 200
Remote Address: <server_IP>:123
Referrer Policy: strict-origin-when-cross-origin

HTTP response headers

Conversely, HTTP response headers are pieces of information included in an HTTP response that provide additional context about the response. They are sent from the server to the client and can contain information such as the type of content being returned, how long it should be cached for, and whether or not it should be accessed using a secure connection.

HTTP response headers can be used for a variety of purposes. For example, the Location header can be used to redirect a client to another URL. The Server header can provide information about the software used by the server. The Strict-Transport-Security header can tell browsers that a website should only be accessed using HTTPS.

Here are some HTTP headers example for the response subtype:

• Age: Indicates how long a resource has been cached by a proxy or browser.
• Location: Used to redirect a client to another URL.
• Server: Contains information about the software used by the server.
• Strict-Transport-Security (HSTS): Lets a website tell browsers that it should only be accessed using HTTPS, instead of using HTTP.

This is a code snippet of a sample HTTP response header:

cache-control: public, max-age=30
cdn-cache: HIT
cdn-cachedat: <cache_date>
cdn-edgestorageid: <edge_storage_id>
cdn-proxyver: 1.0
cdn-pullzone: <pullzone_id>
cdn-requestcountrycode: <country_code>
cdn-requestid: <request_id>
cdn-requestpullcode: 200
cdn-requestpullsuccess: True
cdn-status: 200
cdn-uid: <uid>
content-encoding: br
content-type: text/html; charset=utf-8
date: <date>
server: homeCDN-<node_id>
vary: Accept-Encoding

Frequently Asked Questions