Curl proxy guide

This guide shows how to use curl proxy to access geo-restricted websites. Learn different ways of setting up curl with a proxy and some examples of common commands.

Curl proxy guide
Sharon Bennett
Sharon Bennett 6 min read
Article content
  1. Сurl 101
  2. Your proxy information
  3. Use curl with HTTP(S) proxy
  4. Use curl with SOCKS proxy
  5. Add a curl proxy config file
  6. Disable proxies for specific requests
  7. Frequently Asked Questions

Curl with proxy is a powerful combination for transferring data all across the web. This article will teach you the steps of using curl with proxy to enhance your security and privacy, access geo-restricted websites, and bypass network restrictions. You will also learn how to use different proxy protocols such as HTTP, SOCKS, and HTTPS with curl, and how to configure curl proxy settings for optimal performance and functionality.

Curl 101

📂 Further reading: Introduction to curl: A Light and Powerful Web Scraping Tool

Curl is a command line tool that allows you to transfer data from or to a server using various protocols. It can be used for tasks such as fetching web pages, uploading files, sending emails, and more. curl is powered by libcurl, a library that provides a wide range of features and supports many protocols, such as HTTP, HTTPS, FTP, FTPS, SFTP, SCP, SMTP, POP3, IMAP, MQTT, and more.

Basic curl commands

IMAGE

To run curl commands, we’ll only need a terminal app: Curl is preinstalled in most Linux distributions, macOS, and modern Windows versions. To use curl, you need to specify a URL and (additionally) some options that modify the behavior of the command. The basic syntax of curl is:

curl [options] [url]

For example, to get the content of a web page and print it to the terminal, you can use:

curl HTTPs://example.com

To save the content of a web page to a local file, you can use the -o or -O options:

curl -o page.html HTTPs://example.com
curl -O HTTPs://example.com/index.html

To follow redirects and get the final destination of a URL, you can use the -L option:

curl -L HTTPs://example.com/redirect

To view the response headers of a URL, you can use the -I option:

curl -I HTTPs://example.com

To view the request headers and connection information of a URL, you can use the -v option:

curl -v HTTPs://example.com

Your proxy information

To make curl use a proxy, you’ll need the following proxy details:

  • Proxy server address (e.g. 123.0.0.1 or proxy3.infatica.io),
  • Proxy port (e.g. 6789),
  • Proxy protocol (e.g. HTTP).

Curl proxy authentication

In some instances, you’ll need to run curl with proxy authentication, so you’ll need proxy login details, too:

  • Login (e.g. james),
  • Password (e.g. passw0rd).

To set your curl proxy password and login, you can use the -U option:

curl -U james:passw0rd -x myproxy:80 HTTP://home.com

This curl proxy authentication method may not work with some proxy formats. Using the --proxy-anyauth option, you can force curl to try different methods until the authentication is completed:

curl -U james:passw0rd -x myproxy:80 HTTP://home.com --proxy-anyauth

Use curl with HTTP(S) proxy

curl connects to the target website via proxy

HTTP is the standard curl proxy protocol, so you don’t need special options for setting a HTTP proxy in curl. The default parameters that make curl specify a proxy request (-x and --proxy) are used for both HTTP and HTTPS requests:

curl -x proxy3.infatica.io:80 https://example.com/
curl --proxy proxy3.infatica.io:80 https://example.com/

Use curl with SOCKS proxy

SOCKS is another popular protocol – and we can make curl use SOCKS proxy in just a single command. To make a curl SOCKS5 proxy request, use the --socks5 option:

curl --socks5 proxy3.infatica.io HTTP://www.example.com/

To make a curl SOCKS4 proxy request, use the --socks4 option:

curl --socks4 proxy3.infatica.io HTTP://www.example.com/

Note that these commands don’t resolve the hostname locally. To avoid this, use the --socks4a and --socks5-hostname options, respectively:

curl --socks4a proxy3.infatica.io HTTP://www.example.com/
curl --socks5-hostname proxy3.infatica.io HTTP://www.example.com/

Add a curl proxy config file

curl and its config file

Some users prefer curl to set proxy exclusivity, i.e. to only use the proxy within curl and not other applications. Additionally, you can add as many command line options as you like and not type them manually for each proxy request. This can be achieved via a curl proxy configuration file: For instance, if you want to add proxy to curl and use it automatically, add a line like this:

proxy="http://james:passw0rd@123.0.0.1:6789"

On Windows

On Windows, a curl config file can be named .curlrc or _curlrc. Curl searches for a config file in several folders; one option is placing it in your user profile folder (e.g. C:\Users\james\.curlrc).

On Linux and macOS

On Linux and macOS, curl config files are typically named .curlrc and placed in folders like $CURL_HOME and $HOME.

Disable proxies for specific requests

In some scenarios, you might want to run curl without a proxy for specific requests – this can be useful when you need to access local resources or bypass proxy restrictions. You have two options for doing that: curl proxy environment variables and the --noproxy command argument.

curl --noproxy

The --noproxy option allows you to specify a comma-separated list of hosts or domains that do not use a proxy. For example, if you want curl to disable proxy for localhost and home.com, you can use the following command:

curl --noproxy localhost,home.com HTTP://localhost/mysite

You can also use a wildcard character (*) to match all hosts and effectively make curl ignore proxy for all requests. Note that the * character should be quoted so that it is not expanded by the shell. For example:

curl --noproxy "*" HTTP://localhost/mysite

curl no_proxy environment variable

The no_proxy environment variable works similarly to the --noproxy option, but it affects all commands that use the HTTP_proxy and HTTPs_proxy environment variables. To set the no_proxy variable, you can use the export command in Linux or macOS, or the set command in Windows. For example:

export no_proxy=localhost,home.com
set no_proxy=localhost,home.com

This will make curl bypass proxy for localhost and home.com for all subsequent requests. To unset the no_proxy environment variable, you can use the unset command in Linux or macOS, or the set command with an empty value in Windows. For example:

unset no_proxy
set no_proxy=

Frequently Asked Questions

HTTP, SOCKS, and HTTPS proxies are different types of proxies that use different protocols to transfer data between clients and servers. HTTP proxies only support HTTP or HTTPS requests, while SOCKS proxies support other types of requests. HTTPS proxies are HTTP proxies that encrypt the data with SSL/TLS certificates.

🧭 Further reading: SOCKS5 vs HTTP Proxies

Free proxies are often unreliable, slow, and insecure. They may not support the protocols or options that curl needs to transfer data efficiently and securely. They may also be overloaded with traffic or blocked by websites that detect them as proxies. Using free proxies with curl may result in errors, timeouts, or data leaks.

🧭 Further reading: Paid vs. Free Proxies

To set up a proxy server for curl, you can use the -x or --proxy option followed by the proxy address and port number. For example: curl -x HTTP://proxy.example.com:3128 HTTPs://example.com. You can also set the HTTP_proxy environment variable to use the proxy for all curl requests.

To specify a username and password for a proxy server, you can use the -U or --proxy-user option followed by the credentials. For example: curl -U user:pass -x HTTP://proxy.example.com:3128 HTTPs://example.com. You can also include the credentials in the proxy address like this: curl -x HTTP://user:pass@proxy.example.com:3128 HTTPs://example.com.

To follow redirects and view headers with curl and a proxy, you can use the -L or --location option to tell curl to follow any Location headers that the server sends. For example: curl -L -x HTTP://proxy.example.com:3128 HTTPs://example.com. To view the response headers of a URL, you can use the -I or --head option. For example: curl -I -x HTTP://proxy.example.com:3128 HTTPs://example.com.

To use curl with a proxy for different protocols such as FTP, SMTP, or IMAP, you can specify the protocol in the URL and use the same proxy options as for HTTP or HTTPS requests. For example: curl -x HTTP://proxy.example.com:3128 ftp://example.com/file.txt. You can also use different proxy protocols such as SOCKS or HTTPS with curl by using the --SOCKS5 or --proxy-insecure options. For example: curl --SOCKS5 user:pass@proxy.example.com:1080 HTTPs://example.com.

Sharon Bennett

Sharon Bennett is a networking professional who analyzes various measures of online censorship. She lends her expertise to Infatica to explore how proxies can help to address this problem.

You can also learn more about:

What Is Browser Fingerprinting & How Does It Work?
Proxies and business
What Is Browser Fingerprinting & How Does It Work?

Explore the basics of browser fingerprints in this easy-to-understand article. Learn how they affect online privacy and security in simple terms.

Best Web Scraping Proxies: Everything You Need to Know
Proxies and business
Best Web Scraping Proxies: Everything You Need to Know

Enhance your data collection capabilities with proxies for web scraping, designed for speed, reliability, and anonymity. Access any server globally without restrictions and gather data efficiently.

How To Crawl A Website Without Getting Blocked
Proxies and business
How To Crawl A Website Without Getting Blocked

Scraping without getting blocked can be challenging, but several methods — including proxies, User-Agents, and more — can help you collet data with less blocks.

Get In Touch
Have a question about Infatica? Get in touch with our experts to learn how we can help.