Curl with proxy is a powerful combination for transferring data all across the web. This article will teach you the steps of using curl with proxy to enhance your security and privacy, access geo-restricted websites, and bypass network restrictions. You will also learn how to use different proxy protocols such as HTTP, SOCKS, and HTTPS with curl, and how to configure curl proxy settings for optimal performance and functionality.
Curl 101
📂 Further reading: Introduction to curl: A Light and Powerful Web Scraping Tool
Curl is a command line tool that allows you to transfer data from or to a server using various protocols. It can be used for tasks such as fetching web pages, uploading files, sending emails, and more. curl is powered by libcurl, a library that provides a wide range of features and supports many protocols, such as HTTP, HTTPS, FTP, FTPS, SFTP, SCP, SMTP, POP3, IMAP, MQTT, and more.
Basic curl commands
IMAGE
To run curl commands, we’ll only need a terminal app: Curl is preinstalled in most Linux distributions, macOS, and modern Windows versions. To use curl, you need to specify a URL and (additionally) some options that modify the behavior of the command. The basic syntax of curl is:
curl [options] [url]
For example, to get the content of a web page and print it to the terminal, you can use:
curl HTTPs://example.com
To save the content of a web page to a local file, you can use the -o or -O options:
curl -o page.html HTTPs://example.com
curl -O HTTPs://example.com/index.html
To follow redirects and get the final destination of a URL, you can use the -L option:
curl -L HTTPs://example.com/redirect
To view the response headers of a URL, you can use the -I option:
curl -I HTTPs://example.com
To view the request headers and connection information of a URL, you can use the -v option:
curl -v HTTPs://example.com
Your proxy information
To make curl use a proxy, you’ll need the following proxy details:
- Proxy server address (e.g.
123.0.0.1
orproxy3.infatica.io
), - Proxy port (e.g.
6789
), - Proxy protocol (e.g.
HTTP
).
Curl proxy authentication
In some instances, you’ll need to run curl with proxy authentication, so you’ll need proxy login details, too:
- Login (e.g.
james
), - Password (e.g.
passw0rd
).
To set your curl proxy password and login, you can use the -U option:
curl -U james:passw0rd -x myproxy:80 HTTP://home.com
This curl proxy authentication method may not work with some proxy formats. Using the --proxy-anyauth
option, you can force curl to try different methods until the authentication is completed:
curl -U james:passw0rd -x myproxy:80 HTTP://home.com --proxy-anyauth
Use curl with HTTP(S) proxy
HTTP is the standard curl proxy protocol, so you don’t need special options for setting a HTTP proxy in curl. The default parameters that make curl specify a proxy request (-x
and --proxy
) are used for both HTTP and HTTPS requests:
curl -x proxy3.infatica.io:80 https://example.com/
curl --proxy proxy3.infatica.io:80 https://example.com/
Use curl with SOCKS proxy
SOCKS is another popular protocol – and we can make curl use SOCKS proxy in just a single command. To make a curl SOCKS5 proxy request, use the --socks5
option:
curl --socks5 proxy3.infatica.io HTTP://www.example.com/
To make a curl SOCKS4 proxy request, use the --socks4
option:
curl --socks4 proxy3.infatica.io HTTP://www.example.com/
Note that these commands don’t resolve the hostname locally. To avoid this, use the --socks4a
and --socks5-hostname
options, respectively:
curl --socks4a proxy3.infatica.io HTTP://www.example.com/
curl --socks5-hostname proxy3.infatica.io HTTP://www.example.com/
Add a curl proxy config file
Some users prefer curl to set proxy exclusivity, i.e. to only use the proxy within curl and not other applications. Additionally, you can add as many command line options as you like and not type them manually for each proxy request. This can be achieved via a curl proxy configuration file: For instance, if you want to add proxy to curl and use it automatically, add a line like this:
proxy="http://james:passw0rd@123.0.0.1:6789"
On Windows
On Windows, a curl config file can be named .curlrc
or _curlrc
. Curl searches for a config file in several folders; one option is placing it in your user profile folder (e.g. C:\Users\james\.curlrc
).
On Linux and macOS
On Linux and macOS, curl config files are typically named .curlrc
and placed in folders like $CURL_HOME
and $HOME
.
Disable proxies for specific requests
In some scenarios, you might want to run curl without a proxy for specific requests – this can be useful when you need to access local resources or bypass proxy restrictions. You have two options for doing that: curl proxy environment variables and the --noproxy
command argument.
curl --noproxy
The --noproxy option allows you to specify a comma-separated list of hosts or domains that do not use a proxy. For example, if you want curl to disable proxy for localhost and home.com, you can use the following command:
curl --noproxy localhost,home.com HTTP://localhost/mysite
You can also use a wildcard character (*
) to match all hosts and effectively make curl ignore proxy for all requests. Note that the *
character should be quoted so that it is not expanded by the shell. For example:
curl --noproxy "*" HTTP://localhost/mysite
curl no_proxy environment variable
The no_proxy
environment variable works similarly to the --noproxy
option, but it affects all commands that use the HTTP_proxy
and HTTPs_proxy
environment variables. To set the no_proxy
variable, you can use the export
command in Linux or macOS, or the set
command in Windows. For example:
export no_proxy=localhost,home.com
set no_proxy=localhost,home.com
This will make curl bypass proxy for localhost and home.com for all subsequent requests. To unset the no_proxy
environment variable, you can use the unset
command in Linux or macOS, or the set
command with an empty value in Windows. For example:
unset no_proxy
set no_proxy=
Frequently Asked Questions
HTTP, SOCKS, and HTTPS proxies are different types of proxies that use different protocols to transfer data between clients and servers. HTTP proxies only support HTTP or HTTPS requests, while SOCKS proxies support other types of requests. HTTPS proxies are HTTP proxies that encrypt the data with SSL/TLS certificates.
🧭 Further reading: SOCKS5 vs HTTP Proxies
Free proxies are often unreliable, slow, and insecure. They may not support the protocols or options that curl needs to transfer data efficiently and securely. They may also be overloaded with traffic or blocked by websites that detect them as proxies. Using free proxies with curl may result in errors, timeouts, or data leaks.
🧭 Further reading: Paid vs. Free Proxies
-x
or --proxy
option followed by the proxy address and port number. For example: curl -x HTTP://proxy.example.com:3128 HTTPs://example.com
. You can also set the HTTP_proxy
environment variable to use the proxy for all curl requests.
-U
or --proxy-user
option followed by the credentials. For example: curl -U user:pass -x HTTP://proxy.example.com:3128 HTTPs://example.com
. You can also include the credentials in the proxy address like this: curl -x HTTP://user:pass@proxy.example.com:3128 HTTPs://example.com
.
-L
or --location
option to tell curl to follow any Location headers that the server sends. For example: curl -L -x HTTP://proxy.example.com:3128 HTTPs://example.com
. To view the response headers of a URL, you can use the -I
or --head
option. For example: curl -I -x HTTP://proxy.example.com:3128 HTTPs://example.com
.
curl -x HTTP://proxy.example.com:3128 ftp://example.com/file.txt
. You can also use different proxy protocols such as SOCKS or HTTPS with curl by using the --SOCKS5
or --proxy-insecure
options. For example: curl --SOCKS5 user:pass@proxy.example.com:1080 HTTPs://example.com
.