The curl command is a versatile command line tool used to transfer data to and from servers using various protocols like HTTP, FTP, SFTP, SMTP, and many more. It is powered by the libcurl library and is designed to work without any user interaction, making it very useful for automation, testing APIs, debugging connections, and more.
curl has been around since 1997 and has become an essential tool in many environments given its flexibility. It can not only download or upload files but also be used to access APIs, send emails, scrape webpages, and much more.
What is Curl Command?
Curl is a command-line tool for transferring data to and from servers over various protocols. It is a versatile tool that can be used to send and receive data over the internet, and it is widely used in Linux and other Unix-like operating systems.
Curl supports a wide range of protocols, including HTTP, HTTPS, FTP, SMTP, POP3, IMAP, and many others. It is designed to work with URLs and can be used to perform various operations on them, including sending GET and POST requests, setting headers, and uploading and downloading files.
Installing Curl
Curl is usually pre-installed in most Linux distributions, but if it is not, you can install it using the package manager of your distribution. For example, to install Curl in Debian-based distributions like Ubuntu, you can run the following command:
sudo apt install curl
Similarly, for Red Hat-based distributions like CentOS, you can run the following command:
sudo dnf install curl
Once installed, you can start using Curl to send and receive data over the internet.
Basic syntax
The basic syntax of curl is curl [options] [URL], where options modify the request, and the URL is the remote resource.
Here are some common options:
-O: Save output to a file with the original file name from the URL.-L: Follow redirects if the server reports that the requested page has moved.-v: Provide verbose output for debugging, showing the entire client-server communication.-i: Include HTTP response headers in the output.-k: Allow connections to SSL sites without certificates.
For example, to download a file from a website and save it with the original file name, you can use the -O option:
curl -O https://example.com/file.txt
Retrieving a web page
To retrieve a web page using the curl command, simply provide the URL of the page as an argument. For example, to retrieve the home page of Google, you can use the following command:
curl https://www.google.com
This will fetch the HTML content of the Google home page and display it on your terminal.
Saving output to a file
If you want to save the output of the curl command to a file, you can use the -o or --output option followed by the name of the output file. For example, to save the Google home page to a file named google.html, you can use the following command:
curl -o google.html https://www.google.com
This will save the HTML content of the Google home page to a file named google.html.
Sending data to a server
Curl can also be used to send data to a server using various HTTP methods like POST, PUT, and DELETE. To send data using the POST method, you can use the -d or --data option followed by the data you want to send. For example, to send the data username=admin&password=admin12345 to a server using the POST method, you can use the following command:
curl -d "username=admin&password=admin12345" https://example.com/login
This will send the data to the server at the URL https://your-domain.com/login using the POST method.
Setting HTTP headers
You can set various HTTP headers using the curl command to modify the behavior of the request. To set an HTTP header, you can use the -H or --header option followed by the header name and value. For example, to set the User-Agent header to Mozilla/5.0 in a request, you can use the following command:
curl -H "User-Agent: Mozilla/5.0" https://www.your-domain.com
Following redirects
If a server returns a redirect response (status code 3xx), curl will not follow the redirect by default. To enable automatic following of redirects, you can use the -L or --location option. For example, to follow redirects while retrieving a web page, you can use the following command:
curl -L https://www.your-domain.com
Sending POST Requests with Curl
Curl can also be used to send POST requests to a remote server. Here is an example command to send a POST request to a remote server:
curl -X POST -d "name=Meilana&age=27" https://example.com
This command, -X POST specifies that we are using the HTTP POST method. -d "name=Meilana&age=27" specifies the data we want to send in the POST request. https://example.com is the URL of the server to which we are sending the POST request.
We can also use the -H option to set headers in the POST request. For example, to set the Content-Type header to application/json, we can use the following command:
curl -X POST -H "Content-Type: application/json" -d '{"name": "Meilana", "age": 27}' https://example.com
Uploading Files with PUT
Curl can also be used to upload files with HTTP PUT requests. This is useful for transferring files to a server, such as uploading an image or a document.
To upload a file with curl, use the -T option followed by the path to the file:
curl -X PUT http://your-domain.com/upload -T ~/Documents/file.pdf
In this example, we are uploading the file.pdf located in the ~/Documents/ directory to the http://your-domain.com/upload URL using the PUT method.
Authenticating with Basic Auth
Curl supports HTTP Basic Authentication, which is a simple authentication scheme that uses a username and password. Basic Auth credentials are sent in the HTTP headers with every request.
To authenticate with Basic Auth, use the -u option followed by the username and password separated by a colon:
curl -u username:password http://your-domain.com/protected
In this example, we are authenticating with the username username and password password to access the http://your-domain.com/protected URL.
Using SSL/TLS
Curl can also be used with SSL/TLS encrypted connections. By default, curl verifies the server’s certificate using the system’s trusted certificate authorities. However, it is also possible to disable certificate verification with the -k option (not recommended for production use).
To use SSL/TLS, simply use the https:// URL scheme instead of http://:
curl https://example.com
Using Proxies
Curl can be used with a proxy server by specifying the --proxy option followed by the proxy server’s URL:
curl --proxy http://proxy.example.com http://example.com
In this example, we are using the proxy server at http://proxy.example.com to access the http://example.com URL.
Curl Command Options
Here is a detailed table of Curl command options with their descriptions:
| Option | Description |
|---|---|
| -A, –user-agent <string> | Set custom user agent string |
| -b, –cookie <name=data> | Send cookies from string/file |
| -c, –cookie-jar <file name> | Write cookies to a file after the operation |
| -d, –data <data> | HTTP POST data |
| -e, –referer <URL> | Referrer URL |
| -F, –form <name=content> | HTTP multipart POST data |
| -H, –header <header> | Extra header to include in the request |
| -i, –include | Include protocol headers in the output |
| -I, –head | Fetch headers only |
| -k, –insecure | Allow insecure server connections when using SSL |
| -L, –location | Follow redirects |
| -o, –output <file name> | Write output to a file instead of stdout |
| -O, –remote-name | Write output to a file with the same name as the remote file |
| -s, –silent | Silent mode, don’t output anything |
| -T, –upload-file <file name> | HTTP PUT data from a file |
| -u, –user <user:password> | Set username and password for server authentication |
| -v, –verbose | Verbose mode, outputs more information |
| -w, –write-out <format> | Custom output format |
| -x, –proxy <[protocol://]host[:port]> | Use the specified proxy server |
| –cert <certificate[:password]> | Client certificate file and password |
| –compressed | Enable compression |
| –connect-timeout <seconds> | Maximum time to wait for a connection |
| –digest | Use HTTP Digest authentication |
| –insecure | Allow insecure server connections when using SSL |
| –ipv4 | Use IPv4 only |
| –ipv6 | Use IPv6 only |
| –keepalive-time <seconds> | Time to keep connections alive |
| –max-filesize <bytes> | Maximum size of downloaded files |
| –max-redirs <num> | Maximum number of redirects to follow |
| –netrc | Use credentials from .netrc file |
| –no-keepalive | Disable keepalive |
| –ntlm | Use NTLM authentication |
| –retry <num> | Number of retries on failure |
| –ssl | Use SSL |
| –tlsv1.0, –tlsv1.1, –tlsv1.2 | Use the specified TLS version |
| –user-agent <string> | Set custom user agent string |
It is important to note that these options can be used individually or in combination to customize and fine-tune the Curl command for specific needs. By mastering these options, users can perform a wide range of HTTP/HTTPS requests and automate various web-related tasks on the command line.
Conclusion
Curl is a versatile and powerful tool for transferring data and troubleshooting network issues. Whether you’re a developer testing an API, a system administrator automating file transfers, or a power user downloading files, understanding how to use curl can significantly enhance your command-line proficiency. With its wide array of options and flexibility, curl is a tool that every Linux user should master.
