How to execute multiple curl requests in parallel

Suppose you want to run multiple curl requests concurrently for whatever reason, say 10 requests at the same time. How would you go about it? In this tutorial, we demonstrate how you can execute multiple curly requests concurrently.

Run parallel requests using the xargs command

The xargs command is a command in Linux and UNIX-like operating systems that accepts or takes arguments from standard input and then runs a command for each argument. Simply put, the xargs command can take the output of a command and process it as an argument of a different command.

For instance, in the command below, xargs takes the arguments on the left side and creates directories using the argument names.

$ echo 'Africa Asia Europe America' | xargs mkdir

The xargs command can also be used to run multiple requests concurrently. If you check the manpages, you will find a section that gives you the command-line arguments as indicated below.

Using xargs, we will attempt to command the execution of multiple curl requests at the same time.

The Linux curl command is a command-line utility that is used for file transfer. It provides support for a myriad of protocols such as HTTP, HTTPS, FTP, FTPS, SCP, TFTP, and so many more.

Suppose you want to get the HTTP headers of a website using the curl command. To do this, you would need to invoke the curl command with the -I flag followed by the URL as shown.

$ curl -I "https://linuxways.net"

Now, you have just sent a single curl request. Suppose you want to execute 5 parallel requests; how would you go about this? The solution to this is to use the xargs command as shown alongside the curl command.

$ xargs -I % -P 5 curl -I "https://linuxways.net" < <(printf '%s\n' 1… 10)

The -P flag denotes the number of requests in parallel. The section <(printf '%s\n' 1… 10) prints out the numbers 1 - 10 and causes the curl command to run 10 times with 5 requests running in parallel.

Additionally, you can run the command below to achieve the same result. As with the previous example, the command below executes the curl command 10 times with 5 requests in parallel.

$ seq 1 10 | xargs -n1 -P 5 curl -I "https://linuxways.net"

Let's take another example. Suppose you want to download 3 files concurrently using the curl command. The first step will be to save the URLs of the files from being downloaded on a text file. Here, we have a text file download.txt with the following URLs.

To download the files concurrently, run the following command:

$ xargs -P 5 -n 1 curl -O < download.txt

The command curls every line in the text file 5 times in parallel. The -n 1 flag ensures that the curl command reads one line for every execution of the curl command.

And that's how you execute multiple curl requests in parallel.