cURL command line utility

curl recipes

Determine HTTP status (200, 404, etc.) with `--write-out '%{http_code}'`

health-probe.sh

#!/bin/bash

function httpStatus { echo "$( curl --silent --output /dev/null --write-out '%{http_code}' $1 )" ; }

echo "Azure: $( httpStatus "https://portal.azure.com/" )"

results in Azure: 200.

Send some XML via POST to a URL

Read body to upload from STDIN (via `--data @-`)

POST-XML.sh

#!/bin/bash

cat payload.xml | curl \
  --silent --include \
  --request POST \
  --url "https://localhost/cgi-bin/postsomestuff" \
  --header "Content-Type: application/xml" \
  --data @-

Read body to upload from file

POST-XML.sh

#!/bin/bash

curl \
  --silent --include \
  --request POST \
  --url "https://localhost/cgi-bin/postsomestuff" \
  --header "Content-Type: application/xml" \
  --data @payload.xml

Tracing your cURL calls with Fiddler

When I want to completely see the traffic originating from my cURL instance, I use Fiddler (a Windows-based HTTP(s)-proxy GUI). Fiddler can be configured to decrypt TLS (https://) traffic, but that means that the server certificate for cURL will be untrusted. The following args instruct cURL to use a local (untrusted) HTTPs-proxy:

curl-use-fiddler.sh

#!/bin/bash

curl \
   --get \
   --url "https://management.azure.com/" \
   --proxy http://127.0.0.1:8888/ --insecure

Extract both custom HTTP header values and the body from a request

The following script uses cURL to fetch a web page, and then extracts both an HTTP from the response headers, as well as the body.

This script uses awk, which creates temporary files to store header and body part.

#!/bin/bash

function extractHeaders {
  local curlResponse="$1"

  local tempF="$( mktemp )"
  local tH="${tempF}headers"
  local tB="/dev/null"

  echo "${response}" | awk -v bl=1 "$( printf 'bl{bl=0; h=($0 ~ /HTTP\/1/)} /^\r?$/{bl=1} {print $0>(h?"%s":"%s")}' $tH $tB )"

  headerContents="$( cat "${tH}" ; rm "${tH}" )"

  echo "${headerContents}"
}

function extractBody {
  local curlResponse="$1"

  local tempF="$( mktemp )"
  local tH="/dev/null"
  local tB="${tempF}body"

  echo "${response}" | awk -v bl=1 "$( printf 'bl{bl=0; h=($0 ~ /HTTP\/1/)} /^\r?$/{bl=1} {print $0>(h?"%s":"%s")}' $tH $tB )"

  bodyContents="$( cat "${tB}" ; rm "${tB}" )"

  echo "${bodyContents}"
}

someURL="https://www.google.com/"

response="$( curl \
  --silent --include \
  --request GET --url "${someURL}" )"

body="$( extractBody "${response}" )"

headers="$( extractHeaders "${response}" )"
someHeader="Content-Type"
someHeaderValue="$( echo "${headers}" | grep "^${someHeader}:" | sed -E 's/^(\S+?): (.+)/\2/' )"

echo "The ${someHeader} was ${someHeaderValue}"
echo "The body was ${body}"

Links

Previousbash scripting Nextffmpeg - Processing Media

Last updated 2 years ago

Determine HTTP status (200, 404, etc.) with --write-out '%{http_code}'