Working with the REST API

Working with the REST APIs
Sometimes I need a zero-install way to interact with Azure. I have no specific Azure utilities at hand, no Python, no nothing. Usually, Azure management is done using PowerShell, the az cli or, if you want raw REST calls, the armclient. But for my customer, even can be too much ceremony.
So the question was how can I get going with purely bash, cURL and jq for JSON parsing, and potentially yq and xq for YAML/XML parsing.
#!/bin/bash
​
# Proper install
sudo apt-get -y install jq
sudo pip install yq
​
# YOLO
curl \
   --silent \
   --url https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
   --location \
   --output ./jq
​
chmod +x ./jq
sudo mv ./jq /usr/local/bin
sudo chown root.root /usr/local/bin/jq
​
If you're running inside a VM, with Managed Identity enabled, you can easily fetch a token. But unfortunately the VM wasn't authorized to hit the resource I care about.
Next stop service principals. Problem is customer's AD admin team running a tough regime, and don't hand out service principals.
So ultimately, how can I get my actual AAD user identity avail in the shell? In the end, all I need is a bearer token.
Let's dive right in:
A few variables first
I want to authN against 'my' Azure AD tenant, and want to hit the Azure ARM REST API.
Doing a device login (AAD v2)
For the full user login, i.e. device authN, here's what happens under the hood: The code needs to fetch a device code, and then use that code to poll and validate whether the user authenticated.
If you wanna snoop on cURL's requests with something like fiddler, you should add this --proxy http://127.0.0.1:8888/ --insecure to the calls.
#!/bin/bash
​
# --proxy http://127.0.0.1:8888/ --insecure \
​
aadTenant="chgeuerfte.onmicrosoft.com"
​
# resource="https://management.azure.com/.default"
resource="https://storage.azure.com/.default"
 
deviceResponse="$( curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/devicecode" \
    --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \
    --data-urlencode "scope=${resource}" \
    )"
​
device_code="$(echo "${deviceResponse}" | jq -r ".device_code")"
sleep_duration="$(echo "${deviceResponse}" | jq -r ".interval")"
access_token=""
​
#
# On WSL, copy code to Windows clipboard and launch the site
#
echo "$( echo "${deviceResponse}" | jq -r ".user_code" )" | iconv -f utf-8 -t utf-16le | clip.exe
cmd.exe /C "start $( echo "${deviceResponse}" | jq -r ".verification_uri" )"
​
#
# Poll for result
#
while [ "${access_token}" == "" ]
do
    tokenResponse="$( curl \
        --silent \
        --request POST \
        --url "https://login.microsoftonline.com/{aadTenant}/oauth2/v2.0/token" \
        --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:device_code" \
        --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \
        --data-urlencode "device_code=${device_code}" \
        )"
​
    if [ "$(echo "${tokenResponse}" | jq -r ".error")" == "authorization_pending" ]; then
      echo "$(echo "${deviceResponse}" | jq -r ".message")"
      sleep "${sleep_duration}"
    else
      access_token="$(echo "${tokenResponse}" | jq -r ".access_token")"
      echo "User authenticated"
    fi
done
​
echo "${access_token}"
​
echo "$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${access_token}" )" | jq
​
cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )"
Using a service principal (AAD v1)
Assuming we have a 'real' service principal, we can do this:
#!/bin/bash
​
aadTenant="chgeuerfte.onmicrosoft.com"
SAMPLE_SP_APPID="*** put your service principal application ID here ***"
SAMPLE_SP_KEY="***   put your service principal application secret here ***"
​
# resource="https://management.azure.com/"
resource="https://storage.azure.com/"
access_token="$(curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/token" \
    --data-urlencode "grant_type=client_credentials" \
    --data-urlencode "client_id=${SAMPLE_SP_APPID}" \
    --data-urlencode "client_secret=${SAMPLE_SP_KEY}" \
    --data-urlencode "resource=${resource}" \
    | jq -r ".access_token")"
​
​
resource="https://storage.azure.com/.default"
access_token="$(curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/token" \
    --data-urlencode "response_type=token" \
    --data-urlencode "grant_type=client_credentials" \
    --data-urlencode "client_id=${SAMPLE_SP_APPID}" \
    --data-urlencode "client_secret=${SAMPLE_SP_KEY}" \
    --data-urlencode "scope=${resource}" \
    | jq -r ".access_token")"
Create an AAD app with a specified password
#!/bin/bash
​
aadTenant="xxx.onmicrosoft.com"
​
display_name="something"
​
client_secret="secret123.-"
​
client_id="$( az ad app create --display-name "${display_name}" --password "${client_secret}" | jq -r ".appId" )"
​
echo "client_id: ${client_id}"
​
resource="https://storage.azure.com/"
​
access_token="$(curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/token" \
    --data-urlencode "grant_type=client_credentials" \
    --data-urlencode "client_id=${client_id}" \
    --data-urlencode "client_secret=${client_secret}" \
    --data-urlencode "resource=${resource}" \
    | jq -r ".access_token")"
​
cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )"
The underlying GraphAPI call for creating an app with a given password
POST https://graph.windows.net/${aadTenant}/applications?api-version=1.6 HTTP/1.1
Authorization: Bearer eyJ0eXA...
Content-Type: application/json; charset=utf-8
 
{
     "displayName": "${display_name}",
     "availableToOtherTenants": false, 
     "passwordCredentials": [
         {
           "keyId":     "8b03e38a-9e92-4c35-beb0-04a40252722d",
           "startDate": "2020-11-16T13:40:38.834354Z",
           "endDate":   "2021-11-15T13:40:38.834354Z", 
           "value":     "${client_secret}"
         }
     ]
}
Even though this says that Adding passwordCredential when creating applications is not supported., and the sample shows an empty "passwordCredentials": [] array, the call to az ad app create --display-name "${display_name}" --password "${client_secret}" exactly populates that property.
​application: addPassword​
Using managed VM identity (running inside an Azure VM) (AAD v1)
#!/bin/bash
​
resource="https://management.azure.com/"
#resource="https://storage.azure.com/"
​
access_token="$( curl --silent --get \
    --url "http://169.254.169.254/metadata/identity/oauth2/token" \
    --data-urlencode "api-version=2018-02-01" \
    --data-urlencode "resource=${resource}" \
    --header "Metadata: true" \
    | jq -r '.access_token' \
    )"
Fetch the subscription ID, from the Azure VM's instance metadata endpoint
#!/bin/bash
​
subscriptionId="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=2017-08-01" \
    --header "Metadata: true" \
    | jq -r ".compute.subscriptionId")"
Invoke the ARM API, for example with a listing of resource groups
#!/bin/bash
​
subscriptionId="724467b5-bee4-484b-bf13-d6a5505d2b51"
​
# --proxy http://127.0.0.1:8888/ --insecure \
​
curl --silent --get \
    --url "https://management.azure.com/subscriptions/${subscriptionId}/resourcegroups" \
    --data-urlencode "api-version=2018-05-01" 
    --header "Authorization: Bearer ${access_token}" | \
    jq -r ".value[].name"
Fetching a secret from Azure KeyVault using a managed identity
This little script demonstrates how to fetch a secret from an Azure KeyVault, using a managed identity on an Azure VM. Just adapt key_vault_name and secret_name accordingly, and of course ensure that the managed identity can actually read the secret.
#!/bin/bash
​
get_secret_from_keyvault() {
   local key_vault_name=${1}
   local secret_name=${2}
​
   resource="https://vault.azure.net"
   access_token="$( curl --silent --get \
      --url "http://169.254.169.254/metadata/identity/oauth2/token" \
      --data-urlencode "api-version=2018-02-01" \
      --data-urlencode "bypass_cache=true" \
      --data-urlencode "resource=${resource}" \
      --header "Metadata: true" \
      | jq -r '.access_token' \
      )"
​
   apiVersion="7.0"
​
   #
   # Fetch the latest version
   #
   secretVersion="$(curl --silent --get \
      --url "https://${key_vault_name}.vault.azure.net/secrets/${secret_name}/versions" \
      --data-urlencode "api-version=${apiVersion}" \
      --header "Authorization: Bearer ${access_token}" \
      | jq -r '.value | sort_by(.attributes.created) | .[-1].id' \
      )"
​
   #
   # Fetch the actual secret's value
   #
   secret="$( curl --silent \
      --url "${secretVersion}?api-version=${apiVersion}" \
      --header "Authorization: Bearer ${access_token}" \
      | jq -r '.value' )"
​
   echo "${secret}"
}
​
echo "The secret is $(get_secret_from_keyvault "chgeuerkeyvault" "secret1")"
Force the instance metadata service to skip the token cache
Use the bypass_cache=true parameter when fetching a token from IMDS.
Shutdown a VM, quite radically (skip graceful shutdown, just turn it off)
The skipShutdown=true below is useful in STONITH scenarios.
#!/bin/bash
​
...
subscriptionId="..."
resourceGroup="myrg"
vmName="somevm"
​
curl --silent --include \
  --request POST \
  --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/powerOff" \
  --data-urlencode "api-version=2019-03-01" \
  --data-urlencode "skipShutdown=true"
  --header "Authorization: Bearer ${access_token}" \
  --header "Content-Length: 0"
Talking to Azure Blob Storage
#!/bin/bash
​
json_identity="$( \
    curl --silent \
        --url "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F" \
        --header Metadata:true \
    | jq -r ".access_token")"
​
​
storage_account="tmp1diag889"
host="hybris"
storageApi="2019-12-12"
​
#
# Download file
#
curl \
    --url "https://${storage_account}.blob.core.windows.net/${host}/${file}" \
    --header "Authorization: Bearer ${json_identity}" \
    --header "x-ms-version: ${storageApi}" \
    --header "x-ms-blob-type: BlockBlob" \
    --remote-name
​
#
# List Container
#
# Unfortunately, the REST API returns XML, so 'jq' alone isn't helpful, need to XPath into XML here
#
#
curl \
    --silent \
    --url "https://${storage_account}.blob.core.windows.net/${host}/?comp=list&restype=container" \
    --header "Authorization: Bearer ${json_identity}" \
    --header "x-ms-version: ${storageApi}" \
    | xq '.EnumerationResults.Blobs[]' \
    | jq -r '.[] | .Name'
​
#
# An alternative approach to process the XML response, but ...
#
# XML processing with Regexes guarantees us a place in hell, but we don't need 'pip install yq'
#
curl \
    --silent \
    --url "https://${storage_account}.blob.core.windows.net/${host}/?comp=list&restype=container" \
    --header "Authorization: Bearer ${json_identity}" \
    --header "x-ms-version: ${storageApi}" \
    | sed -e 's|<Name>|\n<Name>|g' -e 's|</Name>|</Name>\n|g' \
    | egrep "^<Name>" \
    | sed -e  's|<Name>||g' -e  's|</Name>||g'
Uploading a blob
filename="1.txt"
curl \
    --request PUT \
    --url "https://${storageAccountName}.blob.core.windows.net/${containerName}/${filename}" \
    --header "x-ms-version: 2019-12-12" \
    --header "x-ms-blob-type: BlockBlob"\
    --header "x-ms-blob-content-disposition: attachment; filename=\"${filename}\"" \
    --header "Content-Type: application/binary" \
    --header "Authorization: Bearer ${access_token}" \
    --header "Content-MD5: $( md5sum "${filename}" | awk '{ print $1 }' | xxd -r -p | base64 )" \
    --upload-file "${filename}"
Commit suicide using managed identity
#!/bin/bash
​
resource="https://management.azure.com/"
​
msiVersion="2018-02-01"
​
access_token="$( curl --silent --get \
    --url "http://169.254.169.254/metadata/identity/oauth2/token" \
    --data-urlencode "api-version=${msiVersion}" \
    --data-urlencode "resource=${resource}" \
    --header "Metadata: true" \
    | jq -r '.access_token' \
    )"
​
imdsVersion="2021-02-01"
​
subscriptionId="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=${imdsVersion}" \
    --header "Metadata: true" \
    | jq -r '.compute.subscriptionId' \
    )"
​
resourceGroup="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=${imdsVersion}" \
    --header "Metadata: true" \
    | jq -r '.compute.resourceGroupName' \
    )"
​
vmName="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=${imdsVersion}" \
    --header "Metadata: true" \
    | jq -r '.compute.name' \
    )"
​
#
# Stop and skip shutdown sequence. STONITH
#
​
virtualMachineARMVersion="2021-03-01"
​
curl --silent \
  --request POST \
  --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/powerOff?api-version=${virtualMachineARMVersion}&skipShutdown=true" \
  --header "Authorization: Bearer ${access_token}" \
  --data ""
​
#
# Properly deallocate
#
curl --silent \
  --request POST \
  --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/deallocate?api-version=${virtualMachineARMVersion}" \
  --header "Authorization: Bearer ${access_token}" \
  --data ""
Azure - Previous
Logging in to Azure
Next - Azure
Tracing HTTP requests with Fiddler
Last modified 3mo ago