Working with the REST API

Working with the REST APIs

Sometimes I need a zero-install way to interact with Azure. I have no specific Azure utilities at hand, no Python, no nothing. Usually, Azure management is done using PowerShell, the az cli or, if you want raw REST calls, the armclient. But for my customer, even can be too much ceremony.

So the question was how can I get going with purely bash, cURL and jq for JSON parsing, and potentially yq and xq for YAML/XML parsing.

#!/bin/bash

# Proper install
sudo apt-get -y install jq
sudo pip install yq

# YOLO
curl \
   --silent \
   --url https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
   --location \
   --output ./jq

chmod +x ./jq
sudo mv ./jq /usr/local/bin
sudo chown root.root /usr/local/bin/jq

If you're running inside a VM, with Managed Identity enabled, you can easily fetch a token. But unfortunately the VM wasn't authorized to hit the resource I care about.

Next stop service principals. Problem is customer's AD admin team running a tough regime, and don't hand out service principals.

So ultimately, how can I get my actual AAD user identity avail in the shell? In the end, all I need is a bearer token.

Let's dive right in:

A few variables first

I want to authN against 'my' Azure AD tenant, and want to hit the Azure ARM REST API.

For the full user login, i.e. device authN, here's what happens under the hood: The code needs to fetch a device code, and then use that code to poll and validate whether the user authenticated.

If you wanna snoop on cURL's requests with something like fiddler, you should add this --proxy http://127.0.0.1:8888/ --insecure to the calls.

#!/bin/bash

# --proxy http://127.0.0.1:8888/ --insecure \

aadTenant="chgeuerfte.onmicrosoft.com"

# resource="https://management.azure.com/.default"
resource="https://storage.azure.com/.default"
 
deviceResponse="$( curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/devicecode" \
    --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \
    --data-urlencode "scope=${resource}" \
    )"

device_code="$(echo "${deviceResponse}" | jq -r ".device_code")"
sleep_duration="$(echo "${deviceResponse}" | jq -r ".interval")"
access_token=""

#
# On WSL, copy code to Windows clipboard and launch the site
#
echo "$( echo "${deviceResponse}" | jq -r ".user_code" )" | iconv -f utf-8 -t utf-16le | clip.exe
cmd.exe /C "start $( echo "${deviceResponse}" | jq -r ".verification_uri" )"

#
# Poll for result
#
while [ "${access_token}" == "" ]
do
    tokenResponse="$( curl \
        --silent \
        --request POST \
        --url "https://login.microsoftonline.com/{aadTenant}/oauth2/v2.0/token" \
        --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:device_code" \
        --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \
        --data-urlencode "device_code=${device_code}" \
        )"

    if [ "$(echo "${tokenResponse}" | jq -r ".error")" == "authorization_pending" ]; then
      echo "$(echo "${deviceResponse}" | jq -r ".message")"
      sleep "${sleep_duration}"
    else
      access_token="$(echo "${tokenResponse}" | jq -r ".access_token")"
      echo "User authenticated"
    fi
done

echo "${access_token}"

echo "$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${access_token}" )" | jq

cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )"

Using a service principal (AAD v1)

Assuming we have a 'real' service principal, we can do this:

#!/bin/bash

aadTenant="chgeuerfte.onmicrosoft.com"
SAMPLE_SP_APPID="*** put your service principal application ID here ***"
SAMPLE_SP_KEY="***   put your service principal application secret here ***"

# resource="https://management.azure.com/"
resource="https://storage.azure.com/"
access_token="$(curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/token" \
    --data-urlencode "grant_type=client_credentials" \
    --data-urlencode "client_id=${SAMPLE_SP_APPID}" \
    --data-urlencode "client_secret=${SAMPLE_SP_KEY}" \
    --data-urlencode "resource=${resource}" \
    | jq -r ".access_token")"


resource="https://storage.azure.com/.default"
access_token="$(curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/token" \
    --data-urlencode "response_type=token" \
    --data-urlencode "grant_type=client_credentials" \
    --data-urlencode "client_id=${SAMPLE_SP_APPID}" \
    --data-urlencode "client_secret=${SAMPLE_SP_KEY}" \
    --data-urlencode "scope=${resource}" \
    | jq -r ".access_token")"

Create an AAD app with a specified password

#!/bin/bash

aadTenant="xxx.onmicrosoft.com"

display_name="something"

client_secret="secret123.-"

client_id="$( az ad app create --display-name "${display_name}" --password "${client_secret}" | jq -r ".appId" )"

echo "client_id: ${client_id}"

resource="https://storage.azure.com/"

access_token="$(curl \
    --silent \
    --request POST \
    --url "https://login.microsoftonline.com/${aadTenant}/oauth2/token" \
    --data-urlencode "grant_type=client_credentials" \
    --data-urlencode "client_id=${client_id}" \
    --data-urlencode "client_secret=${client_secret}" \
    --data-urlencode "resource=${resource}" \
    | jq -r ".access_token")"

cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )"

The underlying GraphAPI call for creating an app with a given password

POST https://graph.windows.net/${aadTenant}/applications?api-version=1.6 HTTP/1.1
Authorization: Bearer eyJ0eXA...
Content-Type: application/json; charset=utf-8
 
{
     "displayName": "${display_name}",
     "availableToOtherTenants": false, 
     "passwordCredentials": [
         {
           "keyId":     "8b03e38a-9e92-4c35-beb0-04a40252722d",
           "startDate": "2020-11-16T13:40:38.834354Z",
           "endDate":   "2021-11-15T13:40:38.834354Z", 
           "value":     "${client_secret}"
         }
     ]
}

Even though this says that Adding passwordCredential when creating applications is not supported., and the sample shows an empty "passwordCredentials": [] array, the call to az ad app create --display-name "${display_name}" --password "${client_secret}" exactly populates that property.

application: addPassword

Using managed VM identity (running inside an Azure VM) (AAD v1)

#!/bin/bash

resource="https://management.azure.com/"
#resource="https://storage.azure.com/"

access_token="$( curl --silent --get \
    --url "http://169.254.169.254/metadata/identity/oauth2/token" \
    --data-urlencode "api-version=2018-02-01" \
    --data-urlencode "resource=${resource}" \
    --header "Metadata: true" \
    | jq -r '.access_token' \
    )"

Fetch the subscription ID, from the Azure VM's instance metadata endpoint

#!/bin/bash

subscriptionId="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=2017-08-01" \
    --header "Metadata: true" \
    | jq -r ".compute.subscriptionId")"

Invoke the ARM API, for example with a listing of resource groups

#!/bin/bash

subscriptionId="724467b5-bee4-484b-bf13-d6a5505d2b51"

# --proxy http://127.0.0.1:8888/ --insecure \

curl --silent --get \
    --url "https://management.azure.com/subscriptions/${subscriptionId}/resourcegroups" \
    --data-urlencode "api-version=2018-05-01" 
    --header "Authorization: Bearer ${access_token}" | \
    jq -r ".value[].name"

Fetching a secret from Azure KeyVault using a managed identity

This little script demonstrates how to fetch a secret from an Azure KeyVault, using a managed identity on an Azure VM. Just adapt key_vault_name and secret_name accordingly, and of course ensure that the managed identity can actually read the secret.

#!/bin/bash

get_secret_from_keyvault() {
   local key_vault_name=${1}
   local secret_name=${2}

   resource="https://vault.azure.net"
   access_token="$( curl --silent --get \
      --url "http://169.254.169.254/metadata/identity/oauth2/token" \
      --data-urlencode "api-version=2018-02-01" \
      --data-urlencode "bypass_cache=true" \
      --data-urlencode "resource=${resource}" \
      --header "Metadata: true" \
      | jq -r '.access_token' \
      )"

   apiVersion="7.0"

   #
   # Fetch the latest version
   #
   secretVersion="$(curl --silent --get \
      --url "https://${key_vault_name}.vault.azure.net/secrets/${secret_name}/versions" \
      --data-urlencode "api-version=${apiVersion}" \
      --header "Authorization: Bearer ${access_token}" \
      | jq -r '.value | sort_by(.attributes.created) | .[-1].id' \
      )"

   #
   # Fetch the actual secret's value
   #
   secret="$( curl --silent \
      --url "${secretVersion}?api-version=${apiVersion}" \
      --header "Authorization: Bearer ${access_token}" \
      | jq -r '.value' )"

   echo "${secret}"
}

echo "The secret is $(get_secret_from_keyvault "chgeuerkeyvault" "secret1")"

Force the instance metadata service to skip the token cache

Use the bypass_cache=true parameter when fetching a token from IMDS.

Shutdown a VM, quite radically (skip graceful shutdown, just turn it off)

The skipShutdown=true below is useful in STONITH scenarios.

#!/bin/bash

...
subscriptionId="..."
resourceGroup="myrg"
vmName="somevm"

curl --silent --include \
  --request POST \
  --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/powerOff" \
  --data-urlencode "api-version=2019-03-01" \
  --data-urlencode "skipShutdown=true"
  --header "Authorization: Bearer ${access_token}" \
  --header "Content-Length: 0"

Talking to Azure Blob Storage

#!/bin/bash

json_identity="$( \
    curl --silent \
        --url "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F" \
        --header Metadata:true \
    | jq -r ".access_token")"


storage_account="tmp1diag889"
host="hybris"
storageApi="2019-12-12"

#
# Download file
#
curl \
    --url "https://${storage_account}.blob.core.windows.net/${host}/${file}" \
    --header "Authorization: Bearer ${json_identity}" \
    --header "x-ms-version: ${storageApi}" \
    --header "x-ms-blob-type: BlockBlob" \
    --remote-name

#
# List Container
#
# Unfortunately, the REST API returns XML, so 'jq' alone isn't helpful, need to XPath into XML here
#
#
curl \
    --silent \
    --url "https://${storage_account}.blob.core.windows.net/${host}/?comp=list&restype=container" \
    --header "Authorization: Bearer ${json_identity}" \
    --header "x-ms-version: ${storageApi}" \
    | xq '.EnumerationResults.Blobs[]' \
    | jq -r '.[] | .Name'

#
# An alternative approach to process the XML response, but ...
#
# XML processing with Regexes guarantees us a place in hell, but we don't need 'pip install yq'
#
curl \
    --silent \
    --url "https://${storage_account}.blob.core.windows.net/${host}/?comp=list&restype=container" \
    --header "Authorization: Bearer ${json_identity}" \
    --header "x-ms-version: ${storageApi}" \
    | sed -e 's|<Name>|\n<Name>|g' -e 's|</Name>|</Name>\n|g' \
    | egrep "^<Name>" \
    | sed -e  's|<Name>||g' -e  's|</Name>||g'

Uploading a blob

filename="1.txt"
curl \
    --request PUT \
    --url "https://${storageAccountName}.blob.core.windows.net/${containerName}/${filename}" \
    --header "x-ms-version: 2019-12-12" \
    --header "x-ms-blob-type: BlockBlob"\
    --header "x-ms-blob-content-disposition: attachment; filename=\"${filename}\"" \
    --header "Content-Type: application/binary" \
    --header "Authorization: Bearer ${access_token}" \
    --header "Content-MD5: $( md5sum "${filename}" | awk '{ print $1 }' | xxd -r -p | base64 )" \
    --upload-file "${filename}"

Commit suicide using managed identity

#!/bin/bash

resource="https://management.azure.com/"

msiVersion="2018-02-01"

access_token="$( curl --silent --get \
    --url "http://169.254.169.254/metadata/identity/oauth2/token" \
    --data-urlencode "api-version=${msiVersion}" \
    --data-urlencode "resource=${resource}" \
    --header "Metadata: true" \
    | jq -r '.access_token' \
    )"

imdsVersion="2021-02-01"

subscriptionId="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=${imdsVersion}" \
    --header "Metadata: true" \
    | jq -r '.compute.subscriptionId' \
    )"

resourceGroup="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=${imdsVersion}" \
    --header "Metadata: true" \
    | jq -r '.compute.resourceGroupName' \
    )"

vmName="$(curl --silent --get \
    --url "http://169.254.169.254/metadata/instance" \
    --data-urlencode "api-version=${imdsVersion}" \
    --header "Metadata: true" \
    | jq -r '.compute.name' \
    )"

#
# Stop and skip shutdown sequence. STONITH
#

virtualMachineARMVersion="2021-03-01"

curl --silent \
  --request POST \
  --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/powerOff?api-version=${virtualMachineARMVersion}&skipShutdown=true" \
  --header "Authorization: Bearer ${access_token}" \
  --data ""

#
# Properly deallocate
#
curl --silent \
  --request POST \
  --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/deallocate?api-version=${virtualMachineARMVersion}" \
  --header "Authorization: Bearer ${access_token}" \
  --data ""

PreviousLogging in to Azure NextTracing HTTP requests with Fiddler

Last updated 1 year ago

A few variables first

Doing a device login (AAD v2)

Using a service principal (AAD v1)

Create an AAD app with a specified password

The underlying GraphAPI call for creating an app with a given password

Using managed VM identity (running inside an Azure VM) (AAD v1)

Fetch the subscription ID, from the Azure VM's instance metadata endpoint

Invoke the ARM API, for example with a listing of resource groups

Fetching a secret from Azure KeyVault using a managed identity

Force the instance metadata service to skip the token cache

Shutdown a VM, quite radically (skip graceful shutdown, just turn it off)

Talking to Azure Blob Storage

Uploading a blob

Commit suicide using managed identity