Tracing HTTP requests with Fiddler

Tracing HTTP requests with Fiddler

When using the az CLI, it sometimes is helpful to understand which APIs it calls 'under the hood', i.e. so see which Azure REST APIs are called. On Windows, you can use a tool like Fiddler to inspect outgoing HTTP(s) requests. Fiddler does this by 'launching a man-in-the-middle attack' against the applications, by injecting a self-signed X.509 certificate into the Windows cert store, and then pretending to be the external web site.

However, different applications read the HTTP/HTTPS proxy information from different locations:

• In .NET, one can set a global (static) System.Net.WebRequest.DefaultWebProxy variable

• Windows applications (like web browsers) check the Windows registry, in particular the HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyServer values

• Unix-style applications (like the Python-based az utility) check the HTTP_PROXY and HTTPS_PROXY environment variables

• Other applications require you to explicitly specify the proxy to use via command line, such as curl supporting the --proxy ... and --insecure args.

On the security side, you need to convince the apps to accept Fiddler's self-cooked TLS cert:

• Windows apps check the X509 cert chain, using the MACHINE\root store

• curl can be convinced using the --insecure arg to ignore certificate validity problems

• The az CLI needs the ADAL_PYTHON_SSL_NO_VERIFY and AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variables to be set, to skip checking the server's TLS cert.

Setting the proxy in Powershell

Here's how to set a gazillion different settings to make sure Fiddler is used.

$fiddlerHost = "127.0.0.1"
$fiddlerPort = "8888"
$fiddlerUrl = "http://$($fiddlerHost):$($fiddlerPort)" 

#
# This ensures the .NET code uses the proxy
#
[System.Net.WebRequest]::DefaultWebProxy = New-Object System.Net.WebProxy ( New-Object System.Uri( $fiddlerUrl ), $true)

#
# This ensures Windows apps (Edge, Teams, Outlook, Windows) use the proxy
#
Set-ItemProperty `
   -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" `
   -Name ProxyServer `
   -Value "http=$($fiddlerHost):$($fiddlerPort);https=$($fiddlerHost):$($fiddlerPort)"
Set-ItemProperty `
   -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" `
   -Name ProxyEnable `
   -Value 1

#
# This ensures Python code (which looks at environment variables) uses the proxy
#
$Env:HTTP_PROXY = $fiddlerUrl 
$Env:HTTPS_PROXY = $fiddlerUrl 

#
# This ensures the `az` CLI doesn't complain when we launch a man-in-the-middle with 
# a self-issued X509 cert
#
$Env:ADAL_PYTHON_SSL_NO_VERIFY = '1'
$Env:AZURE_CLI_DISABLE_CONNECTION_VERIFICATION = '1'

#
# And the `--insecure` also calms curl's desire to be secure
#
C:\Users\chgeuer\bin\curl.exe --proxy $fiddlerUrl --insecure `
    --silent `
    "https://www.microsoft.com"

Checking the current settings

$(Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings").ProxyServer

$(Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings").ProxyEnable

Deleting the registry entry again

Remove-ItemProperty `
   -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" `
   -Name ProxyServer

Set-ItemProperty `
   -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" `
   -Name ProxyEnable -Value 0