# Introduction This site is a collection of various tech tips, which I compiled so I don't need to google/bing for my own solutions any longer. If something's wrong, feel free to send me a pull request in [GitHub](https://github.com/chgeuer/gitbook-chgeuertips), or ping me on [Twitter](https://twitter.com/chgeuer/). # bash scripting Bash recipes * Leverage [ShellCheck.net](https://www.shellcheck.net/) and the [VS Code Addin](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck) ## bash scripting ![bash logo](/files/ycd87RsNsNtk3mUZHhQ3) ### Proper escaping - Everything is a string #### Accessing values {% code title="everything-is-a-string.sh" %} ```bash #!/bin/bash a="some string" # this works, but I don't like it: echo $a # This is how I do it. Curly braces for string values: echo "${a}" ``` {% endcode %} #### Executing a command {% code title="run-a-command.sh" %} ```bash #!/bin/bash # Running a command with backticks, but I don't like that: fileContent=`cat 1.txt` # Running a command with $( xxx ) fileContent="$( cat 1.txt )" fileContent="$( echo "Hi" > 1.txt ; cat "./1.txt" ; rm ./1.txt )" ``` {% endcode %} ### Defining variables #### Creating and accessing an array in bash {% code title="array.sh" %} ```bash #!/bin/bash declare releaseNames=( "api-poi" "api-trip" "api-user" "api-user-java" ) echo "The array contains ${#releaseNames[@]} elements:" for releaseName in "${releaseNames[@]}"; do echo " - ${releaseName}" done ``` {% endcode %} #### Creating and accessing a hashmap / dict bash ```bash declare -A helmValues helmValues["a"]="val a" helmValues["b"]="val b" echo "a: '${helmValues["a"]}'" echo "b: '${helmValues["b"]}'" ``` ### Determine the directory of the script {% code title="show:dir.sh" %} ```bash #!/bin/bash d="$( dirname "$( readlink -f "$0" )" )" echo "Running in directory ${d}" ``` {% endcode %} results in ``` $ pwd /mnt/c/github/chgeuer/tips $ ./show_dir.sh Running in directory /mnt/c/github/chgeuer/tips ``` ### Span a command across multiple lines Use a baskslash (`\`) at the end of the line (no additional whitespace), and preferably indent the next line: {% code title="multi-line.sh" %} ```bash #!/bin/bash response="$( cat payload.xml | curl \ --silent --include \ --request POST \ --url "${triggerURI}" \ --header "Content-Type: application/xml" \ --data @- )" echo "${response}" ``` {% endcode %} ### Lambda-style functions in bash {% code title="health-probe.sh" %} ```bash #!/bin/bash # # Define some function # function httpStatus { local url="$1" echo "$( curl \ --silent \ --output /dev/null \ --write-out '%{http_code}' \ "${url}" )" } function httpStatus2 { echo "$( curl --silent --output /dev/null --write-out '%{http_code}' $1 )" ; } echo "Azure: $( httpStatus "https://portal.azure.com/" )" echo "Homepage: $( httpStatus2 "https://www.microsoft.com/" )" ``` {% endcode %} results in ```bash $ ./health-probe.sh Azure: 200 Homepage: 200 $ ``` ### Creating a text file The `cat > x <<-EOF ... EOF` syntax allows to create a file in the local directory. Please not that the lines 6 and 7 below (the content) are prefixed with a tabstop (), which does not show up in the actual text file. ```bash #!/bin/bash SERVER_IP="127.0.0.1" cat > somefile.ini <<-EOF server=${SERVER_IP} port=5093 EOF ``` ### base64-encode a text The command `base64 --wrap=0` converts input into a long base64-encoded string without line breaks. ```bash #!/bin/bash FILE_CONTENT="$( cat ./foo.bin )" ONE_LONG_BASE64_STR=$(echo "${FILE_CONTENT}" | base64 --wrap=0) ``` ### Bash history Put the following lines in `~/.inputrc`: ``` ## arrow up "\e[A":history-search-backward ## arrow down "\e[B":history-search-forward ``` # cURL command line utility curl recipes ![curl logo](/files/izumniWANkww5o23wYV2) ## Determine HTTP status (200, 404, etc.) with `--write-out '%{http_code}'` {% code title="health-probe.sh" %} ```bash #!/bin/bash function httpStatus { echo "$( curl --silent --output /dev/null --write-out '%{http_code}' $1 )" ; } echo "Azure: $( httpStatus "https://portal.azure.com/" )" ``` {% endcode %} results in `Azure: 200`. ## Send some XML via POST to a URL ### Read body to upload from STDIN (via `--data @-`) {% code title="POST-XML.sh" %} ```bash #!/bin/bash cat payload.xml | curl \ --silent --include \ --request POST \ --url "https://localhost/cgi-bin/postsomestuff" \ --header "Content-Type: application/xml" \ --data @- ``` {% endcode %} ### Read body to upload from file {% code title="POST-XML.sh" %} ```bash #!/bin/bash curl \ --silent --include \ --request POST \ --url "https://localhost/cgi-bin/postsomestuff" \ --header "Content-Type: application/xml" \ --data @payload.xml ``` {% endcode %} ## Tracing your cURL calls with Fiddler When I want to completely see the traffic originating from my cURL instance, I use [Fiddler ](https://www.telerik.com/fiddler)(a Windows-based HTTP(s)-proxy GUI). Fiddler can be configured to decrypt TLS (https\://) traffic, but that means that the server certificate for cURL will be untrusted. The following args instruct cURL to use a local (untrusted) HTTPs-proxy: {% code title="curl-use-fiddler.sh" %} ```bash #!/bin/bash curl \ --get \ --url "https://management.azure.com/" \ --proxy http://127.0.0.1:8888/ --insecure ``` {% endcode %} ![Tracing a cURL interaction in Fiddler](/files/GXGBVsSagKnzuVA7nS9I) ## Extract both custom HTTP header values and the body from a request The following script uses cURL to fetch a web page, and then extracts both an HTTP from the response headers, as well as the body. {% hint style="warning" %} This script uses `awk,` which creates temporary files to store header and body part. {% endhint %} ```bash #!/bin/bash function extractHeaders { local curlResponse="$1" local tempF="$( mktemp )" local tH="${tempF}headers" local tB="/dev/null" echo "${response}" | awk -v bl=1 "$( printf 'bl{bl=0; h=($0 ~ /HTTP\/1/)} /^\r?$/{bl=1} {print $0>(h?"%s":"%s")}' $tH $tB )" headerContents="$( cat "${tH}" ; rm "${tH}" )" echo "${headerContents}" } function extractBody { local curlResponse="$1" local tempF="$( mktemp )" local tH="/dev/null" local tB="${tempF}body" echo "${response}" | awk -v bl=1 "$( printf 'bl{bl=0; h=($0 ~ /HTTP\/1/)} /^\r?$/{bl=1} {print $0>(h?"%s":"%s")}' $tH $tB )" bodyContents="$( cat "${tB}" ; rm "${tB}" )" echo "${bodyContents}" } someURL="https://www.google.com/" response="$( curl \ --silent --include \ --request GET --url "${someURL}" )" body="$( extractBody "${response}" )" headers="$( extractHeaders "${response}" )" someHeader="Content-Type" someHeaderValue="$( echo "${headers}" | grep "^${someHeader}:" | sed -E 's/^(\S+?): (.+)/\2/' )" echo "The ${someHeader} was ${someHeaderValue}" echo "The body was ${body}" ``` ## Links * [curl.haxx.se](https://curl.haxx.se/) * [Windows download](https://curl.haxx.se/windows/) * [The book 'Everything curl'](https://bagder.gitbook.io/everything-curl/) # ffmpeg - Processing Media Describe a couple of ffmpeg recipes ![](/files/kIoAFt7w0m9j8C5ONhN1) ### Download video from the web #### HDS videos on Windows The following snippet uses a PHP Script (executed at the command line) to download an HDS manifest and assembles the video fragments: * Download [PHP for Windows](http://windows.php.net/downloads/releases/php-7.1.2-nts-Win32-VC14-x86.zip) and unpack to `C:\php` * Edit `C:\php\php.ini` and uncomment the `extension=php_curl.dll` line * ´git clone * `php.exe AdobeHDS.php --manifest "http://adaptiv.wdr.de/...mp4.csmil/manifest.f4m?g=...&hdcore=3.10.0&plugin=aasp-3.10.0.29.28" --delete` * Use `ffmpeg` to convert FLV to MP4 (`ffmpeg -i 1.flv -vcodec copy -acodec copy -map_metadata 0 1.mp4`) #### Download HLS videos ```bash ffmpeg \ -i https://...akamaihd.net/.../name/a.mp4/index.m3u8 \ -c copy -bsf:a aac_adtstoasc "foo.mp4" ``` #### Using RTMPDump to fetch an RTMP source * See also * and ``` rtmpdump \ --protocol 0 \ --host cp45414.edgefcs.net \ -a "ondemand?auth=daEa9dhbhaJd4dmc8bicPd1cJdcdzcUcwcd-btFUIl-bWG-CqsEHnBqLEpGnxK&aifp=v001&slist=public/mps_h264_med/public/news/world/1078000/1078809_h264_800k.mp4;public/mps_h264_lo/public/news/world/1078000/1078809_h264_496k.mp4;public/mps_h264_hi/public/news/world/1078000/1078809_h264_1500k.mp4" \ -y "mp4:public/mps_h264_lo/public/news/world/1078000/1078809_h264_496k.mp4" \ -o someresolution.flv ffmpeg -i someresolution.flv -c:v copy -c:a copy someresolution.mp4 rtmpdump --protocol 0 --host cp45414.edgefcs.net \ -a "ondemand?auth=daEa9dhbhaJd4dmc8bicPd1cJdcdzcUcwcd-btFUIl-bWG-CqsEHnBqLEpGnxK&aifp=v001&slist=public/mps_h264_hi/public/news/world/1078000/1078809_h264_1500k.mp4" \ -y "mp4:public/mps_h264_hi/public/news/world/1078000/1078809_h264_1500k.mp4" \ -o 1078809_h264_1500k.flv ffmpeg -i 1078809_h264_1500k.flv -c:v copy -c:a copy 1078809_h264_1500k.mp4 ``` #### Download YouTube and create an animated GIF from sub-part ```bash #!/bin/bash youtube-dl https://www.youtube.com/watch?v=bY73vFGhSVk # Trim time and crop sub-part and save as mp4 ffmpeg \ -i "Zootopia Official US Sloth Trailer-bY73vFGhSVk.mp4" \ -ss 00:01:49 -t 00:00:11.3 \ -vf "crop=480:320:600:100" \ -c:v libx264 \ -c:a aac \ -strict experimental \ -b:a 128k \ "laughing sloth.mp4" # generate color palette ffmpeg \ -i "laughing sloth.mp4" \ -y \ -vf fps=10,scale=320:-1:flags=lanczos,palettegen palette.png # Render GIF using palette ffmpeg \ -i "laughing sloth.mp4" \ -i palette.png \ -filter_complex "fps=10,scale=320:-1:flags=lanczos[x];[x][1:v]paletteuse" \ output.gif ``` ### Convert videos {% code title="convert.ps1" %} ``` dir *.webm | foreach { ffmpeg -i $_.Name -ab 192k $_.Name.Replace("WEBM", "mp3").Replace("webm", "mp3") } dir *.mkv | foreach { ffmpeg -i $_.Name -ab 192k $_.Name.Replace("mkv", "mp3") } dir *.mkv | foreach { ffmpeg -i $_.Name -vcodec copy -acodec copy -map_metadata 0 $_.Name.Replace("mkv", "mp4") } dir *.mkv | foreach { ffmpeg -i $_.Name -vcodec copy -ab 192k -map_metadata 0 $_.Name.Replace("mkv", "mp4") } ``` {% endcode %} ### Figure out which DirectShow input devices I have ``` ffmpeg -list_devices true -f dshow -i dummy ``` On my work laptop, I have an integrated web cam, a built-in microphone, and an additional head set: ``` C:\Users\chgeuer>ffmpeg -list_devices true -f dshow -i dummy ffmpeg version N-69972-g6c91afe Copyright (c) 2000-2015 the FFmpeg developers ... [dshow @ 0000000004d2d540] DirectShow video devices (some may be both video and audio devices) [dshow @ 0000000004d2d540] "Integrated Camera" [dshow @ 0000000004d2d540] DirectShow audio devices [dshow @ 0000000004d2d540] "Microphone (Realtek High Definition Audio)" [dshow @ 0000000004d2d540] "Headset Microphone (Plantronics C520-M)" ``` The strings `"Integrated Camera"`, `"Microphone (Realtek High Definition Audio)"` and `"Headset Microphone (Plantronics C520-M)"` now refer to the different usable sources. In ffmpeg, the `-i` parameter usually refers to the input file. In our case, we can now combine video & audio sources to an input specification for ffmpeg: * `-i video="Integrated Camera":audio="Headset Microphone (Plantronics C520-M)"` * `-i video="Integrated Camera":audio="Microphone (Realtek High Definition Audio)"` #### Determine the capabilities of the hardware ``` ffmpeg -f dshow -i video="Integrated Camera":audio="Microphone (Realtek High Definition Audio)" -list_formats all ``` #### Capture the local web cam & microphone and create a 5sec MP4 video ``` ffmpeg -f dshow -i video="Integrated Camera":audio="Headset Microphone (Plantronics C520-M)" -t 5 5-seconds.mp4 ``` #### Stream TS (untested) Src: ``` ffmpeg -i video="Integrated Camera":audio="Headset Microphone (Plantronics C520-M)" -y -nostdin -hide_banner -loglevel 0 -f v4l2 -framerate 25 -video_size 1280x720 -input_format mjpeg -c libx264 -movflags faststart -f mpegts - ``` #### Screen capture **Screen capture filter** For capturing the local screen, you need a driver to tap into the video card. `ffmpeg` on Windows ships with the GDI grabber [`-f gdigrab`](https://www.ffmpeg.org/ffmpeg-devices.html#gdigrab) filter. It is also possible to use a DirectShow filter (`-f dshow`), but then you need a driver. * * * [http://netcologne.dl.sourceforge.net/project/screencapturer/Setup Screen Capturer Recorder v0.12.8.exe](http://netcologne.dl.sourceforge.net/project/screencapturer/Setup%20Screen%20Capturer%20Recorder%20v0.12.8.exe) After installing the driver above, you will be able to use the ffmpeg input ``` -i video="screen-capture-recorder":audio="virtual-audio-capturer" ``` **Write a 10 second screen capture (at 20 fps) to local MP4 file** ``` ffmpeg -f dshow -i video="screen-capture-recorder":audio="virtual-audio-capturer" -r 20 -t 10 screen-capture.mp4 ffmpeg -f dshow -i video="screen-capture-recorder":audio="Headset Microphone (Plantronics C520-M)" -r 20 -t 10 screen-capture.mp4 ``` **Play back current screen** ``` ffplay -f dshow -i video="screen-capture-recorder" -vf scale=1280:720 ``` #### Capture web cam on my Lenovo ``` ffmpeg -list_devices true -f dshow -i dummy ffmpeg -f dshow -i video="Integrated Camera":audio="Microphone Array (Realtek High Definition Audio)" out.mp4 ``` ### Concatenate videos #### Convert individually to TS ```bash ffmpeg -i "m0-01 - A.mp4" -c copy -bsf:v h264_mp4toannexb -f mpegts "m0-01 - A.ts" ffmpeg -i "m1-01 - B.mp4" -c copy -bsf:v h264_mp4toannexb -f mpegts "m1-01 - B.ts" ``` #### `ts.txt` ``` ("file '" + (((dir "*.ts" | select -ExpandProperty Name) -replace "'", "\'") -join "'`nfile '") + "'") | Out-File -Encoding ascii -FilePath ts.txt ``` ``` file 'm0-01 - A.ts' file 'm1-01 - B.ts' ``` #### Concatenate ```bash ffmpeg -f concat -i ts.txt -c copy -bsf:a aac_adtstoasc output.mp4 ``` ``` dir *.mp4 | foreach { ffmpeg -i $_.Name -c copy -bsf:v h264_mp4toannexb -f mpegts $_.Name.Replace("MP4", "ts").Replace("mp4", "ts") } ("file '" + (((dir "*.ts" | select -ExpandProperty Name) -replace "'", "\'") -join "'`nfile '") + "'") | Out-File -Encoding ascii -FilePath ts.txt ffmpeg -f concat -safe 0 -i ts.txt -c copy -bsf:a aac_adtstoasc output.mp4 ``` ## Azure Media Players ## Streaming to Azure Media Services Live Streaming After creating an Azure Media Services Live channel, we get two RTMP ingest endpoints, which differ in their TCP port number (1935 and 1936): * `rtmp://channel1-mediaservice321.channel.mediaservices.windows.net:1936/live/deadbeef012345678890abcdefabcdef` * `rtmp://channel1-mediaservice321.channel.mediaservices.windows.net:1936/live/deadbeef012345678890abcdefabcdef` For ffmpeg to work, we need to append the channel name `/channel1` to the URLs: * `rtmp://channel1-mediaservice321.channel.mediaservices.windows.net:1936/live/deadbeef012345678890abcdefabcdef/channel1` * `rtmp://channel1-mediaservice321.channel.mediaservices.windows.net:1936/live/deadbeef012345678890abcdefabcdef/channel1` ### Input specs The [Azure Blog](https://azure.microsoft.com/en-us/blog/azure-media-services-rtmp-support-and-live-encoders/) now tells us to use RTMP with H.264 video and AAC audio, a 2-second key-frame interval, and CBR (constant bit rate) encoding. ### Configuring ffmpeg #### Links * [ffmpeg - command line options](https://ffmpeg.org/ffmpeg.html) * [ffmpeg - Streaming](https://trac.ffmpeg.org/wiki/StreamingGuide) * [ffmpeg - Encoding for streaming sites](https://trac.ffmpeg.org/wiki/EncodingForStreamingSites) * #### Command line arguments **misc** * `-y` Overwrite output files without asking * `-loglevel debug` (or verbose, quiet, panic, fatal) #### Input * `-f dshow` use DirectShow Filter * `-i video="Integrated Camera":audio="Microphone (Realtek High Definition Audio)"` use internal web cam and microphone **Video output** * `-s 640x480` Resolution * `-codec:v libx264` H.264 / AVC video * `-pix_fmt yuv420p` pixel format YUV420 * `-preset veryfast` (ultrafast,superfast, veryfast, faster, fast, medium, slow, slower, veryslow, placebo) * `-b:v 200k` target video bit rate * `-minrate 200k` minimum video bit rate * `-maxrate 200k` maximum video bit rate * `-r 30` frame rate * `-keyint_min 60` minimum GOP size * `-g 60` maximum GOP size * `-sc_threshold 0` scene change threshold * `-bsf:v h264_mp4toannexb` bitstream filter. Use `ffmpeg -bsfs` for a full list * `-profile:v main` preset according to [docs](https://trac.ffmpeg.org/wiki/Encode/H.264#a2.Chooseapreset) * `-level 3.1` compatible level according to [docs](https://trac.ffmpeg.org/wiki/Encode/H.264#Compatibility) **Audio output** * `-codec:a libvo_aacenc` AAC audio * `-b:a 128k` audio bit rate * `-ar 44100` audio sampling frequency * `-ac 2` audio channels * `-strict experimental` **overall stream** * `-bufsize 200k` buffer size * `-maxrate 200k` maximim bit rate **Destination** * `-f flv rtmp://chan1-acc2.channel.mediaservices.windows.net:1936/live/deadbeef/chan1` target RTMP endpoint to push to ### Ingest the RTMP stream ``` set DEST=rtmp://channel1-mediaservice321.channel.mediaservices.windows.net:1935/live/deadbeef012345678890abcdefabcdef/channel1 set SRC=video="Integrated Camera":audio="Headset Microphone (GN 2000 USB OC)" ffmpeg -f dshow -i %SRC% -s 640x480 -preset veryfast -codec:v libx264 -pix_fmt yuv420p -b:v 200k -minrate 200k -maxrate 200k -bufsize 200k -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -codec:a aac -b:a 48k -f flv %DEST% set VIDEOBITRATE=200k ffmpeg -f dshow -i %SRC% -s 640x480 -preset veryslow -codec:v libx264 -pix_fmt yuv420p -pass 1 -b:v %VIDEOBITRATE% -minrate %VIDEOBITRATE% -maxrate %VIDEOBITRATE% -bufsize %VIDEOBITRATE% -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -profile:v main -level 3.1 -codec:a aac -ar 44100 -b:a 96k -ac 2 -f flv %DEST% set DEST=rtmp://channel1-mediaservice321.channel.mediaservices.windows.net:1935/live/deadbeef012345678890abcdefabcdef/channel1 set SRC="C:\Users\chgeuer\Cosmos Laundromat - First Cycle. Official Blender Foundation release.-Y-rmzh0PI3c.webm" ffmpeg -re -f dshow -i %SRC% -s 640x480 -preset veryslow -codec:v libx264 -pix_fmt yuv420p -pass 1 -b:v %VIDEOBITRATE% -minrate %VIDEOBITRATE% -maxrate %VIDEOBITRATE% -bufsize %VIDEOBITRATE% -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -profile:v main -level 3.1 -codec:a aac -ar 44100 -b:a 96k -ac 2 -f flv %DEST% ``` You can use the [DASHPlayer](http://dashplayer.azurewebsites.net/) or [aka.ms/azuremediaplayer](http://amsplayer.azurewebsites.net/azuremediaplayer.html). Don't forget to append `(format=mpd-time-csf)` or `(format=m3u8-aapl)` to the streams for DASH or HLS streaming. ## MPEG TS Streaming ``` RTP protocol (MPEG Transport Streams) encoded MPEG-2 -f mpegts udp://127.0.0.1:10000?pkt_size=1316 -f rtp rtp://127.0.0.1:1234 ``` * \[FFMPEG for TS streaming]\([https://www.wowza.com/forums/content.php?213-How-to-use-FFmpeg-with-Wowza-Media-Server-(MPEG-TS](https://www.wowza.com/forums/content.php?213-How-to-use-FFmpeg-with-Wowza-Media-Server-%28MPEG-TS))) ``` ffmpeg -re -i %SRC% -s 640x480 -preset veryslow -codec:v libx264 -pix_fmt yuv420p -pass 1 -b:v %VIDEOBITRATE% -minrate %VIDEOBITRATE% -maxrate %VIDEOBITRATE% -bufsize %VIDEOBITRATE% -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -profile:v main -level 3.1 -codec:a aac -ar 44100 -b:a 96k -ac 2 -f rtp rtp://127.0.0.1:1234 ``` * [Flash Media Live Encoder (FMLE)](http://www.adobe.com/de/products/flash-media-encoder.html) and [MainConcept AAC Encoder 1.0.6 Plugin for Adobe Flash Media Live Encoder](http://www.mainconcept.com/eu/products/plug-ins/plug-ins-for-adobe/aac-encoder-fmle.html) * [Open Broadcaster Software](https://obsproject.com/) (OBS Classic & OBS Studio) * [Blog: Azure Media Services RTMP Support and Live Encoder](https://azure.microsoft.com/en-us/blog/azure-media-services-rtmp-support-and-live-encoders/) * [Telestream Wirecast Trial Version](http://www.telestream.net/wirecast/overview.htm) * [nginx RTMP](https://obsproject.com/forum/resources/how-to-set-up-your-own-private-rtmp-server-using-nginx.50/) * [Red 5 Server](http://red5.org/) ### Single bitrate: ``` ffmpeg -v verbose -i MysampleVideo.mp4 -strict -2 -codec:a aac -b:a 128k -ar 44100 -codec:v libx264 -b:v 400000 -bufsize 400k -maxrate 400k -preset medium -r 30 -g 60 -keyint_min 60 -f flv rtmp://channel001-streamingtest.channel.media.windows.net:1935/live/a9bcd589da4b424099364f7ad5bd4940/mystream1 ``` ### Multi bitrate ( 3 bit rates 500, 300 and 150 Kbps) ``` ffmpeg -threads 15 -re -i MysampleVideo.mp4 -strict experimental -codec:a aac -ab 128k -ac 2 -ar 44100 -codec:v libx264 -s 800x600 -b:v 500k -minrate 500k -maxrate 500k -bufsize 500k -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -f flv rtmp://channel001-streamingtest.channel.media.windows.net:1935/live/a9bcd589da4b424099364f7ad5bd4940/Streams_500 -strict experimental -codec:a aac -ab 128k -ac 2 -ar 44100 -codec:v libx264 -s 640x480 -b:v 300k -minrate 300k -maxrate 300k -bufsize 300k -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -f flv rtmp://channel001-streamingtest.channel.media.windows.net:1935/live/a9bcd589da4b424099364f7ad5bd4940/Streams_300 -strict experimental -codec:a aac -ab 128k -ac 2 -ar 44100 -codec:v libx264 -s 320x240 -b:v 150k -minrate 150k -maxrate 150k -bufsize 150k -r 30 -g 60 -keyint_min 60 -sc_threshold 0 -f flv rtmp://channel001-streamingtest.channel.media.windows.net:1935/live/a9bcd589da4b424099364f7ad5bd4940/Streams_150 ``` ## Interesting FFMPEG Articles * [ffmbc - FFMedia Broadcast](https://code.google.com/p/ffmbc/) ## Using FFMPEG to convert audio files ``` ffmpeg -i infile.flac outfile.wav REM http://etree.org/shnutils/shntool/ shntool.exe split -f infile.cue -t %n-%t -m /- outfile.wav dir *.wav | foreach { ffmpeg -i $_.Name -ab 320k $_.Name.Replace("wav", "mp3") } REM Convert FLAC to MP3 VBR dir *.flac | foreach { ffmpeg -i $_.Name -qscale:a 1 $_.Name.Replace("flac", "mp3") } REM Convert FLAC to MP3 320k dir *.flac | foreach { ffmpeg -i $_.Name -ab 320k $_.Name.Replace("flac", "mp3") } REM Create M4B from MP3 collection ffmpeg -i "concat:01.mp3|02.mp3" -c:a libvo_aacenc -vn out.m4a ren out.m4a out.m4b REM Convert mp3 to m4a dir *.mp3 | foreach { ffmpeg -i $_.Name -c:a libvo_aacenc -vn $_.Name.Replace("mp3", "m4a") } ``` ## Convert a bunch of MP3 files to an iPod audio book ``` # Convert a bunch of MP3 files to an iPod audio book $folder = "C:\Users\Public\Music\Star Wars Episode 1 - Die dunkle Bedrohung" Function concatenate($lines) { $sb = New-Object -TypeName "System.Text.StringBuilder"; [void]$sb.Append(""""); [void]$sb.Append("concat:"); for ($i=0; $i -le $lines.Length; $i++) { [void]$sb.Append($lines[$i].Name); if ($i -le ($lines.Length - 2)) { [void]$sb.Append("|"); } } [void]$sb.Append(""""); return $sb.ToString(); } Set-Location $folder $filename = (Get-Item $folder).Name $inputfiles = Get-ChildItem -Filter *.mp3 | Sort-Object -Property Name $concatenation = concatenate($inputfiles) # ffmpeg -i "concat:01.mp3|02.mp3" -c:a libvo_aacenc -vn 1.m4a ffmpeg -i $concatenation -c:a libvo_aacenc -vn "$filename.m4a" # Rename-Item -Path "$filename.m4a" -NewName "$filename.m4b" # compare two videos @see http://ianfeather.co.uk/compare-two-webpagetest-videos-using-ffmpeg/ ffmpeg -i before.mp4 -i after.mp4 -filter_complex "[0:v:0]pad=iw*2:ih[bg]; [bg][1:v:0]overlay=w" output.mp4 ``` ## Using FFMPEG to create VOD files ``` SET FFMPEG="c:\program files\ffmpeg\bin\ffmpeg.exe" SET GOPSIZE=-g 25 SET GOPSIZE= SET VIDEOBITRATE=-b:v 1500k SET RESOLUTION=-s "960x540" SET RESOLUTION= REM http://www.idude.net/index.php/how-to-watermark-a-video-using-ffmpeg SET WATERMARK= -filter_complex "overlay=main_w-overlay_w-10:main_h-overlay_h-10" SET WATERMARK= -filter_complex "overlay=(main_w+overlay_w)/2:(main_h+overlay_h)/2" SET WATERMARK= -vf "movie=logo2.png [watermark]; [in][watermark] overlay=main_w-overlay_w-10:main_h-overlay_h-10 [out]" SET CODEC_MP4= -vcodec libx264 -pix_fmt yuv420p %WATERMARK% %GOPSIZE% %VIDEOBITRATE% SET CODEC_WEBM= -vcodec libvpx -acodec libvorbis -ab 160000 -f webm %WATERMARK% %GOPSIZE% %VIDEOBITRATE% SET CODEC_OGV= -vcodec libtheora -acodec libvorbis -ab 160000 %WATERMARK% %GOPSIZE% %VIDEOBITRATE% SET CODEC_POSTER= -ss 00:02 -vframes 1 -r 1 -f image2 %WATERMARK% %FFMPEG% -i %1 %CODEC_MP4% %RESOLUTION% "%~n1.mp4" %FFMPEG% -i %1 %CODEC_WEBM% %RESOLUTION% "%~n1.webm" %FFMPEG% -i %1 %CODEC_OGV% %RESOLUTION% "%~n1.ogv" %FFMPEG% -i %1 %CODEC_POSTER% %RESOLUTION% "%~n1.jpg" REM http://stackoverflow.com/questions/7333232/concatenate-two-mp4-files-using-ffmpeg REM file '1.mp4' REM file '2.mp4' REM %FFMPEG% -f concat -i mylist.txt -c copy output REM Remux MOV to MP4 ffmpeg -i input.mov -vcodec copy -acodec libvo_aacenc -map_metadata 0 result.mp4 dir *.MOV | foreach { ffmpeg -i $_.Name -vcodec copy -acodec libvo_aacenc $_.Name.Replace("MOV", "mp4") } REM Remux MKV to MP4 ffmpeg -i a.mkv -vcodec copy -ab 128k -acodec libvo_aacenc -map_metadata 0 a.mp4 ffmpeg -i a.mkv -vcodec copy -acodec copy -map_metadata 0 a.mp4 ``` ## Convert FLV (Flash Video) into real MP4 Files are FLVs, but named MP4. Make them *real* MP4. The `-map_metadata 0` ensures that metadata like date etc flows over to the new file. ``` dir *.mp4 | foreach { Rename-Item $_.Name $_.Name.Replace("MP4", "flv").Replace("mp4", "flv") } dir *.flv | foreach { ffmpeg -i $_.Name -vcodec copy -acodec copy -map_metadata 0 $_.Name.Replace('FLV', 'mp4').Replace('flv', 'mp4') } dir *.MOV | foreach { ffmpeg -i $_.Name -vcodec copy -acodec libvo_aacenc -map_metadata 0 $_.Name.Replace('MOV', 'mp4').Replace('mov', 'mp4') } ``` ``` powershell -Command "dir *.MOV | foreach { ffmpeg -i $_.Name -vcodec copy -acodec libvo_aacenc -map_metadata 0 $_.Name.Replace('MOV', 'mp4').Replace('mov', 'mp4') }" powershell -Command "dir *.AVI | foreach { ffmpeg -i $_.Name -map_metadata 0 $_.Name.Replace('AVI', 'mp4').Replace('avi', 'mp4') }" ``` ## Concatenate video files In order to concatenate MP4 files, each file must be converted into a Transport Stream (.ts), i.e. without a MOOV atom, and then concatenated and re-written into a proper .mp4 file (with MOOV atom): ``` dir *.mp4 | foreach { ffmpeg -i $_.Name -c copy -bsf:v h264_mp4toannexb -f mpegts $_.Name.Replace("MP4", "ts").Replace("mp4", "ts") } ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy -bsf:a aac_adtstoasc output.mp4 ``` Alternatively, you can list all input files in a text file: ``` $ cat m.txt file 'm1-01 - Introduction - Introduction.ts' file 'm1-02 - Introduction - Tools.ts' $ ffmpeg -f concat -i list.txt -c copy -bsf:a aac_adtstoasc output.mp4 ``` ## Create MP4 from single images ``` ffmpeg -start_number 3407 -i img_%4d.jpg -c:v libx264 -s "1404x936" out.mp4 ``` * [YouTube Advanced encoding settings](http://support.google.com/youtube/answer/1722171) ``` ffmpeg -i "1.mkv" -vcodec h264 -acodec libvo_aacenc "1.mp4" ffmpeg -i "1.mkv" -vcodec copy -acodec libvo_aacenc "1.mp4" ``` ## Render Game Frame previews from a series of PNGs Download art from the [Game Frame Art Forum](https://ledseq.com/forums/forum/game-frame/game-frame-art/) ``` ffmpeg -start_number 0 -i %d.bmp -c:v libx264 -s "256x256" -sws_flags neighbor tetris.mp4 ffmpeg -start_number 0 -i %d.bmp -s "256x256" -sws_flags neighbor tetris.gif ``` ## Repair MP4 ## Generate HLS ### ```bash ffmpeg -y -framerate 24 -i 720/sintel_trailer_2k_%4d.png -i sintel_trailer-audio.flac -c:a aac -strict experimental -ac 2 -b:a 64k -ar 44100 -c:v libx264 -pix_fmt yuv420p -profile:v baseline -level 1.3 -maxrate 192K -bufsize 1M -crf 18 -r 10 -g 30 -f hls -hls_time 9 -hls_list_size 0 -s 320x180 ts/320x180.m3u8 ffmpeg -y -framerate 24 -i 720/sintel_trailer_2k_%4d.png -i sintel_trailer-audio.flac -c:a aac -strict experimental -ac 2 -b:a 64k -ar 44100 -c:v libx264 -pix_fmt yuv420p -profile:v baseline -level 2.1 -maxrate 500K -bufsize 2M -crf 18 -r 10 -g 30 -f hls -hls_time 9 -hls_list_size 0 -s 480x270 ts/480x270.m3u8 ffmpeg -y -framerate 24 -i 720/sintel_trailer_2k_%4d.png -i sintel_trailer-audio.flac -c:a aac -strict experimental -ac 2 -b:a 96k -ar 44100 -c:v libx264 -pix_fmt yuv420p -profile:v baseline -level 3.1 -maxrate 1M -bufsize 3M -crf 18 -r 24 -g 72 -f hls -hls_time 9 -hls_list_size 0 -s 640x360 ts/640x360.m3u8 ffmpeg -y -framerate 24 -i 720/sintel_trailer_2k_%4d.png -i sintel_trailer-audio.flac -c:a aac -strict experimental -ac 2 -b:a 96k -ar 44100 -c:v libx264 -pix_fmt yuv420p -profile:v main -level 3.2 -maxrate 2M -bufsize 6M -crf 18 -r 24 -g 72 -f hls -hls_time 9 -hls_list_size 0 -s 1280x720 ts/1280x720.m3u8 ``` ### ```bash @echo off REM Line 1 sets the universal encoding parameters for all video files including the recommended 2-second GOP size. REM Lines 2 – 5 set the encoding parameters and identifies the four video files. You can add any encoding parameter to any line so long as you designate which file it is using the v:# syntax shown. I listed the 540p file first to loosely comply with Apple’s recommendation of starting with a 2 Mbps file. You can add or subtract files from the ladder so long as you adjust the mappings on lines 7 and 8. REM Line 6 produces the audio file. REM Line 7 maps the single input video file to all video files in the encoding ladder and maps the single audio file to the audio output. Note that there are four -map 0:v switches, one for each video file. If you add or subtract video files you’d have to adjust this line accordingly. REM Line 8 chooses the HLS format and then maps the individual video files to the single audio file. Again, you need a mapping statement for each video file in the encoding ladder. REM This is what it looks like in the master manifest file. You see the two video files shown both play the group_audio file specified in line 8. REM Line 9 sets normal HLS options like producing a single file rather than multiple segments (-hls_flags single_file), producing a fragmented MP4 file rather than an MPEG-2 transport stream (-hls_segment_type fmp4), including all segments in each manifest file (-hls_list_size 0), choosing six-second segments (-hls_time 6) and then naming the master and media manifest files. ffmpeg.exe ^ -threads 0 -i TOS_1080p.mov -r 24 -g 48 -keyint_min 48 -sc_threshold 0 -c:v libx264^ -s:v:0 960x540 -b:v:0 2400k -maxrate:v:0 2640k -bufsize:v:0 2400k^ -s:v:1 1920x1080 -b:v:1 5200k -maxrate:v:1 5720k -bufsize:v:1 5200k^ -s:v:2 1280x720 -b:v:2 3100k -maxrate:v:2 3410k -bufsize:v:2 3100k^ -s:v:3 640x360 -b:v:3 1200k -maxrate:v:3 1320k -bufsize:v:3 1200k^ -b:a 128k -ar 44100 -ac 2^ -map 0:v -map 0:v -map 0:v -map 0:v -map 0:a^ -f hls -var_stream_map "v:0,agroup:audio v:1,agroup:audio v:2,agroup:audio v:3,agroup:audio a:0,agroup:audio"^ -hls_flags single_file -hls_segment_type fmp4 -hls_list_size 0 -hls_time 6 -master_pl_name master.m3u8 -y TOS%v.m3u8 ``` ### 2-second TS {% code title="" %} ```bash #!/bin/bash ffmpeg \ -threads 0 \ -i input.mp4 \ -r 24 -g 48 -keyint_min 48 -sc_threshold 0 -c:v libx264 \ -s:v:0 640x360 -b:v:0 1200k -maxrate:v:0 1320k -bufsize:v:0 1200k \ -s:v:1 960x540 -b:v:1 2400k -maxrate:v:1 2640k -bufsize:v:1 2400k \ -s:v:2 1280x720 -b:v:2 3100k -maxrate:v:2 3410k -bufsize:v:2 3100k \ -s:v:3 1920x1080 -b:v:3 5200k -maxrate:v:3 5720k -bufsize:v:3 5200k \ -b:a 128k -ar 44100 -ac 2 \ -map 0:v -map 0:v -map 0:v -map 0:v -map 0:a \ -f hls -var_stream_map "v:0,agroup:audio v:1,agroup:audio v:2,agroup:audio v:3,agroup:audio a:0,agroup:audio" \ -hls_segment_type mpegts \ -hls_list_size 0 \ -hls_playlist_type vod \ -hls_time 2 \ -hls_allow_cache 1 \ -hls_segment_filename vid-%%v-%%03d.ts \ -master_pl_name master.m3u8 \ -y sub-%%v.m3u8 ``` {% endcode %} {% embed url="" %} # JOSE from the command line {% code title="jose-demo.sh" %} ```bash #!/bin/bash # https://tools.ietf.org/html/rfc7515#appendix-C function create_base64_url { local base64text="$1" echo -n "${base64text}" | sed -E s%=+$%% | sed s%\+%-%g | sed -E s%/%_%g } function hmac_sha256 { local base64Key="$1" local signature_input="$2" local hexkey base64hmac hexkey="$( echo -n "${base64Key}" | base64 -d | od -t x1 -An | tr -d '\n ' )" base64hmac="$( echo -n "${signature_input}" | openssl dgst -sha256 -mac hmac -macopt "hexkey:${hexkey}" -binary | base64 --wrap=0 )" create_base64_url "${base64hmac}" } function json_to_base64 { local jsonText="$1" local encoded encoded="$( echo -n "${jsonText}" | base64 --wrap=0 )" create_base64_url "${encoded}" } function sign_json { local base64Key="$1" local jsonPayloadText="$2" local algorithm header_json header payload signature_input sig # https://tools.ietf.org/html/rfc7515 # header="$( json_to_base64 '{"alg":"HS256","typ":"JWT"}' )" algorithm="HS256" header_json="$( echo "{}" | \ jq --arg x "${algorithm}" '.alg=($x)' | \ jq --arg x "JWT" '.typ=($x)' | \ iconv --from-code=ascii --to-code=utf-8 )" header="$( json_to_base64 "${header_json}" )" payload="$( json_to_base64 "${jsonPayloadText}" )" signature_input="$( echo -n "${header}.${payload}" | iconv --to-code=ascii )" sig="$( hmac_sha256 "${base64Key}" "${signature_input}" )" echo "${header}.${payload}.${sig}" | iconv --to-code=ascii } function get_current_utc_time { date --utc +"%Y-%m-%dT%H:%M:%SZ" } function generate_request { local base64Key="$1" local tenantID="$2" local subscriptionID="$3" local timestamp="$4" local json json="$( echo "{}" | \ jq --arg x "${tenantID}" '.tenantId=($x)' | \ jq --arg x "${subscriptionID}" '.subscriptionId=($x)' | \ jq --arg x "${timestamp}" '.timeStamp=($x)' | \ jq --arg x '[ "subj" ]' '.claims=($x | fromjson)' | \ jq -c -M | iconv --from-code=ascii --to-code=utf-8 )" sign_json "${base64Key}" "${json}" } base64Key="pDzCAKG9KSaCWY2kLaqf0UWJ89i/gy/6IGvndSWe4eo=" tenantID="chgeuerfte.onmicrosoft.com" subscriptionID="fb7fdc26-b0e5-45b6-8119-7bc48bc12e4e" token="$( generate_request "${base64Key}" "${tenantID}" "${subscriptionID}" "$( get_current_utc_time )" )" echo "${token}" # cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${token}" )" ``` {% endcode %} # jq The [`jq`](https://stedolan.github.io/jq/) utility helps parsing, and modifying JSON structures. ### Install ```bash curl \ --silent \ --url https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \ --location \ --output ./jq chmod +x ./jq sudo mv ./jq /usr/local/bin sudo chown root.root /usr/local/bin/jq ``` When I neet to create a larger JSON structure in a shell script, I don't like the approach of creating a honking big string: ```bash #!/bin/bash someVar1="Something something" someVar2="something else" # Create a large JSON string, not very maintainable json1="{\"v1\":\"${someVar1}\",\"v2\": \"${someVar2}\"}" echo "${json1}" | jq . ``` Instead, I prefer a step-by-step creation process. Below, we're starting with an empty JSON object (`"{}"`), and piping it through a bunch of `jq` calls. In such a call, we bind the shell variable's value to the `jq` variable `x`, and set the value on the fly: ```bash #!/bin/bash someVar1="Something something" someVar2="something else" # Create a large JSON string, not very maintainable json="$( echo "{}" \ | jq --arg x "${someVar1}" '.v1=$x' \ | jq --arg x "${someVar2}" '.v2=$x' \ )" echo "${json}" | jq . ``` This outputs ```json { "v1": "Something something", "v2": "something else" } ``` Alternatively, try this: ```bash #!/bin/bash someVar1="Something something" someVar2="true" # Create JSON step-by-step json="$( echo "{}" \ | jq --arg x "${someVar1}" '.v1=$x' \ | jq --arg x "${someVar2}" '.trueAsAString=$x' \ | jq --arg x "${someVar2}" '.trueAsABoolean=($x | fromjson)' \ | jq '.someArray=[]' \ )" # # Now let's add a bash array to the JSON structure, element by element # allThreads=(1 2 '"hi"' "true" false '{}') for t in ${allThreads[@]}; do json="$( echo "${json}" | jq --arg xx "${t}" '.someArray[.someArray | length] |= .+ ($xx | fromjson)' )" done echo "${json}" | jq . ``` Which gives ```json { "v1": "Something something", "trueAsAString": "true", "trueAsABoolean": true, "someArray": [ 1, 2, "hi", true, false, {} ] } ``` ### Incrementally adding to an array in JQ ```bash #!/bin/bash x="$( echo "{}" | jq ".x=[]" )" x="$( echo "${x}" | jq ".x[.x | length] |= .+ \"foo\"" )" x="$( echo "${x}" | jq ".x[.x | length] |= .+ 1" )" x="$( echo "${x}" | jq ".x[.x | length] |= .+ true" )" echo "${x}" ``` results in ```json { "x": [ "foo", 1, true ] } ``` ### Forgot what I did here... ```bash #!/bin/bash sample='[{"name":"foo"},{"name":"bar"}]' for row in $(echo "${sample}" | jq -r ".[] | @base64" ); do _jq() { echo ${row} | base64 --decode | jq -r ${1} } echo "$( _jq ".name" )" done ``` ### Crack up the claims part of a JWT ```bash #!/bin/bash access_token="eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImtleTEifQ.eyJpc3MiOiJodHRwczovL2NoZ2V1ZXIuYmxvYi5jb3JlLndpbmRvd3MubmV0L3B1YmxpYyIsImF1ZCI6ImFwaTovL0F6dXJlQURUb2tlbkV4Y2hhbmdlIiwic3ViIjoic3ViamVjdCIsImlhdCI6MTY0NDQ0NjQzNSwiZXhwIjoxNjQ0NDQ3MDM1fQ.GG1n7B81D8HZIvCS2e-bWzomMRyqMb-zs-8oF2Oh_dgIQy2TT15DoJaBZrybsMvRRjfJbsmQz_Pay1dFSwtfDMY4WF2MwxPk95VKz4f9MO4SA_NVY8lMHB49gosBqUpQrPD8DzKEIm-cvIpJJTQzNpcefoNf_Ax2AlbDy5h3zazlvqpFUjim-3nwCTN2ERd0FB3iTsTMMk6VG0bu76qkK-9t3Zh8tdkENCcLVQslHBkcFhN3F9hRLLYiEtOTyLTFaWSr0PDbsiSsMmK8XGxMHNF2K03LGk-whb0mPjKQ17mvclMLgJ9M187KC8HljAIj3Powtw9YsVp6kC2y" claims="$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${access_token}" )" echo "${claims}" | jq . echo "${claims}" | jq .iss ``` # Misc. command line tools Misc command line tools ### Base64 encoding * Without whitespace: `base64 --wrap=0` ### Text encodings * Convert ASCII to UTF8: `iconv --from-code=ascii --to-code=utf-8` ### Windows / WSL interop * Copy WSL text into the Windows clipboard: `echo "Hello äöü" | iconv -f utf-8 -t utf-16le | clip.exe` * This converts it to UTF16 little endian and pipes it to the Windows clipboard :-) * Start a web browser on Windows from WSL: `cmd.exe /C "start $( echo "https://github.com/" )"` ### git tips #### `chmod +x` in git ```bash git update-index --chmod=+x ./sample.sh ``` ### Image renaming Use [`jhead`](https://www.sentex.ca/~mwandel/jhead/) to extract EXIF metadata, and rename image files. For example, this renames all images based on the date-taken EXIF metadata ```bash jhead -n%Y-%m-%d--%H-%M-%S_%f *.jpg ``` * `-n[format-string]` * `%Y-%m-%d` Year (4 digit 1980-2036) / month number / day of month * `%H-%M-%S` hour (24h) / minute / second * `%f` original file name # Zettelkasten / Markdown As I move more and more to a purely Markdown-based workflow for keeping notes (Zettelkasten ruins my mind), I was looking for a simpler way to create fresh empty markdown files in a folder. Up till now, I right-clicked in Windows Explorer, selected 'New Text file', glanced at the system clock, converted the current time to something like `YYYYMMDDhhmmss`, and named the text-file accordingly, like `20210108180300.md`. It struck me that this is precisely what I certainly should automate, so I did the following: I created a small Windows batch file named `new-markdown.cmd`, which I stored in my personal `bin` directory (`C:\Users\chgeuer\bin\new-markdown.cmd`): ## Contents of `new-markdown.cmd` ```batch @ECHO OFF :: https://stackoverflow.com/questions/12635541/safe-way-to-get-current-day-month-and-year-in-batch :: Check WMIC is available WMIC.EXE Alias /? >NUL 2>&1 || GOTO s_error :: Use WMIC to retrieve date and time FOR /F "skip=1 tokens=1-6" %%G IN ('WMIC Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year /Format:table') DO ( IF "%%~L"=="" goto s_done Set _yyyy=%%L Set _mm=00%%J Set _dd=00%%G Set _hour=00%%H SET _minute=00%%I SET _second=00%%K ) :s_done :: Pad digits with leading zeros Set _mm=%_mm:~-2% Set _dd=%_dd:~-2% Set _hour=%_hour:~-2% Set _minute=%_minute:~-2% Set _second=%_second:~-2% :: Display the date/time in ISO 8601 format: SET _isodate=%_yyyy%%_mm%%_dd%%_hour%%_minute%%_second% cd /D %1 set HEAD=# %_yyyy%-%_mm%-%_dd% %_hour%:%_minute%:%_second% echo %HEAD% > "%_isodate%.md" explorer.exe "%_isodate%.md" ``` ## Grab some ICO and hook the batch to the registry Then, I downloaded some ICO file representing markdown, and stored it alongside the batch file, in the `bin` folder. As a last step, I created these registry entries: ``` Windows Registry Editor Version 5.00 [HKEY_CLASSES_ROOT\Directory\Background\shell\NewMarkdown] @="Add Markdown file" "Icon"="C:\\Users\\chgeuer\\bin\\markdown_106519.ico" [HKEY_CLASSES_ROOT\Directory\Background\shell\NewMarkdown\command] @="\"C:\\Users\\chgeuer\\bin\\new-markdown.cmd\" \"%V\"" ``` As a result, you get this entry, when you right-click on the empty space in a folder ![Markdown Menu Entry](/files/HBjVQ00bmiC4Wx052Sc5) The last thing in the batch file (this `explorer.exe "%_isodate%.md"` thing) kicks off your favorite Markdown editor on the newly created MD file. ### Links * [Safe way to get current day month and year in batch - Stack Overflow](https://stackoverflow.com/questions/12635541/safe-way-to-get-current-day-month-and-year-in-batch) * [Windows: How to add batch-script action to Right Click menu - Super User](https://superuser.com/questions/444726/windows-how-to-add-batch-script-action-to-right-click-menu) * [Some Markdown Icons](https://icon-icons.com/de/suche/symbole/markdown) Alternatively, you can run this utility (with admin rights): [`install_markdown_YYYYmmdd.cmd`](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/.gitbook/assets/install_markdown_YYYYmmdd.cmd). ``` @echo off set ICO=https://icon-icons.com/downloadimage.php?id=160764^&root=2648/ICO/128/^&file=dev_markdown_icon_160764.ico cmd.exe /c powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -Command "(New-Object System.Net.WebClient).DownloadFile('%ICO%','%USERPROFILE%\bin\markdown.ico')" set Name=NewMarkdown SETLOCAL EnableDelayedExpansion reg.exe ADD HKCR\Directory\Background\shell\%Name% /f /t REG_SZ /d "Add Markdown file" reg.exe ADD HKCR\Directory\Background\shell\%Name% /f /v Icon /t REG_SZ /d "%USERPROFILE%\bin\markdown.ico" reg.exe ADD HKCR\Directory\Background\shell\%Name%\command /f /t REG_SZ /d "\"^%USERPROFILE%\bin\new-markdown.cmd\" \"^%V\"" reg.exe ADD HKCR\Directory\Background\shell\%Name%\command /f /t REG_SZ /d "\"^%USERPROFILE%\bin\nircmd.exe\" exec hide \"^%USERPROFILE%\bin\new-markdown.cmd\" \"^%V\"" for /f "delims=:" %%a in ('findstr -n "^___DATA___" %0') do set "Line=%%a" (for /f "skip=%Line% tokens=* eol=_" %%a in ('type %0') do echo(%%a) > "%USERPROFILE%\bin\new-markdown.cmd" goto:EOF ___DATA___ @ECHO OFF cd /D %1 :: https://stackoverflow.com/questions/12635541/safe-way-to-get-current-day-month-and-year-in-batch :: Check WMIC is available WMIC.EXE Alias /? >NUL 2>&1 || GOTO s_error :: Use WMIC to retrieve date and time FOR /F "skip=1 tokens=1-6" %%G IN ('WMIC Path Win32_LocalTime Get Day^,Hour^,Minute^,Month^,Second^,Year /Format:table') DO ( IF "%%~L"=="" goto s_done Set _yyyy=%%L Set _mm=00%%J Set _dd=00%%G Set _hour=00%%H SET _minute=00%%I SET _second=00%%K ) :s_done :: Pad digits with leading zeros Set _mm=%_mm:~-2% Set _dd=%_dd:~-2% Set _hour=%_hour:~-2% Set _minute=%_minute:~-2% Set _second=%_second:~-2% SET _isodate=%_yyyy%%_mm%%_dd%%_hour%%_minute%%_second% SET HEAD=# %_yyyy%-%_mm%-%_dd% %_hour%:%_minute%:%_second% echo %HEAD% > "%_isodate%.md" echo, >> "%_isodate%.md" echo, >> "%_isodate%.md" echo, >> "%_isodate%.md" explorer.exe "%_isodate%.md" ``` # Logging in to Azure Authentication and logging-in to Azure * Azure Resource Manager Login Stuff * [Certificate-based auth with Azure Service Principals from Linux command line](http://blogs.msdn.com/b/arsen/archive/2015/09/18/certificate-based-auth-with-azure-service-principals-from-linux-command-line.aspx) * [Authenticating a service principal with Azure Resource Manager](https://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-principal/) * [Python access to ARM](https://github.com/gbowerman/azurerm) * [Create an Azure service principal with Azure CLI 2.0](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?view=azure-cli-latest) ``` "" is something like "adadadad-adad-adad-adad-adadadadadad" "" is something like "40302010-feda-deaf-beef-deadbeef0123" ``` ## Setup service principal in Powershell ### Install Azure Powershell according the [docs](https://azure.microsoft.com/en-us/documentation/articles/powershell-install-configure/) ``` # Install the Azure Resource Manager modules from the PowerShell Gallery Install-Module AzureRM Install-AzureRM Install-Module Azure Import-AzureRM Import-Module Azure ``` ### Create a certificate using `makecert.exe` ``` $subjectName = "CN=AzureServicePrincipal" $certificateFile = [System.IO.Path]::Combine($env:USERPROFILE, "Desktop", "$($subjectName.Replace('CN=', '')).cer") $azureADTenantID = $env:AzureADTenantID; # "adadadad-adad-adad-adad-adadadadadad" $subscriptionID = $env:AzureSubscriptionID; # "706df49f-998b-40ec-aed3-7f0ce9c67759" $manualBillingAdmin = $env:AzureManualBillingAdmin; # "billingoperator@contoso.onmicrosoft.com" $appName = "Azure Service Principal for Automation" $dummyUrl = "http://localhost/serviceprincipal" # fetch makecert from some random dude on the Internet :-/ (New-Object Net.WebClient).DownloadFile('https://gist.github.com/chgeuer/f2334a3222215ef93ff234fd7dcf1a01/raw/9bba6abee1812e9917c21dd8c50fe226bcdfcc7d/makecert.exe', 'makecert.exe') .\makecert.exe -r -pe -len 2048 -a sha512 -h 0 -sky signature -ss My -n "$($subjectName)" $cer = (dir Cert:\CurrentUser\My\ | where { $_.Subject -eq $subjectName }) $certThumbprint = $cer.Thumbprint $certOctets = $cer.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert) $credValue = [System.Convert]::ToBase64String($certOctets) # $certOctets = Get-Content -Path $certificateFile -Encoding Byte # $cer = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList @(,[System.Byte[]]$certOctets) ``` ### Fiddle with PowerShell * ``` $proxyString = "http://127.0.0.1:8888" $proxyUri = new-object System.Uri($proxyString) [System.Net.WebRequest]::DefaultWebProxy = New-Object System.Net.WebProxy ($proxyUri, $true) ``` ### Fill in your Azure details ``` $credential = Get-Credential -UserName $manualBillingAdmin -message "Provide your organizational credentials for $($manualBillingAdmin)" Login-AzureRmAccount -Tenant $azureADTenantID -SubscriptionId $subscriptionID -Credential $credential Select-AzureRmSubscription -SubscriptionId $subscriptionID $application = New-AzureRmADApplication -DisplayName $appName -HomePage $dummyUrl -IdentifierUris $dummyUrl -KeyType AsymmetricX509Cert -KeyValue $credValue Start-Sleep -Seconds 1 # Remove-AzureRmADServicePrincipal -ObjectId New-AzureRmADServicePrincipal -ApplicationId $application.ApplicationId Start-Sleep -Seconds 1 New-AzureRmRoleAssignment -ServicePrincipalName $application.ApplicationId -RoleDefinitionName Contributor Write-Host "Use SubscriptionID == $($subscriptionID)" Write-Host "Use azureADTenantID == $($azureADTenantID)" Write-Host "Use clientID == $($application.ApplicationID)" Write-Host "Use certThumbprint == $($certThumbprint)" ``` ## Powershell / X509 * Src: [Authenticate service principal with certificate - PowerShell](https://github.com/Azure/azure-content/blob/master/articles/resource-group-authenticate-service-principal.md#authenticate-service-principal-with-certificate---powershell) ``` $tenantId = "942023a6-efbe-4d97-a72d-532ef7337595" $applicationId = "4bc204cb-3282-43b1-aa1f-960f5faa4b23" $certThumbprint = "B8789A48A020FB1F5589C9ACAF63A4EBFFF5FA1C" Login-AzureRmAccount ` -ServicePrincipal ` -TenantId $tenantId ` -ApplicationId $applicationId ` -CertificateThumbprint $certThumbprint ``` ## Powershell / Password ``` $tenantId = "942023a6-efbe-4d97-a72d-532ef7337595" $applicationId = "4bc204cb-3282-43b1-aa1f-960f5faa4b23" $password = "shdfhskjfskhfkjh" Login-AzureRmAccount ` -ServicePrincipal ` -TenantId $tenantId ` -ApplicationId $applicationId ` -Credential $(New-Object -TypeName System.Management.Automation.PSCredential ` -ArgumentList $applicationId, ` $(ConvertTo-SecureString -Force -AsPlainText $password)) ``` ## Azure XPLAT CLI / X509 ```bash #!/bin/bash cert=$(openssl x509 -in "C:\certificates\examplecert.pem" -fingerprint -noout | \ sed 's/SHA1 Fingerprint=//g' | sed 's/://g') tenantId=$(azure account show -s --json | jq '.[0].tenantId' | \ sed -e 's/^"//' -e 's/"$//') appId=$(azure ad app show --search exampleapp --json | jq '.[0].appId' | sed -e 's/^"//' -e 's/"$//') azure login \ --service-principal \ --tenant "$tenantId" \ -u "$appId" \ --certificate-file C:\certificates\examplecert.pem \ --thumbprint "$cert" ``` ## Azure XPLAT CLI / Password * [Authenticate service principal with password - Azure CLI](https://github.com/Azure/azure-content/blob/master/articles/resource-group-authenticate-service-principal.md#authenticate-service-principal-with-password---azure-cli) ```bash azure login --service-principal --tenant "" -u "" -p "" ``` In my customer engagements, I usually push early for deployment automation of some sort. My preferred way to deploy to Azure is using Azure Resource Manager JSON Templates, alongside with developer-side automated scripts. Personally I also appreciate the notion of Service Principals, i.e. using "strong" credentials such as an X.509 Certificate to authenticate to Azure Resource Manager (ARM) API. In order to make it a bit more interesting, this article uses the "Microsoft Azure Germany" environment, instead of the 'regular' Azure. ### Registering Azure Germany under the hood When you install the latest Powershell for Azure (v1.5.0 at time of this writing), the command `Get-AzureEnvironment | select Name` should look like this: ``` PS C:\> Get-AzureEnvironment | select Name Name ---- AzureCloud AzureChinaCloud AzureUSGovernment AzureGermanCloud ``` The last line `AzureGermanCloud` indicates that Powershell already knows the specific management endpoints for Germany. *If* you do not have that, you might consider re-installing the Powershell module ``` # Install the Azure Resource Manager modules from the PowerShell Gallery Install-Module AzureRM Install-AzureRM Install-Module Azure Import-AzureRM Import-Module Azure ``` For the `azure-cli` side of things, the output of `azure account env list` should look like this: ``` PS C:\> azure account env list info: Executing command account env list data: Name data: ----------------- data: AzureCloud data: AzureChinaCloud data: AzureUSGovernment data: AzureGermanCloud info: account env list command OK ``` *If* you miss that last line, you can add the environment yourself: ``` azure account env add ^ --environment AzureGermanCloud ^ --portal-url http://portal.microsoftazure.de/ ^ --publishing-profile-url https://manage.microsoftazure.de/publishsettings/index ^ --management-endpoint-url https://management.core.cloudapi.de/ ^ --resource-manager-endpoint-url https://management.microsoftazure.de/ ^ --gallery-endpoint-url https://gallery.cloudapi.de/ ^ --active-directory-endpoint-url https://login.microsoftonline.de ^ --active-directory-resource-id https://management.core.cloudapi.de/ ^ --active-directory-graph-resource-id https://graph.cloudapi.de/ ^ --storage-endpoint-suffix .core.cloudapi.de ^ --key-vault-dns-suffix .vault.microsoftazure.de ^ --sql-server-hostname-suffix .database.cloudapi.de ``` ### Setup of a Service Principal in Azure Active Directory (AAD) The following Powershell script can be used to 1. Login interactively to Azure 2. Create a new application in Azure Active Directory. An application is a process which is cryptographically known to Azure AD (AAD). 3. Promote that application to become a service principal, i.e. giving it the right to request authN tokens from AAD. 4. Registering that new service principal as a `Contributor` to my Azure Subscription. #### Loggin in interactively ``` azure login -e AzureGermanCloud -u {username} azure login --environment AzureGermanCloud --user chgeuer@msftger.onmicrosoft.de --password XXX ``` #### A few variables to start with The initial log-in to Azure Germany happens with a regular Azure AD user, in my case that's `chgeuer@msftger.onmicrosoft.de`. ``` $subscriptionId = "deadbeef-fb63-43e6-afa2-d832f709f700" $tenantId = "deadbeef-e2bf-48c0-b025-23e47c410293" $userName = "chgeuer@msftger.onmicrosoft.de" $environmentName = "AzureGermanCloud" ``` #### Get the user's interactive password into the Powershell environment ``` $cred = Get-Credential ` -UserName $userName ` -Message "Login $userName to $environmentName" ``` #### Login to Azure with the interactive credential ``` Add-AzureRmAccount ` -EnvironmentName $environmentName ` -Tenant $tenantId ` -Credential $cred Login-AzureRmAccount ` -EnvironmentName $environmentName ` -TenantId $tenantId ` -SubscriptionId $subscriptionId ` -Credential $cred ``` #### Register the application In order to authenticate to Azure later, I want my service principal to use an X.509 Certificate. You can just bake yourself an own one using `makecert.exe` if you like. In my case, I saved a copy of the actual certificate on my local harddisk, which I then read into Powershell: ``` $certificateFile = "D:\credentials\azure-work\CN_Lenovo W530 Cert Christian.cer" $certOctets = Get-Content -Path $certificateFile -Encoding Byte $credValue = [System.Convert]::ToBase64String($certOctets) $cer = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 ` -ArgumentList @(,[System.Byte[]]$certOctets) ``` #### Create the Azure AD application Each application must have a name and a URL. In case your application is an actual web application, that URL would correspond to the real web site address. In my case, that's just some non-existent dummy URL: ``` $appName = "Service Principal Lenovo my Laptop $($userName)" $dummyUrl = "http://some-domain.com/whatever" $application = New-AzureRmADApplication ` -DisplayName $appName ` -HomePage $dummyUrl ` -IdentifierUris $dummyUrl ` -KeyType AsymmetricX509Cert ` -KeyValue $credValue ``` #### Promote the app to become a service principal As part of a larger script, you should pause execution for a few seconds, as it might take 1-2 seconds for that service principal information to propagate through AAD. ``` New-AzureRmADServicePrincipal ` -ApplicationId $application.ApplicationId Start-Sleep -Seconds 2 ``` #### Tell Azure that the service principal can manage my subscription ``` New-AzureRmRoleAssignment ` -ServicePrincipalName $application.ApplicationId ` -RoleDefinitionName Contributor Write-Host "Login like this: " Write-Host "" Write-Host "Login-AzureRmAccount \`" Write-Host " -ServicePrincipal \`" Write-Host " -TenantId '$($tenantId)' \`" Write-Host " -ApplicationId '$($application.ApplicationId)' \`" Write-Host " -CertificateThumbprint '$($cer.Thumbprint)' \`" Write-Host " -EnvironmentName 'AzureGermanCloud'" ``` ### Use that service principal to log-in to Azure #### Use that service principal to log-in to Azure using Powershell The following code assumes that you imported the certificate into your Windows Certificate store. As you can see, the `CurrentUser\My` certificate store contains the X509 cert, and I also own the private key: ``` Get-ChildItem Cert:\CurrentUser\My | ` where { $_.Thumbprint -eq "B8789A48A020FB1F5589C9ACAF63A4EBFFF5FA1C" } | ` select ThumbPrint,Subject,HasPrivateKey ``` Output is ``` Thumbprint Subject HasPrivateKey ---------- ------- ------------- B8789A48A020FB1F5589C9ACAF63A4EBFFF5FA1C CN=Lenovo W530 Cert Christian True ` ``` With this information I can now login with the service principal's identity: ``` Login-AzureRmAccount ` -ServicePrincipal ` -TenantId 'deadbeef-e2bf-48c0-b025-23e47c410293' ` -ApplicationId 'deadbeef-0980-46a6-a7fa-7ca8845aaca1' ` -CertificateThumbprint 'B8789A48A020FB1F5589C9ACAF63A4EBFFF5FA1C' ` -EnvironmentName 'AzureGermanCloud' ``` Output is ``` Environment : AzureGermanCloud Account : deadbeef-0980-46a6-a7fa-7ca8845aaca1 TenantId : deadbeef-e2bf-48c0-b025-23e47c410293 SubscriptionId : deadbeef-fb63-43e6-afa2-d832f709f700 SubscriptionName : MSFTGER Test Subscription CurrentStorageAccount : ``` #### Use that service principal to log-in to Azure using node.js / azure-cli The same thing can be done using the azure-cli. The main difference is that the azure-cli isn't aware of Windows certificate stores, but still requires access to the certificate's private key. In this case, the private key is in a PEM-file on my laptop's harddisk: ``` azure config mode arm azure login ^ --environment AzureGermanCloud ^ --service-principal ^ --tenant "deadbeef-e2bf-48c0-b025-23e47c410293" ^ --username "deadbeef-0980-46a6-a7fa-7ca8845aaca1" ^ --thumbprint "B8789A48A020FB1F5589C9ACAF63A4EBFFF5FA1C" ^ --certificate-file "D:\credentials\azure-work\CN_Lenovo W530 Cert Christian.pem" ^ --json ^ --verbose ``` Output is ``` info: Executing command login verbose: Authenticating... info: Added subscription MSFTGER Test Subscription info: login command OK ``` ## Add mgmt cert to Azure Germany via ASM API ``` Invoke-WebRequest ` -uri https://management.core.cloudapi.de/$subID/certificates ` -Method Post ` -Headers @{"x-ms-version"="2012-03-01"} ` -Certificate $authcert ` -Body $xml.outerxml ` -ContentType "application/xml" ``` ## Update Azure CLI 2.0 (Python) and change cloud ``` pip install --upgrade azure-cli az cloud set --name AzureGermanCloud az cloud set --name AzureCloud set password=superSecret123! set subscriptionName=chgeuer-work call az cloud set --name AzureCloud call az login call az account list call az account set --subscription %subscriptionName% call az ad app create --display-name "Christian SP Demo" --homepage "http://foo" --identifier-uris "http://foo2" --key-type Password --password %password% | jq .appId > appid.txt set /p appId=<"appId.txt" call az ad sp create --id %appId% | jq .objectId > spObjectId.txt set /p spObjectId=<"spObjectId.txt" call az ad sp list call az account show | jq .id > subscriptionId.txt set /p subscriptionId=<"subscriptionId.txt" call az role assignment create --role Contributor --assignee %spObjectId% --scope "/subscriptions/%subscriptionId%" ``` ## Use fiddler or mitm * ``` # Where does fiddler listen export HTTP_PROXY=http://127.0.0.1:8888 export HTTPS_PROXY=http://127.0.0.1:8888 # For the old Azure CLI v1, command name `azure` (the Node.js-based one) export NODE_TLS_REJECT_UNAUTHORIZED=0 # For the new Azure CLI v2, command name `az` (the Python-based one) export ADAL_PYTHON_SSL_NO_VERIFY=1 export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 ``` ## Signin via service principal and debug a session ``` # Have fiddler listen on :8888 export HTTP_PROXY=http://127.0.0.1:8888 export HTTPS_PROXY=http://127.0.0.1:8888 # For the old xplat cli (node.js based `azure` command line client) export NODE_TLS_REJECT_UNAUTHORIZED=0 # For the new Azure `az` CLI (python based) export ADAL_PYTHON_SSL_NO_VERIFY=1 export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 export AZURE_SERVICEPRINCIPAL_APPID=deadbeef-1234-5678-abcd-fabf7cf9368e export AZURE_SERVICEPRINCIPAL_PASSWORD=SuperSecret123.- export AZURE_TENANTID=942023a6-efbe-4d97-a72d-532ef7337595 export AZURE_SUBSCRIPTION_ID=724467b5-bee4-484b-bf13-d6a5505d2b51 az cloud set --name AzureCloud az login --service-principal --tenant $AZURE_TENANTID --username $AZURE_SERVICEPRINCIPAL_APPID --password $AZURE_SERVICEPRINCIPAL_PASSWORD az account set --subscription $AZURE_SUBSCRIPTION_ID az vm list ``` ### See the latest accessToken ```bash cat ~/.azure/accessTokens.json | jq -r .[-1].refreshToken ``` ## Set Windows Proxy information ``` # https://martin.hoppenheit.info/blog/2015/set-windows-proxy-with-powershell/ Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyServer -Value "http=127.0.0.1:8888;https=127.0.0.1:8888" Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyEnable -Value 1 $(Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings").ProxyServer $(Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings").ProxyEnable Remove-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyServer Set-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyEnable -Value 0 ``` # Working with the REST API Working with the REST APIs Sometimes I need a zero-install way to interact with Azure. I have no specific Azure utilities at hand, no Python, no nothing. Usually, Azure management is done using PowerShell, the [az cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) or, if you want raw REST calls, the [armclient](https://github.com/projectkudu/ARMClient). But for my customer, even can be too much ceremony. So the question was how can I get going with purely `bash`, [`cURL`](https://curl.haxx.se/) and [`jq`](https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64) for JSON parsing, and potentially [`yq` and `xq`](https://github.com/jeffbr13/xq) for YAML/XML parsing. ```bash #!/bin/bash # Proper install sudo apt-get -y install jq sudo pip install yq # YOLO curl \ --silent \ --url https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \ --location \ --output ./jq chmod +x ./jq sudo mv ./jq /usr/local/bin sudo chown root.root /usr/local/bin/jq ``` If you're running inside a VM, with Managed Identity enabled, you can easily fetch a token. But unfortunately the VM wasn't authorized to hit the resource I care about. Next stop service principals. Problem is customer's AD admin team running a tough regime, and don't hand out service principals. So ultimately, how can I get my actual AAD user identity avail in the shell? In the end, all I need is a bearer token. Let's dive right in: ## A few variables first I want to authN against 'my' Azure AD tenant, and want to hit the Azure ARM REST API. ## Doing a device login (AAD v2) For the full user login, i.e. device authN, here's what happens under the hood: The code needs to fetch a device code, and then use that code to poll and validate whether the user authenticated. {% hint style="info" %} If you wanna snoop on cURL's requests with something like [fiddler](https://www.telerik.com/fiddler), you should add this `--proxy http://127.0.0.1:8888/ --insecure` to the calls. {% endhint %} ```bash #!/bin/bash # --proxy http://127.0.0.1:8888/ --insecure \ aadTenant="chgeuerfte.onmicrosoft.com" # resource="https://management.azure.com/.default" resource="https://storage.azure.com/.default" deviceResponse="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/devicecode" \ --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \ --data-urlencode "scope=${resource}" \ )" device_code="$(echo "${deviceResponse}" | jq -r ".device_code")" sleep_duration="$(echo "${deviceResponse}" | jq -r ".interval")" access_token="" # # On WSL, copy code to Windows clipboard and launch the site # echo "$( echo "${deviceResponse}" | jq -r ".user_code" )" | iconv -f utf-8 -t utf-16le | clip.exe cmd.exe /C "start $( echo "${deviceResponse}" | jq -r ".verification_uri" )" # # Poll for result # while [ "${access_token}" == "" ] do tokenResponse="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/{aadTenant}/oauth2/v2.0/token" \ --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:device_code" \ --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \ --data-urlencode "device_code=${device_code}" \ )" if [ "$(echo "${tokenResponse}" | jq -r ".error")" == "authorization_pending" ]; then echo "$(echo "${deviceResponse}" | jq -r ".message")" sleep "${sleep_duration}" else access_token="$(echo "${tokenResponse}" | jq -r ".access_token")" echo "User authenticated" fi done echo "${access_token}" echo "$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${access_token}" )" | jq cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )" ``` ## Using a service principal (AAD v1) Assuming we have a 'real' service principal, we can do this: ```bash #!/bin/bash aadTenant="chgeuerfte.onmicrosoft.com" SAMPLE_SP_APPID="*** put your service principal application ID here ***" SAMPLE_SP_KEY="*** put your service principal application secret here ***" # resource="https://management.azure.com/" resource="https://storage.azure.com/" access_token="$(curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${aadTenant}/oauth2/token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_id=${SAMPLE_SP_APPID}" \ --data-urlencode "client_secret=${SAMPLE_SP_KEY}" \ --data-urlencode "resource=${resource}" \ | jq -r ".access_token")" resource="https://storage.azure.com/.default" access_token="$(curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/token" \ --data-urlencode "response_type=token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_id=${SAMPLE_SP_APPID}" \ --data-urlencode "client_secret=${SAMPLE_SP_KEY}" \ --data-urlencode "scope=${resource}" \ | jq -r ".access_token")" ``` ### Create an AAD app with a specified password ```bash #!/bin/bash aadTenant="xxx.onmicrosoft.com" display_name="something" client_secret="secret123.-" client_id="$( az ad app create --display-name "${display_name}" --password "${client_secret}" | jq -r ".appId" )" echo "client_id: ${client_id}" resource="https://storage.azure.com/" access_token="$(curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${aadTenant}/oauth2/token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_id=${client_id}" \ --data-urlencode "client_secret=${client_secret}" \ --data-urlencode "resource=${resource}" \ | jq -r ".access_token")" cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )" ``` ### The underlying GraphAPI call for creating an app with a given password ```rest POST https://graph.windows.net/${aadTenant}/applications?api-version=1.6 HTTP/1.1 Authorization: Bearer eyJ0eXA... Content-Type: application/json; charset=utf-8 { "displayName": "${display_name}", "availableToOtherTenants": false, "passwordCredentials": [ { "keyId": "8b03e38a-9e92-4c35-beb0-04a40252722d", "startDate": "2020-11-16T13:40:38.834354Z", "endDate": "2021-11-15T13:40:38.834354Z", "value": "${client_secret}" } ] } ``` Even though [this](https://docs.microsoft.com/en-us/graph/api/application-post-applications?view=graph-rest-1.0\&tabs=http) says that *Adding passwordCredential when creating applications is not supported.*, and the sample shows an empty `"passwordCredentials": []` array, the call to `az ad app create --display-name "${display_name}" --password "${client_secret}"` exactly populates that property. * [application: addPassword](https://docs.microsoft.com/en-us/graph/api/application-addpassword?view=graph-rest-1.0\&tabs=http) ## Using managed VM identity (running inside an Azure VM) (AAD v1) ```bash #!/bin/bash resource="https://management.azure.com/" #resource="https://storage.azure.com/" access_token="$( curl --silent --get \ --url "http://169.254.169.254/metadata/identity/oauth2/token" \ --data-urlencode "api-version=2018-02-01" \ --data-urlencode "resource=${resource}" \ --header "Metadata: true" \ | jq -r '.access_token' \ )" ``` ## Fetch the subscription ID, from the Azure VM's instance metadata endpoint ```bash #!/bin/bash subscriptionId="$(curl --silent --get \ --url "http://169.254.169.254/metadata/instance" \ --data-urlencode "api-version=2017-08-01" \ --header "Metadata: true" \ | jq -r ".compute.subscriptionId")" ``` ## Invoke the ARM API, for example with a listing of resource groups ```bash #!/bin/bash subscriptionId="724467b5-bee4-484b-bf13-d6a5505d2b51" # --proxy http://127.0.0.1:8888/ --insecure \ curl --silent --get \ --url "https://management.azure.com/subscriptions/${subscriptionId}/resourcegroups" \ --data-urlencode "api-version=2018-05-01" --header "Authorization: Bearer ${access_token}" | \ jq -r ".value[].name" ``` ## Fetching a secret from Azure KeyVault using a managed identity This little script demonstrates how to fetch a secret from an Azure KeyVault, using a managed identity on an Azure VM. Just adapt `key_vault_name` and `secret_name` accordingly, and of course ensure that the managed identity can actually read the secret. ```bash #!/bin/bash get_secret_from_keyvault() { local key_vault_name=${1} local secret_name=${2} resource="https://vault.azure.net" access_token="$( curl --silent --get \ --url "http://169.254.169.254/metadata/identity/oauth2/token" \ --data-urlencode "api-version=2018-02-01" \ --data-urlencode "bypass_cache=true" \ --data-urlencode "resource=${resource}" \ --header "Metadata: true" \ | jq -r '.access_token' \ )" apiVersion="7.0" # # Fetch the latest version # secretVersion="$(curl --silent --get \ --url "https://${key_vault_name}.vault.azure.net/secrets/${secret_name}/versions" \ --data-urlencode "api-version=${apiVersion}" \ --header "Authorization: Bearer ${access_token}" \ | jq -r '.value | sort_by(.attributes.created) | .[-1].id' \ )" # # Fetch the actual secret's value # secret="$( curl --silent \ --url "${secretVersion}?api-version=${apiVersion}" \ --header "Authorization: Bearer ${access_token}" \ | jq -r '.value' )" echo "${secret}" } echo "The secret is $(get_secret_from_keyvault "chgeuerkeyvault" "secret1")" ``` ## Force the instance metadata service to skip the token cache Use the `bypass_cache=true` parameter when fetching a token from IMDS. ## Shutdown a VM, quite radically (skip graceful shutdown, just turn it off) The `skipShutdown=true` below is useful in STONITH scenarios. ```bash #!/bin/bash ... subscriptionId="..." resourceGroup="myrg" vmName="somevm" curl --silent --include \ --request POST \ --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/powerOff" \ --data-urlencode "api-version=2019-03-01" \ --data-urlencode "skipShutdown=true" --header "Authorization: Bearer ${access_token}" \ --header "Content-Length: 0" ``` ## Talking to Azure Blob Storage ```bash #!/bin/bash json_identity="$( \ curl --silent \ --url "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com%2F" \ --header Metadata:true \ | jq -r ".access_token")" storage_account="tmp1diag889" host="hybris" storageApi="2019-12-12" # # Download file # curl \ --url "https://${storage_account}.blob.core.windows.net/${host}/${file}" \ --header "Authorization: Bearer ${json_identity}" \ --header "x-ms-version: ${storageApi}" \ --header "x-ms-blob-type: BlockBlob" \ --remote-name # # List Container # # Unfortunately, the REST API returns XML, so 'jq' alone isn't helpful, need to XPath into XML here # # curl \ --silent \ --url "https://${storage_account}.blob.core.windows.net/${host}/?comp=list&restype=container" \ --header "Authorization: Bearer ${json_identity}" \ --header "x-ms-version: ${storageApi}" \ | xq '.EnumerationResults.Blobs[]' \ | jq -r '.[] | .Name' # # An alternative approach to process the XML response, but ... # # XML processing with Regexes guarantees us a place in hell, but we don't need 'pip install yq' # curl \ --silent \ --url "https://${storage_account}.blob.core.windows.net/${host}/?comp=list&restype=container" \ --header "Authorization: Bearer ${json_identity}" \ --header "x-ms-version: ${storageApi}" \ | sed -e 's||\n|g' -e 's||\n|g' \ | egrep "^" \ | sed -e 's|||g' -e 's|||g' ``` ## Uploading a blob ```bash filename="1.txt" curl \ --request PUT \ --url "https://${storageAccountName}.blob.core.windows.net/${containerName}/${filename}" \ --header "x-ms-version: 2019-12-12" \ --header "x-ms-blob-type: BlockBlob"\ --header "x-ms-blob-content-disposition: attachment; filename=\"${filename}\"" \ --header "Content-Type: application/binary" \ --header "Authorization: Bearer ${access_token}" \ --header "Content-MD5: $( md5sum "${filename}" | awk '{ print $1 }' | xxd -r -p | base64 )" \ --upload-file "${filename}" ``` ## Commit suicide using managed identity ```bash #!/bin/bash resource="https://management.azure.com/" msiVersion="2018-02-01" access_token="$( curl --silent --get \ --url "http://169.254.169.254/metadata/identity/oauth2/token" \ --data-urlencode "api-version=${msiVersion}" \ --data-urlencode "resource=${resource}" \ --header "Metadata: true" \ | jq -r '.access_token' \ )" imdsVersion="2021-02-01" subscriptionId="$(curl --silent --get \ --url "http://169.254.169.254/metadata/instance" \ --data-urlencode "api-version=${imdsVersion}" \ --header "Metadata: true" \ | jq -r '.compute.subscriptionId' \ )" resourceGroup="$(curl --silent --get \ --url "http://169.254.169.254/metadata/instance" \ --data-urlencode "api-version=${imdsVersion}" \ --header "Metadata: true" \ | jq -r '.compute.resourceGroupName' \ )" vmName="$(curl --silent --get \ --url "http://169.254.169.254/metadata/instance" \ --data-urlencode "api-version=${imdsVersion}" \ --header "Metadata: true" \ | jq -r '.compute.name' \ )" # # Stop and skip shutdown sequence. STONITH # virtualMachineARMVersion="2021-03-01" curl --silent \ --request POST \ --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/powerOff?api-version=${virtualMachineARMVersion}&skipShutdown=true" \ --header "Authorization: Bearer ${access_token}" \ --data "" # # Properly deallocate # curl --silent \ --request POST \ --url "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroup}/providers/Microsoft.Compute/virtualMachines/${vmName}/deallocate?api-version=${virtualMachineARMVersion}" \ --header "Authorization: Bearer ${access_token}" \ --data "" ``` # Tracing HTTP requests with Fiddler Tracing HTTP requests with Fiddler When using the `az` CLI, it sometimes is helpful to understand which APIs it calls 'under the hood', i.e. so see which Azure REST APIs are called. On Windows, you can use a tool like Fiddler to inspect outgoing HTTP(s) requests. Fiddler does this by 'launching a man-in-the-middle attack' against the applications, by injecting a self-signed X.509 certificate into the Windows cert store, and then pretending to be the external web site. However, different applications read the HTTP/HTTPS proxy information from different locations: * In .NET, one can set a global (static) `System.Net.WebRequest.DefaultWebProxy` variable * Windows applications (like web browsers) check the Windows registry, in particular the `HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyServer` values * Unix-style applications (like the Python-based `az` utility) check the `HTTP_PROXY` and `HTTPS_PROXY` environment variables * Other applications require you to explicitly specify the proxy to use via command line, such as `curl` supporting the `--proxy ...` and `--insecure` args. On the security side, you need to convince the apps to accept Fiddler's self-cooked TLS cert: * Windows apps check the X509 cert chain, using the `MACHINE\root` store * `curl` can be convinced using the `--insecure` arg to ignore certificate validity problems * The `az` CLI needs the `ADAL_PYTHON_SSL_NO_VERIFY` and `AZURE_CLI_DISABLE_CONNECTION_VERIFICATION` environment variables to be set, to skip checking the server's TLS cert. ## Setting the proxy in Powershell Here's how to set a gazillion different settings to make sure Fiddler is used. ```powershell $fiddlerHost = "127.0.0.1" $fiddlerPort = "8888" $fiddlerUrl = "http://$($fiddlerHost):$($fiddlerPort)" # # This ensures the .NET code uses the proxy # [System.Net.WebRequest]::DefaultWebProxy = New-Object System.Net.WebProxy ( New-Object System.Uri( $fiddlerUrl ), $true) # # This ensures Windows apps (Edge, Teams, Outlook, Windows) use the proxy # Set-ItemProperty ` -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" ` -Name ProxyServer ` -Value "http=$($fiddlerHost):$($fiddlerPort);https=$($fiddlerHost):$($fiddlerPort)" Set-ItemProperty ` -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" ` -Name ProxyEnable ` -Value 1 # # This ensures Python code (which looks at environment variables) uses the proxy # $Env:HTTP_PROXY = $fiddlerUrl $Env:HTTPS_PROXY = $fiddlerUrl # # This ensures the `az` CLI doesn't complain when we launch a man-in-the-middle with # a self-issued X509 cert # $Env:ADAL_PYTHON_SSL_NO_VERIFY = '1' $Env:AZURE_CLI_DISABLE_CONNECTION_VERIFICATION = '1' # # And the `--insecure` also calms curl's desire to be secure # C:\Users\chgeuer\bin\curl.exe --proxy $fiddlerUrl --insecure ` --silent ` "https://www.microsoft.com" ``` ### Checking the current settings ```powershell $(Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings").ProxyServer $(Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings").ProxyEnable ``` ### Deleting the registry entry again ```powershell Remove-ItemProperty ` -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" ` -Name ProxyServer Set-ItemProperty ` -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" ` -Name ProxyEnable -Value 0 ``` # Upload a file from bash This short demo shows how one can upload a file into Azure storage, using just basic utilities (i.e. no Microsoft-provided tooling). ## Requirements To do so, we need a few requirements satisfied: * `bash`, `curl` and `jq` * The `cURL` utility must be locally installed * The `jq` utility () must be locally installed * You need to have an Azure storage account * In that storage account, there needs to be a storage container already created * You need to have the `Storage Blob Data Contributor` role assigned * Obviously you need some local file * Given that we'll upload the file in a single shot, it must be less than [5000 MiB](https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs) in size ## How it works 1. The script attempts a device code signin to the given Azure AD tenant, requesting a token for Azure storage 2. Once the script outputs the device code, you need to open `https://microsoft.com/devicecode` in a web browser and punch in the device code 3. The script locally computes the MD5 hash of the file (using `md5sum`, `awk`, `xxd` and `base64`), to ensure upload errors will be catched 4. The script uploads the file to blob storage. ```bash #!/bin/bash aadTenant="chgeuerfte.onmicrosoft.com" storageAccountName="chgeuer" containerName="public" filename="/mnt/c/Users/chgeuer/Desktop/dump.md" resource="https://storage.azure.com/.default" az_cli_client_id="04b07795-8ddb-461a-bbee-02f9e1bf7b46" client_id="${az_cli_client_id}" deviceResponse="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/devicecode"\ --data-urlencode "client_id=${client_id}" \ --data-urlencode "scope=${resource}" \ )" device_code="$(echo "${deviceResponse}" | jq -r ".device_code")" sleep_duration="$(echo "${deviceResponse}" | jq -r ".interval")" access_token="" while [[ "${access_token}" == "" ]] do tokenResponse="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/{aadTenant}/oauth2/v2.0/token" \ --data-urlencode "client_id=${client_id}" \ --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:device_code" \ --data-urlencode "device_code=${device_code}" \ )" if [[ "$( echo "${tokenResponse}" | jq -r ".error" )" == "authorization_pending" ]]; then echo "$( echo "${deviceResponse}" | jq -r ".message" )" sleep "${sleep_duration}" else access_token="$( echo "${tokenResponse}" | jq -r ".access_token" )" echo "User authenticated" fi done curl \ --request PUT \ --url "https://${storageAccountName}.blob.core.windows.net/${containerName}/$( basename "${filename}" )" \ --header "x-ms-version: 2019-12-12" \ --header "x-ms-blob-type: BlockBlob"\ --header "x-ms-blob-content-disposition: attachment; filename=\"$( basename "${filename}" )\"" \ --header "Content-Type: application/binary" \ --header "Authorization: Bearer ${access_token}" \ --header "Content-MD5: $( md5sum "${filename}" | awk '{ print $1 }' | xxd -r -p | base64 )" \ --upload-file "${filename}" echo "File uploaded to https://${storageAccountName}.blob.core.windows.net/${containerName}/$( basename "${filename}" )" ``` # Azure CLI ## Account management defaults ### Set subscription by name ```bash subscriptionName="chgeuer-work" az account set --subscription "${subscriptionName}" ``` ### Get subscription ID of current account ```bash subscriptionID="$( az account show --output json | jq -r ".id" )" ``` ## Create an ARM deployment and pass in a parameter of type 'object' The file `definition.json` contains a JSON object, which is passed as-is into the ARM deployment: ```bash resourceGroup="foo" logicAppName="someapp" deploymentResult="$( az group deployment create \ --resource-group "${resourceGroup}" \ --mode Incremental \ --template-file ./azuredeploy-minimal.json \ --parameters logicAppName="${logicAppName}" \ --parameters logicAppDefinition="@./definition.json" )" ``` The template has a corresponding `type=object` parameter: ```javascript { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "logicAppName": { "type": "string", "defaultValue": "..." }, "logicAppDefinition": { "type": "object" } }, ... } ``` ## Run the Docker container ```bash docker pull mcr.microsoft.com/azure-cli:latest docker run --mount "src=${HOME}/.azure,target=/root/.azure,type=bind" -it --rm mcr.microsoft.com/azure-cli:latest ``` # terraform ![Azure loves Terraform](/files/JILVpqzSVBdIp2Emq65t) ## ARM and terraform - Side by Side | [ARM Templates](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/) | Terraform | | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | | JSON w/ comments | HCL | | [Parameters](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-parameters) | Variables | | [Variables](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-variables) | Local variables | | Resources | Resources | | [Functions](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions) | [Functions](https://www.terraform.io/docs/configuration/functions.html) | | [Nested templates](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/linked-templates) | Modules | | Explicit | Automatic | | Refer by reference or resourceId | Refer by resource or data source | ## Providers * Azure (i.e. Azure *Resource* Manager) * ~~the 'old' Azure *Service* Management (ASM) provider~~ * Azure Active Directory (AAD) * Azure Stack (on-premises) ## Authentication * AZ CLI - if environment has `az` CLI installed, re-use existing session * Azure Managed Identity (on Azure compute resource) * Azure Service Principals (with client secrets or X.509 certs) ## The [`"azurerm"`](https://www.terraform.io/docs/providers/azurerm/index.html) Provider (Azure Resource Manager) * * ``` provider "azurerm" { version = "~> 1.40" alias = "networking" subscription_id = var.subscription_id client_id = var.client_id client_secret = var.client_secret } ``` ## The [`"azure_ad"`](https://www.terraform.io/docs/providers/azurerm/index.html) Provider (Azure Resource Manager) ``` provider "azure_rm" { version = "~> 0.7" subscription_id = var.subscription_id client_id = var.client_id client_secret = var.client_secret } ``` ### Azure-specific environment variables * `ARM_ENVIRONMENT` - `public`, `usgovernment`, `german`, `china` * `ARM_SUBSCRIPTION_ID` - Azure subscription ID * `ARM_TENANT_ID` - Azure AD tenant ID for service principal * `ARM_USE_MSI` - Use Managed Service Identity * `ARM_CLIENT_ID` - Service principal ID * `ARM_CLIENT_SECRET` - Service principal secret ## Remote state: the [`"azurerm"`](https://www.terraform.io/docs/backends/types/azurerm.html) backend Stores state in a blob, in a container, in an Azure storage account. ``` terraform { backend "azurerm" { resource_group_name = "longterm" storage_account_name = "chgeuer" container_name = "terraformstate" key = "demo2.tfstate" } } ``` ### Authenticating to remote state backend * Inherit authN info from outer environment, such as `az` CLI or service principal * `use_msi`: Managed identity within Azure Compute * `access_key`: The storage account's access key * `sas_token`: A 'shared access signature' token ```bash terraform init –backend-config="sas_token=gh67il=="` ``` Alternatively, Azure CosmosDB provides an etcd protocol head. ## Data Sources Many data sources, including * `azurerm` * [`"azurerm_subscriptions"`](https://www.terraform.io/docs/providers/azurerm/d/subscriptions.html): information about all the Subscriptions currently available * [`"azurerm_subscription"`](https://www.terraform.io/docs/providers/azurerm/d/subscription.html): information about an existing Subscription. * [`"azurerm_resource_group"`](https://www.terraform.io/docs/providers/azurerm/d/resource_group.html) * KeyVault, Networking, API Management, Compute, ... * `azuread` * [`"azuread_application"`](https://www.terraform.io/docs/providers/azuread/r/application.html) * [`"azuread_service_principal"`](https://www.terraform.io/docs/providers/azuread/r/service_principal.html) * Users, Groups, Roles, ... ## Azure Modules in the Terraform Registry ![Screenshot from aka.ms/tfmodules](/files/PW08dBGCadZW8OkrAMTW) ## ARM / Terraform Interoperability ### [`"azurerm_template_deployment"`](https://www.terraform.io/docs/providers/azurerm/r/template_deployment.html): use ARM templates within Terraform Example: ``` resource "azurerm_template_deployment" "logicapp" { name = "deployment-${formatdate("YYYY-MM-DD--hh-mm-ss", timestamp())}" resource_group_name = var.resource_group_name deployment_mode = "Incremental" template_body = file(local.arm_template_file) parameters = { "logicAppName" = var.logic_app_name "logicAppDefinition" = var.logic_app_definition } } ``` ### Terraform Resource Provider (RP) in Azure ARM * Private preview supporting three providers: Kubernetes, Cloudflare and Datadog * ![](/files/dYzDSMdwaaPDyy8AJOUG) ## Available environments * TF [installed](https://docs.microsoft.com/en-us/azure/terraform/terraform-cloud-shell) in the Azure Cloud Shell [shell.azure.com](https://shell.azure.com) * [Marketplace VM images w/ terraform and MSI](http://aka.ms/aztfmkt) * [VS Code Plugin for terraform](https://docs.microsoft.com/en-us/azure/terraform/terraform-vscode-extension) ![Screenshot from the Terraform VM marketplace image](/files/XZL1qdeW0tpPUlB9lYST) ## Learning resources for terraform on Azure * [aka.ms/tfhub -- docs.microsoft.com](https://docs.microsoft.com/en-us/azure/terraform/) * [Hashicorp Azure learning track](https://learn.hashicorp.com/terraform?track=azure#azure) * [chrismatteson/hashicorp\_azure\_training](https://github.com/chrismatteson/hashicorp_azure_training) and the [slides](https://chrismatteson.github.io/hashicorp_azure_training/#55) * [CardinalNow/TerraformWorkshop](https://github.com/CardinalNow/TerraformWorkshop) * [Source Code aka.ms/tfgit](http://aka.ms/tfgit) * [Using Azure DevOps pipelines to deploy via Terraform](https://www.azuredevopslabs.com/labs/vstsextend/terraform/) # Azure Logic Apps ![](/files/58eK776K4WHkix1Djro9) ## Simple web API to POST some XML and extract some information * [chgeuer/azure-snippets/logic-app-reading-xml](https://github.com/chgeuer/azure-snippets/tree/master/logic-app-reading-xml) # Azure Web Apps Azure Web Apps ## Determine the slot in which we're running ### Via incoming HTTP header If your code in an Azure Web App for Linux needs to determine in which deployment slot it's running, then the incoming `WAS-DEFAULT-HOSTNAME` HTTP header seems to be the only reliable way. Also confirmed [here](https://github.com/microsoft/ApplicationInsights-dotnet/issues/1368). When you're in the production slot, then the value looks like this: `someappname.azurewebsites.net`. When you created a slot called `stage1`, then this header is `someappname-stage1.azurewebsites.net`. The absence of the suffix points to the production slot, otherwise the suffix gives the user-chosen name. There's a second header called `X_SITE_DEPLOYMENT_ID`, but this one contains an identifier like `someappname__f375`, which isn't too helpful. ```php

Server

``` ### Via system-assigned managed identity Another, certainly much more clear way, could be using a system-assigned managed identity, assuming you assigned one to all deployment slots. If you fetch an `access_token`, then the `xms_mirid` claim in the JWT contains the real instance ID, such as * `"/subscriptions/.../resourcegroups/.../providers/Microsoft.Web/sites/someappname"` for the production slot, or * `"/subscriptions/.../resourcegroups/.../providers/Microsoft.Web/sites/someappname/slots/stage1"` for the `stage1` slot. ## Fetching a managed-identity `access_token` from PHP in an Azure Web App for Linux * Inside Azure Web Apps for Linux, you can't simply query the instance metadata endpoint, you need a special endpoint from an environment variable * Also api-version must be a special one ```php '2019-08-01', 'resource' => $resource); $url = $endpoint . '?' . http_build_query($params); $headers = array( 'Metadata: true', 'X-IDENTITY-HEADER: ' . $_ENV['IDENTITY_HEADER'] ); $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); $response = curl_exec($ch); curl_close ($ch); $response_json = json_decode($response); $access_token = $response_json->{'access_token'}; ?> See the JWT in https://jwt.ms ``` # Azure Python code snippets Azure Python code snippets ```python # Python SDK needs these for symmetric service principal authN os.environ['AZURE_TENANT_ID'] = '...' os.environ['AZURE_CLIENT_ID'] = '...' os.environ['AZURE_CLIENT_SECRET'] = '...' # Want to have all TLS traffic man-in-the-middle-intercepted by Fiddler os.environ['HTTP_PROXY'] = 'http://127.0.0.1:8888/' os.environ['HTTPS_PROXY'] = 'http://127.0.0.1:8888/' os.environ['ADAL_PYTHON_SSL_NO_VERIFY'] = '1' os.environ['CURL_CA_BUNDLE'] = '' # https://stackoverflow.com/questions/48391750/disable-python-requests-ssl-validation-for-an-imported-module credential = DefaultAzureCredential(logging_enable=true) ``` # SSH keys in ARM The Azure ARM system allows you to [upload](https://docs.microsoft.com/en-us/azure/virtual-machines/ssh-keys-portal) [ssh public keys](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-ssh-keys-detailed) as a first-class object in ARM. This sample illustrates how to create such an SSH public key, and also how to dynamically use it. During interactive VM creation in the portal, you can [dynamically select an existing public key](https://docs.microsoft.com/en-us/azure/virtual-machines/ssh-keys-portal) for your new VM. However, for template-based creation of a VM, you need to use the `reference()` ARM function to retrieve the value. The ARM schema for VMs doesn't currently allow you to refer to a key object, instead you must provide the literal SSH key value as a string to the VM, in the `.osProfile.linuxConfiguration.ssh.publicKeys[0].keyData` value. The following little Bicep sample demonstrates these two concepts: 1. Creating the `'Microsoft.Compute/sshPublicKeys'` ARM resource, as well as 2. dynamically retrieving it. For the sake of the example, I'm not really creating a VM, but just fetch the ssh public key and output it in the template. ```bicep param keyName string = 'chgeuer' param sshPublicKey string = 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChtYrL..... chgeuer@beam' resource mySshkey 'Microsoft.Compute/sshPublicKeys@2020-12-01' = { name: keyName location: resourceGroup().location properties: { publicKey: sshPublicKey } } var sshkeyId = '/subscriptions/${subscription().subscriptionId}/resourceGroups/${resourceGroup().name}/providers/Microsoft.Compute/sshPublicKeys/${keyName}' var sshkeyId2 = mySshkey.id var apiversion = '2020-12-01' output key string = reference(sshkeyId, apiversion).publicKey output key2 string = reference(mySshkey.id, apiversion).publicKey ``` # Minimal "Azure AD Workload identity federation" The Azure Active Directory team recently released a new preview feature, called [workload identity federation](https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation), which "allows you to access Azure AD-protected resources, without needing to manage secrets (for supported scenarios)". Not having secrets sounds cool, but how does it work? > I'm using the terms "Azure Active Directory", "Azure AD" and "AAD" synonymously in this article. My colleague Arsen Vladimirsky published a great [blog article with a 30-min video](https://arsenvlad.medium.com/azure-active-directory-workload-identity-federation-with-external-oidc-idp-4f06c9205a26) to describe more how it works in a production environment, using an external (non-AAD) Identity Provides (Auth0 in his sample). The article features many moving parts, Postman, the Azure portal, the Auth0 web site, showing that in production, a lot of things just happen 'in the back'. I wanted to understand how the absolute minimal external setup would have to look like for this to work, explore along the way how workload identity federation works, and do some funky `bash` command line things. ## What you will find in this article * A simple demo for Azure Active Directory Workload Identity Federation * Some simple OpenID Connect interactions * See how you can create JSON Web Tokens (JWT) in the command line in bash * See one (IMHO) good pattern how to create JSON structures in the command line * A bunch of REST calls against Azure APIs ## Overview In this demo, we will simulate to be our own IdP (identity provider), by generating our own JSON Web Tokens, which (hopefully) AzureAD would be accepting. So we need to become proficient using `OpenSSL` on the command line for cryptographic operations, just know enough about JWT (JSON Web Tokens) and JWS (JSON Web Signing) to be dangerous, use `jq` to manipulate JSON locally, and of course our all-time friend `cURL` to finally talk to Azure AD. Concretely, we'll do this: 1. *Use `OpenSSL` to generate a plain RSA key*. Of course, in today's world, X.509 certificates are the hotness, but maybe just a dumb RSA key will be enough 😏. This RSA key should be enough to pretend to be a proper identity provider, so we can generate tokens, which AAD is willing to accept. 2. *Spin up an 'Identity Provider' in Azure Blob Storage*. Wait, what, an IdP in STORAGE??? Yes. We will obviously not run any code in storage, but we can upload just enough bits into blob storage, to convince AAD that there's a real token-issuing IDP running on `https://whatever.blob.core.windows.net` 🤯. 3. *Configure an existing application in Azure AD to allow sign-in via 'Workload Identity Federation'*. We'll be adding a `federatedIdentityCredential` to the application's credential section, telling Azure AD that this application signs in with a bearer token from our external IdP. 4. *Pretend to be an IdP, and generate a minimal (and hopefully valid) self-issued JWT*. 5. *Do a client-credential grant dance with AAD, exchanging our self-issued JWT against a real production token*, which we can e.g. use to talk to the Azure Management API or some other service. ## Follow step-by-step The rest of this article interleaves various `bash` shell commands, which you can 1:1 put into practice yourself, by copy/pasting (and executing 🤓) them in your own environment. > Follow-along at your own risk... You will see various places, where I define variables, such as `storage_account="chgeuer"`. You will obviously need to tweak the values, to match your own environment. The more complex commands are split across multiple lines, so please note the `\` symbol at the end of the lines as indicator for continuation in the next line. ### Packages needed You will need a few packages installed, namely * `OpenSSL`, [`curl`](https://curl.se/), `sed`, `bash`, `iconv`, which are usually pre-installed on your Unix/Linux system of choice. * [`jq`](https://stedolan.github.io/jq/), in order to manipulate JSON, which you can install using `sudo apt-get install jq` * For quickly peeking into JWT tokens, I'm calling into `cmd.exe` to launch a web browser on the Windows site, which is primarily for convenience, and isn't strictly necessary. ### Azure setup You need a few Azure resources existing: * A storage account, with a container with public access enabled, so we can upload some files * An Azure AD application, for which you know the `client_id` * We'll be using the Azure CLI `az`, for which you ideally are already signed-in, so I don't need to have credentials (like storage account keys etc.) visible in the commands below. With all that, let's rock a bit... ## Use `OpenSSL` to generate a plain RSA key First, we have to create an RSA key, consisting of the key pair in the `private_key_file`. This is your simulated identity provider's most sensitive token-issuance credential. ```shell #!/bin/bash # Generate an RSA key private_key_file="key.pem" public_key_file="key.pub" openssl genrsa -out "${private_key_file}" 2048 openssl rsa -in "${private_key_file}" -pubout > "${public_key_file}" ``` > From now on, I'll be omitting the `#!/bin/bash` part from shell scripts... As these commands build on top of each other, you need to execute things in the same shell session, otherwise your variable values might be lost. ## Pretend there's an 'Identity Provider' in Azure Blob Storage ### The OIDC well-known configuration What is an 'Identity Provider'? Usually, an IdP is a full-blown service, with user management, web site, and many moving parts. For our demo, we just need enough to convince Azure Active Directory that we're having an OpenID-Connect (OIDC) compliant IdP. For this discovery to happen, Azure AD expects to be able to download metadata about our IdP, from some well-known location, and this is the `.well-known/openid-configuration` file. We'll be hosting this file in Azure Blob Storage. > You might check the [openid-connect-discovery-1\_0 spec](https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderConfig) for further details... We need to give our IdP a name (`'https://chgeuer.blob.core.windows.net/public'`), which will be used by AAD as a base URL to find the IdP's configuration in `'https://chgeuer.blob.core.windows.net/public/.well-known/openid-configuration'`: ```json { "issuer": "https://chgeuer.blob.core.windows.net/public", "jwks_uri": "https://chgeuer.blob.core.windows.net/public/jwks_uri/keys", "token_endpoint": "https://chgeuer.blob.core.windows.net/public", "id_token_signing_alg_values_supported": ["RS256"], "token_endpoint_auth_methods_supported": ["client_secret_post"], "response_modes_supported": ["form_post"], "response_types_supported": ["id_token"], "scopes_supported": ["openid"], "claims_supported": ["sub","iss","aud","exp","iat","name"] } ``` In a real production IdP, this information is much more detailed. For example, check the AAD configuration for my own AAD tenant [here](https://login.microsoftonline.com/chgeuerfte.onmicrosoft.com/v2.0/.well-known/openid-configuration). The two very relevant parts of this are the `issuer` name, and the `jwks_uri` location, where Azure AD can find the cryptograhic key material to validate the authenticity of our self-issued tokens. So let's create and upload that file. > Again, as mentioned previously, please change variables like `storage_account` and `container_name` to reflect your own environment. This long pipeline within the `openid_config_json="$( echo '{..}' | jq ... | jq ... )"` part of the script below is an incremental build-up of the JSON structure, in which I'm passing JSON-formatted strings along the pipeline, and each `jq --arg x "..." '.foo=$x'` step essentially tweaks one property at a time. > Check this [little recipe here](https://cookbook.geuer-pollmann.de/command-line-utilities/jq), if you like that. After writing the JSON to a local file, we upload it to `".well-known/openid-configuration"` location. Unfortunately, the `az` command has currently no piping contents to be uploaded via STDIN, so we use a local temporary file. ```shell storage_account="chgeuer" container_name="public" issuer_path="https://${storage_account}.blob.core.windows.net/${container_name}" jwks_keys="${issuer_path}/jwks_uri/keys" openid_config_json="$( \ echo '{"issuer":"","token_endpoint":"","jwks_uri":"","id_token_signing_alg_values_supported":["RS256"],"token_endpoint_auth_methods_supported":["client_secret_post"],"response_modes_supported":["form_post"],"response_types_supported":["id_token"],"scopes_supported":["openid"],"claims_supported":["sub","iss","aud","exp","iat","name"]}' | \ jq --arg x "${issuer_path}" '.issuer=$x' | \ jq --arg x "${issuer_path}" '.token_endpoint=$x' | \ jq --arg x "${jwks_keys}" '.jwks_uri=$x' | \ jq -c -M "." | \ iconv --from-code=ascii --to-code=utf-8 )" echo "${openid_config_json}" > openid-configuration.json az storage blob upload \ --account-name "${storage_account}" \ --container-name "${container_name}" \ --content-type "application/json" \ --file openid-configuration.json \ --name ".well-known/openid-configuration" ``` ### IdP key material For Azure AD to validate our fake tokens, it needs to know our IdP's cryptographic identity, contained in the URL in the `jwks_uri` property of the metadata. This is a JSON file, containing a list of all cryptographic keys valid at the current point in time. For example, if you [look at Azure AD itself](https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/discovery/v2.0/keys), you can see that the `keys` array contains multiple X.509 certificates (and their literal public keys, extracted for convenience). > In our demo, we won't use a real production X.509 certificate, primarily because I don't know where to quickly get a cert which is allowed for the purpose of signing data (issuing tokens). Getting some cert from LetsEncrypt is cool for peer-entity authentication (TLS authN against your web site), but token issuance is a different story. So let's see if AAD lets us get away with just putting a self-cooked RSA key into the JWKS structure. An RSA public key has two components, the exponent and the modulus. OpenSSL seems to usually select the integer value 2^16+1 (65537) as exponent, so this is hard-coded below. For the modulus, we extract it out of the key file, trim away some stuff, convert it to Base64. You also need to define a key ID (`key_id` in the script). As you could see, a production IdP might have many valid keys, and in the JWT, the IdP can indicate which key it used for issuing a token. The JWKT `kid` property on the key is that key ID. We're just using `key1` here as value. After generating the JSON structure, we need to upload it to the location previously configured in your `".well-known/openid-configuration"` file. ```shell # # Get the modulus out ('n' in jwks lingo) # modulus="$( openssl rsa -in "${private_key_file}" -modulus -pubout -noout | \ sed 's/Modulus=//' | \ xxd -r -p | \ base64 --wrap=0 )" ## If the public exponent is 65537, it is "AQAB" base64-encoded # You can run the command below to see the exponent: # # openssl rsa -in "${private_key_file}" -text -noout | grep publicExponent | sed 's/publicExponent: //' # exponent="AQAB" key_id="key1" jwks_keys_json="$( echo "{}" | \ jq --arg x "RSA" '.keys[0].kty=$x' | \ jq --arg x "${issuer_path}" '.keys[0].issuer=$x' | \ jq --arg x "${key_id}" '.keys[0].kid=$x' | \ jq --arg x "${exponent}" '.keys[0].e=$x' | \ jq --arg x "${modulus}" '.keys[0].n=$x' | \ iconv --from-code=ascii --to-code=utf-8 )" echo "${jwks_keys_json}" | jq . > keys.json az storage blob upload \ --account-name "${storage_account}" \ --container-name "${container_name}" \ --content-type "application/json" \ --file keys.json \ --name "jwks_uri/keys" # az storage blob upload --account-name chgeuer --container-name public --content-type "application/json" --file openid-configuration.json --name ".well-known/openid-configuration" # az storage blob upload --account-name chgeuer --container-name public --content-type "application/json" --file keys.json --name "jwks_uri/keys" ``` After this step, we have a key pair on our computer to issue tokens, and we uploaded the metadata to blob storage, to convince AAD that we're a live-operating IdP STS. This is how the `jwks_uri` JSON looks like (key value trimmed for brevity): ```json { "keys": [ { "kty": "RSA", "issuer": "https://chgeuer.blob.core.windows.net/public", "kid": "key1", "e": "AQAB", "n": "ytazPuyVvkHY/ZwmFl+hdVuU//someVeryLongValueWithManyDigitsBecause2048bitWantToLiveSomewhere//...59LutVLQ==" } ] } ``` ## Configure an existing application in Azure AD to allow sign-in via 'Workload Identity Federation' We're now ready to add a `federatedIdentityCredential` to our existing application's credential section, telling Azure AD that this application signs in with a bearer token, issued by our external IdP. This newly introduced `federatedIdentityCredentials` section in Azure AD consists of three important parts: * The `issuer` is the name (or base URL) of the IdP. In your case, that's our storage container, relative to where the `".well-known/openid-configuration"` path will be resolved. * The `audiences` array lists acceptable URIs under which AAD might be known to the external IdP. Simply speaking, AAD needs to defend against an attacker taking a token which was issued to be used at a completely different service, and then just presented to AAD to obtain a token. The Azure AD docs suggest to use an `audience` value of `"api://AzureADTokenExchange"`, so we're just using this here. * The `subject' field is the name under which our application would be known to the external IdP, and which the external IdP will put as` sub\` claim into the JWT. First, we query AAD (using `az ad app show`) to retrieve the object ID for our application, then we POST the plain REST request with the new `federatedIdentityCredential` (using the `az rest` call against the Graph API). ```shell # federation appId="2f40da7e-e023-4ae0-928e-cb906cd8ec49" objId="$( az ad app show --id "${appId}" | jq -r ".objectId" )" audience="api://AzureADTokenExchange" subject="subject" # https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation-create-trust # # Add the credential to the application # az rest \ --method POST \ --uri "https://graph.microsoft.com/beta/applications/${objId}/federatedIdentityCredentials" \ --body "{\"name\":\"Testing\",\"issuer\":\"${issuer_path}\",\"subject\":\"${subject}\",\"description\":\"Testing\",\"audiences\":[\"${audience}\"]}" # # Just read it again to see what was uploaded # az rest \ --method GET \ --uri "https://graph.microsoft.com/beta/applications/${objId}/federatedIdentityCredentials" \ | jq '.' ``` ## Pretend to be an IdP, and generate a minimal and valid self-issued JWT After all that Azure setup, we can have some local fun, massaging bits and bytes into cryptographically acceptable tokens. All the `sed` stuff in the `create_base64_url` bash function essentially replaces plus symbols with dashes, slashes with underscores, and trims away trailing equal signs, to minimize base64-encoded information, and make it possible to use it in URLs. Read the frickin [spec](https://tools.ietf.org/html/rfc7515#appendix-C) if you really care... We now want to create a JWT, pointing to our `key1` in the header, use RSA with SHA256 and PKCS1.5 padding, to sign the payload, which claims that we are indeed the expected subject, and that the token is intended to be used by AAD (`audience = "api://AzureADTokenExchange"`). Given that we're now an IdP, we can choose how long our token is valid (when it expires), using the `exp` property in the JWT. We're using 10 minutes here. ```shell # https://tools.ietf.org/html/rfc7515#appendix-C # base64 --wrap=0 # openssl base64 # # tr -d '\n' | tr '/+' '_-' | tr -d '=' # sed 's/\+/-/g' | sed 's/\//_/g' sed 's/=//g' # sed -E s%=+$%% | sed s%\+%-%g | sed -E s%/%_%g function create_base64_url { local base64text="$1" echo -n "${base64text}" | sed -E s%=+$%% | sed s%\+%-%g | sed -E s%/%_%g } function json_to_base64 { local jsonText="$1" create_base64_url "$( echo -n "${jsonText}" | base64 --wrap=0 )" } # `jq -c -M` gives a condensed/Monochome(no ANSI codes) representation header="$( echo "{}" | \ jq --arg x "JWT" '.typ=$x' | \ jq --arg x "RS256" '.alg=$x' | \ jq --arg x "${key_id}" '.kid=$x' | \ jq -c -M "." | \ iconv --from-code=ascii --to-code=utf-8 )" token_validity_duration="+10 minute" payload="$( echo "{}" | \ jq --arg x "${issuer_path}" '.iss=$x' | \ jq --arg x "${audience}" '.aud=$x' | \ jq --arg x "${subject}" '.sub=$x' | \ jq --arg x "$( date +%s )" '.iat=($x | fromjson)' | \ jq --arg x "$( date --date="${token_validity_duration}" +%s )" '.exp=($x | fromjson)' | \ jq -c -M "." | \ iconv --from-code=ascii --to-code=utf-8 )" toBeSigned="$( echo -n "$( json_to_base64 "${header}" ).$( json_to_base64 "${payload}" )" | iconv --to-code=ascii )" # RSASSA-PKCS1-v1_5 using SHA-256 signature="$( echo -n "${toBeSigned}" | \ openssl dgst -sha256 --binary -sign "${private_key_file}" | \ base64 --wrap=0 | \ sed s%\+%-%g | \ sed -E s%/%_%g | \ sed -E s%=+$%% )" self_issued_jwt="${toBeSigned}.${signature}" cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${self_issued_jwt}" )" ``` The last command contains two funny tricks: * On WSL (Windows Subsystem for Linux), `cmd.exe /C "start $( echo "https://foo" )"` launches the user's default web browser with the given URL. * The URL `https://jwt.ms/#access_token=...` points to Microsoft's pretty awesome [`jwt.ms`](https://jwt.ms/) page, which is a single-page application to display JWT tokens **locally** in your browser. The `#access_token` fragment identifier essentially handles the JWT payload to the client-side JavaScript, i.e. the token is *not* sent over the Internet to the server, which is kind-of cool. [Click here](https://jwt.ms/#access_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImtleTEifQ.eyJpc3MiOiJodHRwczovL2NoZ2V1ZXIuYmxvYi5jb3JlLndpbmRvd3MubmV0L3B1YmxpYyIsImF1ZCI6ImFwaTovL0F6dXJlQURUb2tlbkV4Y2hhbmdlIiwic3ViIjoic3ViamVjdCIsImlhdCI6MTY0NDQ0NjQzNSwiZXhwIjoxNjQ0NDQ3MDM1fQ.GG1n7B81D8HZIvCS2e-bWzomMRyqMb-zs-8oF2Oh_dgIQy2TT15DoJaBZrybsMvRRjfJbsmQz_Pay1dFSwtfDMY4WF2MwxPk95VKz4f9MO4SA_NVY8lMHB49gosBqUpQrPD8DzKEIm-cvIpJJTQzNpcefoNf_Ax2AlbDy5h3zazlvqpFUjim-3nwCTN2ERd0FB3iTsTMMk6VG0bu76qkK-9t3Zh8tdkENCcLVQslHBkcFhN3F9hRLLYiEtOTyLTFaWSr0PDbsiSsMmK8XGxMHNF2K03LGk-whb0mPjKQ17mvclMLgJ9M187KC8HljAIj3Powtw9YsVp6kC2yMOBGuQ) to see how this looks in practice... You can essentially see the JWT, and the decoded payload: ``` { "typ": "JWT", "alg": "RS256", "kid": "key1" }.{ "iss": "https://chgeuer.blob.core.windows.net/public", "aud": "api://AzureADTokenExchange", "sub": "subject", "iat": 1644446435, "exp": 1644447035 }.[Signature] ``` ## Do a client-credential grant dance with AAD, exchanging our self-issued JWT against a real production token Our last and final step in the journey is to try, whether our self-issued JWT will finally be accepted by Azure AD, and can be used to retrieve a 'real' AAD token, which we can then use to talk to other applications. So we're asking AAD to exchange our self-issued JWT access token with an AAD-issued access token. We'll be running a client-credentials grant call, a capability supported by AAD since a long time already. The new aspect, workload identity federation-related difference is that we now can indicate that we have a `client_assertion` (our self-issued token), and that the `client_assertion_type` is a `jwt-bearer` token. In the example below, I ask 'my' own AAD tenant (`chgeuerfte.onmicrosoft.com`) to issue me a token, so I can talk to the Azure ARM API: ```shell resource="https://management.azure.com/.default" aadTenant="chgeuerfte.onmicrosoft.com" token_response="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${aadTenant}/oauth2/v2.0/token" \ --data-urlencode "response_type=token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \ --data-urlencode "client_id=${appId}" \ --data-urlencode "client_assertion=${self_issued_jwt}" \ --data-urlencode "scope=${resource}" \ )" echo "${token_response}" | jq . access_token="$( echo "${token_response}" | jq -r ".access_token" )" cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )" ``` The token which AAD now gives back contains much more information, containing the resource we want to talk to (`aud`), the issuer being our AAD tenant, and the `"appid"` being our original application's client\_id for which we enabled the workload identity federation. ```json { "typ": "JWT", "alg": "RS256", "x5t": "Mr5-AUibfBii7Nd1jBebaxboXW0", "kid": "Mr5-AUibfBii7Nd1jBebaxboXW0" }.{ "aud": "https://management.azure.com", "iss": "https://sts.windows.net/942023a6-efbe-4d97-a72d-532ef7337595/", "idp": "https://sts.windows.net/942023a6-efbe-4d97-a72d-532ef7337595/", "tid": "942023a6-efbe-4d97-a72d-532ef7337595", "iat": 1644446379, "nbf": 1644446379, "exp": 1644450279, "appidacr": "2", "idtyp": "app", "aio": "...", "appid": "2f40da7e-e023-4ae0-928e-cb906cd8ec49", "oid": "e03a9fa1-525e-4a7a-884f-d598fc0fae86", "sub": "e03a9fa1-525e-4a7a-884f-d598fc0fae86", "rh": "...", "uti": "...", "ver": "1.0" }.[Signature] ``` So you might compare and contrast these two tokens: ```shell cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${self_issued_jwt}" )" cmd.exe /C "start $( echo "https://jwt.ms/#access_token=${access_token}" )" ``` ## Invoke some real service Last, we can of course use our real access\_token to invoke the service, like listing resource groups in an Azure subscription (assuming our app has the right to do so), or whatnot... ```shell subscriptionId="724467b5-...." curl --silent \ --get \ --url "https://management.azure.com/subscriptions/${subscriptionId}/resourcegroups" \ --data-urlencode "api-version=2018-05-01" \ --header "Authorization: Bearer ${access_token}" \ | jq -r '.value[].name' ``` That's it. That's "running" an IdP in Blob Storage, hand-crafting and RSA-signing JSON web tokens in `bash` (don't do this in production, kids), and working with AAD, to play with the Workload Identity Federation preview feature... If you like what you saw, feel free to ping me on Twitter ([@chgeuer](https://twitter.com/chgeuer/)), or use the same alias `@microsoft.com` to shoot me an e-mail. # Federated credentials from GitHub and GitLab pipelines to Azure ![Teaser](/files/ViWKsGRameID4pBfDR83) ## Azure access from GitHub and GitLab pipelines - without secrets > This post also appeared in the [FastTrack Blog](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/azure-access-from-github-and-gitlab-pipelines-without-secrets/ba-p/3858885). ### tl;dr * Federated credentials / workload identity federation allows your CI/CD pipelines in GitHub and GitLab to access your Azure subscription without any secrets stored in the pipeline config. * GitHub's [`azure/login@1`](https://github.com/Azure/login) task handles this transparently, but I also explain how it works under the hood. GitLab supplies the necessary token directly to your pipeline run. * Both GitHub and GitLab are easy to setup and federate securely with your Azure subscription. * BitBucket can't be setup that way, because tokens issued by BitBucket don't have a predicable subject identifier. ### Overview This article demonstrates how to configure, and securely access, an Azure environment from within a GitHub and a GitLab CI/CD pipeline, without having to store credentials on the GitHub/GitLab side. The article also briefly explains why BitBucket currently doesn't support that capability. CI/CD pipelines often have to interact with an Azure cloud environment, e.g. to upload artifacts to a storage account, read values from a Key Vault or deploy resources. Service principal credentials are a well-established way for such access, but have the disadvantage of using secrets and passwords, which have to be managed securely. Using 'federated identity credentials' (also sometimes called 'workload identity federation'), such cloud access can happen without having to store secrets in the CI/CD pipeline. **Old fashioned service principals**: Traditionally, a service principal credential consists of three values: The `client_id` of the service principal or app registration, the Azure AD `tenant_id` where the service principal exists, and a `client_secret` password needed to fetch a token. Ideally, we want to avoid having to handle a `client_secret`; these secrets often have a lifetime (they expire and have to be updated after a couple of months), and access to these secrets must be protected so only authorized parties can access these secrets. **Federated credentials**: Modern CI/CD systems, such as GitHub or GitLab, allow their users to run pipelines on container-based runners in their infrastructure. As part of these environments, they also expose a service-provider-specific OAuth2 identity provider (IdP), that the code in the CI/CD can fetch tokens from. The idea behind a federated credential is to say: "In Azure, there isa user-assigned managed identity (or a service principal), and the CI/CD pipeline should be able to use a token from the local IdP, to sign-in to that UAMI/SP". So there are two token exchanges: * The CI/CD pipeline somehow talks to the 'local' IdP, says "I am the main branch within this project, please issue me a JWT token which I can then use to sign-in to Azure". That security token has an issuer being GitHub or GitLab, an audience of Azure AD, and the token's subject being information about which CI/CD pipeline is currently running. * The CI/CD pipeline then talks to Azure Active Directory, and exchanges the GitHub-issued token with one that can be used to access the desired Azure resource. The exchange basically says "Here's a security token, showing that I'm this CI/CD pipeline, please give me a token to call into Azure KeyVault (or ARM, or Storage, or whatever it might be)". The [Microsoft Entra 'Workload Identity Federation' docs](https://learn.microsoft.com/en-us/azure/active-directory/workload-identities/workload-identity-federation) show in depth how the flow works in general: ![Workload Identity Federation](/files/Dg1LPH434risIe7cDgjQ) In our scenario, the 'external identity provider' is the GitHub/GitLab-internal IdP. Simply speaking, 1. the CI/CD pipeline fetches the token from GitHub, 2. fetches the Azure token from Azure AD, 3. during that request, Azure AD validates the GitHub-issued token by retrieving the external IdP's signing credentials and checking the token signature, and 4. finally the CI/CD code can access the Azure resources: ![Interactions](/files/vYtnHtkJCI8BRAW63l10) ### Service Principals and user-assigned managed identities Federated credentials are supported by both service principals and user-assigned managed identities (UAMI). In the end, a UAMI under the hood is represented by a service principal in Azure AD, too. The identities of both a service principal and UAMI can be granted access to Azure resources, so both are a fit here, too. However, the lifecycle management and API surface for these two identities is very different: A **service principal** is created and configured within Azure Active Directory (for example by calling `az ad app create`), and adding a `https://graph.microsoft.com/beta/applications/${applicationObjectId}/federatedIdentityCredentials/` via Microsoft Graph API. Depending on where you work, writing to Azure AD and Graph API might be tightly regulated; many companies prevent regular users from directly creating a service principal, so this route might be challenging. A **user-assigned managed identity** on the other hand can be completely handled in the Azure Resource Manager (ARM) control plane. A UAMI (and it's federated identity configuration) is a first-party ARM object, so that might be more approachable for teams who have full control over their Azure subscription (but lack Azure AD privileges). #### The federated credential configuration for Azure Both the service principal configuration (via Microsoft Graph API), as well as the UAMI configuration (via ARM API) require the same configuration data: * `name`: Each SP or UAMI might have up to 20 different federated credentials configured, and each credential must have a `name` attribute. * `issuer`: Each federated credential must have the issuer URL configured. * The `issuer` is something like `"https://token.actions.githubusercontent.com"` or `"https://gitlab.com"`, or a custom domain name in case of a dedicated GitLab instance. It must be equivalent to the `iss` claim in the security token. * How is it used? Azure AD appends the path `.well-known/openid-configuration` to the issuer URL, to retrieve the IdP's signing credentials, used to check the signature on the tokens. * `audience`: An array (with exactly one string) of the `aud` claim in the federated identity token. * By default, this is `"api://AzureADTokenExchange"`, but you can customize that if desired. * `subject`: The subject value is the `sub` claim of the security token, and is determined by the CI/CD environment. * For **GitHub**, this subject for example looks like `"repo:chgeuer/azure-workload-identity-github:ref:refs/heads/main"`, in which `chgeuer/azure-workload-identity-github` represents the user or organization (`chgeuer`), and the repository (`azure-workload-identity-github`), while `ref:refs/heads/main` indicates a CI/CD pipeline running on the `main` branch. * For **GitLab**, this subject looks similar, like `"project_path:chgeuer/azure-workload-identity-federation-demo:ref_type:branch:ref:main"`, i.e. `chgeuer` being the user, `azure-workload-identity-federation-demo` being the repository and `main` being the branch. * Unfortunately, **BitBucket** handles this differently: For federated credential sign-in to work well, the expected `sub` claim in the security token must have a predictable value. BitBucket's `sub` claims look like this:`"{ad073b2b-7126-4f19-9eed-1c9b10abe160}:{2b2ac083-d564-4064-8ea1-43e6aeff2b96}:{37416cfa-3260-4c31-bea4-a6b2f29272a7}"`. The three GUIDs are `"{repositoryUuid}:{deploymentEnvironmentUuid}:{stepUuid}"`. The `repositoryUuid` and the `deploymentEnvironmentUuid` are stable, but unfortunately, the 3rd element in the tuple, the `stepUuid`, is re-generated with each pipeline run. Therefore, each time a new security token has a different `sub` claim. Given that an Azure federated identity credential expects a fixed subject, and does not allow semantics like 'Subject claim starts with ... or conforms to a regular expression', BitBucket's tokens can't be used for federated credential flows. These four values, represented in JSON, in a GitHub configuration would look like this: ```json { "name": "githubfedcred", "issuer": "https://token.actions.githubusercontent.com", "audience": [ "api://AzureADTokenExchange" ], "subject": "repo:chgeuer/azure-workload-identity-github:ref:refs/heads/main" } ``` while a GitLab config would look like this: ```json { "name": "gitlabcred", "issuer": "https://gitlab.com", "audience": [ "api://AzureADTokenExchange" ], "subject": "project_path:chgeuer/azure-workload-identity-federation-demo:ref_type:branch:ref:main" } ``` **Example on creating a service principal with a federated credential (using a script)** The following bash script gives you an idea on how the service principal would be created (`az ad app create`), and how you can add the `federatedIdentityCredentials` JSON to Graph API ```shell #!/bin/bash az ad app create --display-name "${appDisplayName}" applicationObjectId="$( az ad app list --display-name "${appDisplayName}" | jq -r '.[0].id' )" az rest \ --method POST \ --uri "https://graph.microsoft.com/beta/applications/${applicationObjectId}/federatedIdentityCredentials/" \ --body '{"name": "github","issuer": "https://token.actions.githubusercontent.com", "audience": [ "api://AzureADTokenExchange" ], "subject": "repo:chgeuer/azure-workload-identity-github:ref:refs/heads/main"}' ``` **Example creating a user-assigned managed identity with a federated credential (using infra-as-code)** When creating a UAMI via Bicep, these two steps could be combined in a single representation: ```bicep resource uami 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = { name: uamiName location: location resource federatedCred 'federatedIdentityCredentials' = { name: 'githubcred' properties: { issuer: 'https://token.actions.githubusercontent.com' audiences: [ 'api://AzureADTokenExchange' ] subject: 'repo:chgeuer/azure-workload-identity-github:ref:refs/heads/main' } } } ``` #### Information from Azure, needed for the CI/CD environment Once the service principal or the UAMI is created on the Azure side, you need two pieces of information from Azure: * The **Azure Active Directory's Tenant ID**, i.e. either the tenant's GUID (something like `942023a6-efbe-4d97-a72d-532ef7337595`), or one of the configured domain names (such as `chgeuerfte.onmicrosoft.com`). * The service principal's or UAMI's **client ID**, which always is a GUID. These two values must be configured in the CI/CD environment, as environment variables, or secrets (even though they're strictly speaking not *secret*). ### GitHub-side setup with full integration The following sample shows how to ZIP the repo's source code and upload it into a storage account: ```yaml name: ZIP the source and upload on: workflow_dispatch: permissions: id-token: write contents: read jobs: build: name: Zip and upload runs-on: ubuntu-latest env: account_name: 'isvreleases' container_name: 'backendrelease' filename: 'src.zip' steps: - uses: actions/checkout@v3 - name: 'Create ZIP' run: | zip -r "${filename}" src/ - name: 'Login via azure/login@v1' uses: azure/login@v1 with: tenant-id: ${{ secrets.AZURE_TENANT_ID }} client-id: ${{ secrets.AZURE_CLIENT_ID }} allow-no-subscriptions: true environment: azurecloud # audience: api://AzureADTokenExchange - name: 'Use the token to upload to storage' uses: azure/CLI@v1 with: azcliversion: 2.37.0 inlineScript: | blob_name="src-azure-workload-identity-github-${GITHUB_SHA}.zip" blob_url="https://${account_name}.blob.core.windows.net/${container_name}/${blob_name}" az storage blob upload --auth-mode login --account-name "${account_name}" --container-name "${container_name}" --overwrite --file "${filename}" --name "${blob_name}" echo "Uploaded to ${blob_url}" ``` The use of Azure Storage is just a sample of an Azure resource, that we can access from within GitHub. You can see a few interesting parts in that `YAML` file: The YAML file must contain the following lines, so that the GitHub IdP is able to issue a token: ```yaml permissions: id-token: write contents: read ``` The [`azure/login@1`](https://github.com/Azure/login) task on GitHub automagically handles all the federated sign-in to an SP or a UAMI. In the sample below, we set `tenant-id` and `client-id` based on GitHub secret values. If the expected `audience` on Azure would be different from `"api://AzureADTokenExchange"`, it would also be possible to tweak that value: ```yaml - name: 'Login via azure/login@v1' uses: azure/login@v1 with: tenant-id: ${{ secrets.AZURE_TENANT_ID }} client-id: ${{ secrets.AZURE_CLIENT_ID }} allow-no-subscriptions: true environment: azurecloud # audience: api://AzureADTokenExchange ``` A full example of that configuration can be found here: ### GitHub-side setup - The hard way For those interested in understanding what the [`azure/login@1`](https://github.com/Azure/login) task on GitHub does under the hood, we can mimic it all in a `bash` script with `curl` and `jq`. This allows us to inspect the security tokens, along the way. Let's tweak the step in the CI/CD to just run a shell script: ```yaml name: ZIP the source and upload on: workflow_dispatch: permissions: id-token: write contents: read jobs: build: name: Build the stuff runs-on: ubuntu-latest env: account_name: 'isvreleases' container_name: 'backendrelease' filename: 'src.zip' steps: ... - name: 'Interact with the Github IDP and Azure Workload Identity Federation from shell' env: AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }} run: | ./action.sh ``` In this code, we copy the tenant ID and client ID from the secrets into environment variables. In [`action.sh`](https://github.com/chgeuer/azure-workload-identity-github/blob/main/action.sh), we now manually do the two token exchanges, and print out the claims from the JWT: ```shell #!/bin/bash encodedAudience="api%3A%2F%2FAzureADTokenExchange" gh_access_token="$( curl \ --silent \ --url "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=${encodedAudience}" \ --header "Authorization: Bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ | jq -r ".value" )" azure_access_token="$( curl \ --silent \ --request POST \ --data-urlencode "response_type=token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \ --data-urlencode "client_id=${AZURE_CLIENT_ID}" \ --data-urlencode "client_assertion=${gh_access_token}" \ --data-urlencode "scope=https://storage.azure.com/.default" \ "https://login.microsoftonline.com/${AZURE_TENANT_ID}/oauth2/v2.0/token" \ | jq -r ".access_token" )" gh_claims="$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${gh_access_token}" )" aad_claims="$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${azure_access_token}" )" echo "# Tokens | Token Issuer | Claim | Value | | ------------ | -------- | ------------------------------------------- | | GitHub | Issuer | \`iss=$( echo "${gh_claims}" | jq .iss )\` | | GitHub | Audience | \`aud=$( echo "${gh_claims}" | jq .aud )\` | | GitHub | Subject | \`sub=$( echo "${gh_claims}" | jq .sub )\` | | Azure | Issuer | \`iss=$( echo "${aad_claims}" | jq .iss )\` | | Azure | Audience | \`aud=$( echo "${aad_claims}" | jq .aud )\` | | Azure | Subject | \`sub=$( echo "${aad_claims}" | jq .sub )\` | " >> "${GITHUB_STEP_SUMMARY}" ``` First, we fetch a token from the GitHub's IdP: the environment variable `ACTIONS_ID_TOKEN_REQUEST_URL` contains the URL of the IdP. The URL-encoded audience goes into the query string, and we supply a GitHub-internal security token from the environment variable `ACTIONS_ID_TOKEN_REQUEST_TOKEN` as bearer token. Second, we issue a token issuance request against our Azure AD tenant, in which we specify the UAMI's or SP's `client_id`, and supply the GitHub-issued JWT as `client_assertion`. Then we use `jq -R 'split(".") | .[1] | @base64d | fromjson'` to extract the claims portion from both JWT tokens, and write out a Markdown-formatted table with `iss`, `aud` and `sub` claims of both tokens to the file `"${GITHUB_STEP_SUMMARY}"`, so that the table shows up in the CI/CD's pipeline output: | Token Issuer | Claim | Value | | ------------ | -------- | ----------------------------------------------------------------------- | | GitHub | Issuer | `iss="https://token.actions.githubusercontent.com"` | | GitHub | Audience | `aud="api://AzureADTokenExchange"` | | GitHub | Subject | `sub="repo:chgeuer/azure-workload-identity-github:ref:refs/heads/main"` | | Azure | Issuer | `iss="https://sts.windows.net/***/"` | | Azure | Audience | `aud="https://storage.azure.com"` | | Azure | Subject | `sub="079fd90b-a298-480a-b951-257d0974f77e"` | In the above output table, GitHub blanks-out (`***`) the Azure AD tenant, in the Azure/Issuer value, because that string comes from the `AZURE_TENANT_ID` secret. In the rest of shell script, the `azure_access_token` shell variable can be used to call Azure services, in this sample Azure Storage. ### GitLab-side setup On the GitLab side, we don't have task like GitHub's [`azure/login@1`](https://github.com/Azure/login) action, so we follow a pure script-based approach in our GitLab YAML: ```yaml image: mcr.microsoft.com/azure-cli:latest build1: stage: build id_tokens: ID_TOKEN_FOR_AZURE: aud: "api://AzureADTokenExchange" script: - echo "##### Logging-in to the Azure user-assigned managed identity" - az login --service-principal --tenant "${AZURE_TENANT_ID}" --username "${AZURE_UAMI_CLIENT_ID}" --federated-token "${ID_TOKEN_FOR_AZURE}" --allow-no-subscriptions - echo "##### Demo: fetching a secret from Key Vault" - az keyvault secret show --vault-name "${AZURE_KEYVAULT_NAME}" --name "${AZURE_KEYVAULT_SECRET_NAME}" | jq . - echo "##### Full token contents" - jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${ID_TOKEN_FOR_AZURE}" - echo "##### Config necessary for Azure" - jq -R 'split(".") | .[1] | @base64d | fromjson | {issuer:.iss,audiences:[.aud],subject:.sub}' <<< "${ID_TOKEN_FOR_AZURE}" ``` A difference you can see above is that token issuance within GitLab is handled differently: You don't need to use `curl` to request a GitLab-issued token from some token endpoint. Instead, you just specify a [`id_tokens`](https://docs.gitlab.com/ee/ci/yaml/index.html#id_tokens) section, in which you name a desired environment variable (`ID_TOKEN_FOR_AZURE` in the above example), and the audience for that token, and GitLab stored the JWT token in your environment variable of choice, prior running your job. We request the whole thing to use the Azure CLI Docker image (`mcr.microsoft.com/azure-cli:latest`), so we can use commands like `az login` in our script. Inside that script, we can then use ```shell az login --service-principal \ --tenant "${AZURE_TENANT_ID}" \ --username "${AZURE_UAMI_CLIENT_ID}" \ --federated-token "${ID_TOKEN_FOR_AZURE}" \ --allow-no-subscriptions ``` To login to a given UAMI or service principal, using a federated token from GitLab. For illustration purposes, we can fetch a secret from a Key Vault (assuming our UAMI is authorized to read that secret), and print out the token contents on screen. Given that a service principal, or a UAMI, can have up to 20 federated credentials configured, one can also hook up multiple pipelines (from different providers) to the same Azure identity: ![20 federated credentials allowed per SP or UAMI](/files/yD5TVF8wf78o1IOJIvqt) ### What about Azure DevOps? Since [March '23](https://devblogs.microsoft.com/devops/introducing-service-principal-and-managed-identity-support-on-azure-devops/), Managed Identity [support](https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/service-principal-managed-identity?view=azure-devops) for Azure DevOps is in public preview. When you're running ADO-based CI/CD pipelines on a compute resource (such as a VM) in your own subscription, you can bind a system-assigned or user-assigned managed identity to that compute resource, and from within your pipeline run access all the Azure resources the managed identity has access to. Federated identity credentials are currently targeted to allow external identity providers (non-Azure AD) to facilitate sign-in to Azure environments. Given that both Azure DevOps and your Azure resources are all governed by Azure Active Directory, there's not necessarily a need to use a federated credential. As of now (June '23), federated identity credentials unfortunately don't work across Azure AD-tenant boundaries (error message AAD STS 700222). Should you want to allow 'inbound' connections from a managed identity in another tenant into resources in your Azure DevOps tenant, check the team's guidance on ["Can I add a managed identity from a different tenant to my organization?"](https://learn.microsoft.com/en-us/azure/devops/integrate/get-started/authentication/service-principal-managed-identity?view=azure-devops#q-can-i-add-a-managed-identity-from-a-different-tenant-to-my-organization) ### Examples In case that article sparked your interest, and you'd like to go deeper with end-to-end samples, check out the following resources... * * CI/CD Provider: Github * Azure Identity: Service Principal, created and configured using a script * YAML: Demo both the 'proper' [`azure/login@1`](https://github.com/Azure/login) task and a bash script. * Scenario Azure Service: Uploading a file to blob storage * Azure Environment, and GitHub variables, all configured as part of [`setup.sh`](https://github.com/chgeuer/azure-workload-identity-github/blob/main/setup.sh) * * CI/CD Provider: GitHub * Azure Identity: A user-assigned managed identity, created using Bicep * YAML: Using all the good pre-defined actions, no fiddling with bash * Scenario Azure Service: Fetch a secret from Key Vault, and base64 encode it to GitHub's secret filter doesn't kick in 😬 * * CI/CD Provider: GitLab * Azure Identity: A user-assigned managed identity, created using Bicep * YAML: Using `az` CLI commands for Azure access. * Scenario Azure Service: Fetch a secret from Key Vault, fetching the secret using the `az cli`. * [Minimal "Azure AD Workload identity federation"](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/azure/workload-identity-federation/README.md) * In case you're interested in fully understanding how the federated credential / workload identity federation works with Azure, this blog post shows how to create an IdP signing credential, upload it into a publicly accessible location (blob storage), and show how to create a self-issued token, which can be used to sign-in to a federated credential on Azure. # Azure Marketplace Metered Billing- Picking the correct ID when submitting usage events Copy of > tl;dr: Don't know whether to use `resourceId` and `resourceUri` when submitting usage events to `[https://marketplaceapi.microsoft.com/api/usageEvent](https://marketplaceapi.microsoft.com/api/usageEvent)` or `.../batchUsageEvent`? Read on... In Azure Marketplace, both *SaaS* offers, and *Azure Managed Applications* allow the publisher to submit custom usage events to the [Metering service API](https://docs.microsoft.com/en-us/azure/marketplace/marketplace-metering-service-apis) (if you have enabled custom metering dimensions on your marketplace offer). The metering service API accepts either a single usage event (on the `usageEvent` endpoint), or a batch (on the `batchUsageEvent` endpoint). A single usage event JSON message looks like this: ``` { "effectiveStartTime": "2018-12-01T08:00:00", "planId": "id-of-the-plan-as-configured-in-partner-center" "dimension": "id-of-the-billing-dimension-as-configured-in-partner-center", "quantity": 5.0, "resourceId": "deadbeef-123...guid" ... or ... "resourceUri": "/subscription/..." } ``` In this message, the first 4 lines are kind-of-easy: `effectiveStartTime` is the hour (in UTC) for which the usage has been aggregated. `planId` and `dimension` correspond are the identifiers configured in Partner Center, `quantity` is the JSON serialization of the quantity of usage that happened for the given dimension in the given hour, either something like `5` or `5.0` (or `0.02` or `23892`). But what about that `resource...` stuff, do I have to use a `"resourceId"`, or `"resourceUri"`, or even both?!? And where does that value come from? Well, that depends on the offer type you have.. ## Software-as-a-Service (SaaS) offers - use `resourceId` For a [SaaS](https://docs.microsoft.com/en-us/azure/marketplace/partner-center-portal/saas-metered-billing) offer, it is comparably easy: When a customer purchases a SaaS offer, they get sent to your '[landing page](https://docs.microsoft.com/en-us/azure/marketplace/azure-ad-transactable-saas-landing-page)', from which you call the '[SaaS fulfillment Subscription APIs v2](https://docs.microsoft.com/en-us/azure/marketplace/partner-center-portal/pc-saas-fulfillment-subscription-api)', specifically the [`api/saas/subscriptions/resolve`](https://docs.microsoft.com/en-us/azure/marketplace/partner-center-portal/pc-saas-fulfillment-subscription-api#resolve-a-purchased-subscription) endpoint, which returns (amongst other things) the 'purchased SaaS subscription ID' back to your solution: ``` { "id": "", // purchased SaaS subscription ID, which will be used as resourceId in a usage event "subscriptionName": "Contoso Cloud Solution", // SaaS subscription name "offerId": "offer1", // purchased offer ID "planId": "silver", // purchased offer's plan ID ... } ``` You should use the `id` from that response, i.e. the *purchased SaaS subscription ID*, as `resourceId` value in your metering API usage event submissions. This value is a GUID, which your SaaS solution can use to track and aggregate the usage of your customer. The SaaS offer *also* has a `resourceUri`. When purchasing a SaaS offer, the purchaser puts the offer into one of their resource groups (`customer-rg1` below), and gives it a name (`SomeNameTheCustomerChose` below). The ARM resource ID of that would be the `resourceUri`, and looks like this: ``` /subscriptions/.../resourceGroups/customer-rg1/providers/Microsoft.SaaS/resources/SomeNameTheCustomerChose ``` Having said that, you certainly can ignore the `resourceUri` with SaaS offer, as you already have the `resourceId`, which is sufficient. ## Azure Managed Applications - use `resourceUri` An *Azure Application Offer* can either be a '*solution template*' or a '*managed application*'. Only the 'managed app' (Azure Managed Application) can submit usage to the [Metering service API](https://docs.microsoft.com/en-us/azure/marketplace/marketplace-metering-service-apis). A managed app is a bunch of resources described in an ARM template, which are deployed into a '*managed resource group*' in the customer's Azure subscription. The ISV/publisher has administrative control over these resources to 'manage' them, therefore the name 'managed app'; the customer themselves usually cannot change the resource in the managed resource group, even though they live in the customer's subscription. When the customer purchases the managed app via the Marketplace in the Azure Portal, they select a resource group where the 'managed app' object itself lives, and that object is something of type `Microsoft.Solutions/applications`. The resource group which contains the managed app resource object is *owned by the customer*. This managed app resource 'points' to the managed resource group, and vice versa. Consider the following purchase screen (omitting the app-specific details): ![](/files/2nlI4oOClooCM1CoEIR2) You can see the following relevant details: 1. The '**Resource group**' (1) at the top, named **`customer-owned-rg`**, is the resource group in which the managed app object will be deployed. The customer has full control over it. In 2. The '**Application Name**' (2) in the 'Managed Application Details' is the customer-chosen name for the managed app object. Our example here is **`myapp123`**. 3. The '**Managed Resource Group**' (3) at the bottom is the RG where all resources from the ISV's ARM-template will be deployed to. In our example, this is **`managed-rg-for-myapp123`**. * The customer has (usually) no control over this. * The ISV/publisher can manage the resources in it. * The 'Managed Resource Group' field initially has a value of `'MTG-somethingsomething'`, but the customer can choose the managed resource group's name, to better align with own naming conventions. The purchase confirmation summarizes the data again: ![](/files/ylHBs5ds1974JDdVLfPn) After the managed app has been fully deployed, the customer can see the managed app object in their own resource group: ![](/files/ULwRZyN8fjtZ7SD4SLjk) Clicking into this managed app resource gives us the internal details: ![](/files/iyo0VzIGNoQepkjm0e4s) To summarize our progress up until now, in our sample, we have the following names: | Data | Name | | ---------------------- | ------------------------- | | Resource group | `customer-owned-rg` | | Application Name | `myapp123` | | Managed Resource Group | `managed-rg-for-myapp123` | The resource ID of the managed app in the ARM control plane is this: ``` /subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123 ``` The resource ID of the managed resource group is this: ``` /subscriptions/.../resourceGroups/managed-rg-for-myapp123 ``` When you now fetch ARM properties of the managed resource group on the Azure management API, you get the following response: ```http GET /subscriptions/.../resourceGroups/managed-rg-for-myapp123?api-version=2019-07-01 { "id": "/subscriptions/.../resourceGroups/managed-rg-for-myapp123", "type": "Microsoft.Resources/resourceGroups", "name": "managed-rg-for-myapp123", "location": "westeurope", "managedBy": "/subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123", "properties": { "provisioningState": "Succeeded" } } ``` You can see that the description of the managed resource group has a `managedBy` property, which points to the resource ID of the managed app. ### The `resourceUri` for a managed app And now we have which `resourceUri` we could use when submitting usage records to the metering API: The `resourceUri` is `managedBy` property in the the ARM resource representation **of the managed resource group**: ``` { "effectiveStartTime": "2018-12-01T08:00:00", "planId": "plan1" "dimension": "id-of-the-billing-dimension-as-configured-in-partner-center", "quantity": 5.0, "resourceUri": "/subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123" } ``` The good thing with the `resourceUri` is that it's immediately available, even during the actual ARM deployment... ### The `resourceId` for a managed app When we submit the usage to the metering API, we get back a response like this: ```json { "status": "Accepted", "usageEventId": "...", "messageTime": "2018-12-01T10:00:25.7798568Z", "effectiveStartTime": "2018-12-01T08:00:00", "planId": "plan1" "dimension": "id-of-the-billing-dimension-as-configured-in-partner-center", "quantity": 5.0, "resourceUri": "/subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123", "resourceId": "accd441b-01e8-48c7-8c73-90a0ea5cec0a" } ``` It echoes our submitted request, and augments it with further details, such as `messageTime` (when the request arrived), the `"status": "Accepted"`, or a unique `usageEventId` (under which Azure tracks that submission internally). But - there is also a `resourceId`. Where did that came from, and could we have retrieved that ourself? The `resourceId` is actually an ARM property on the managed app. Once Azure has fully (and successfully) provisioned all the resources in the managed resource group, Azure sets a `billingDetails` property on the managed app, which contains a `resourceUsageId` property, which is our `resourceId` for usage events: ```http GET /subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123 { "id": "/subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123", "name": "myapp123", "type": "Microsoft.Solutions/applications", "kind": "MarketPlace", "location": "westeurope" "properties": { "billingDetails": { "resourceUsageId": "accd441b-01e8-48c7-8c73-90a0ea5cec0a" }, "managedResourceGroupId": "/subscriptions/.../resourceGroups/managed-rg-for-myapp123", "provisioningState": "Succeeded", "managementMode": "Managed", "customerSupport": {...}, "supportUrls": {...} }, "plan": { "product": "metered-billing-test-01-preview", "name": "plan1", "publisher": "...", "version": "1.0.0" } } ``` So after the (successful) deployment, **if you are authorized to read the managed app, which is not in the managed resource group, but in the customer's resource group**, then you can get the `.properties.billingDetails.resourceUsageId`. It is important to note that the `billingDetails` are only set by Azure, once the deployment successfully went through. You will not have access to the `billingDetails` during the ARM template provisioning. So in a usage event, you can submit ```json { "effectiveStartTime": "...", "planId": "...", "dimension": "..", "quantity": ..., "resourceId": "accd441b-01e8-48c7-8c73-90a0ea5cec0a" } ``` or ```json { "effectiveStartTime": "...", "planId": "...", "dimension": "..", "quantity": ..., "resourceUri": "/subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123" } ``` or even both ```json { "effectiveStartTime": "...", "planId": "...", "dimension": "..", "quantity": ..., "resourceUri": "/subscriptions/.../resourceGroups/customer-owned-rg/providers/Microsoft.Solutions/applications/myapp123", "resourceId": "accd441b-01e8-48c7-8c73-90a0ea5cec0a" } ``` However, when submitting both, these must obviously belong together... Having said that, even if it is possible to submit both values, it's unnecessary work, so you certainly want to focus on the `resourceUri` . Just do what's simplest for you to retrieve. Keep it simple. ## Summary For a **SaaS offer**, you should be using the SaaS subscription ID (a GUID) as `resourceId`. For a **managed app**, you should use the ARM resource ID of the managed resource group (something like `/subscriptions/.../resourceGroups/...`) as `resourceUri`. Last, but not least: If you are building a solution with metered billing, you might want to have a look at our 'metered billing accelerator' code base here: [microsoft/metered-billing-accelerator (github.com)](https://github.com/microsoft/metered-billing-accelerator). This is an event-sourced solution to do the aggregation and submission for you. # Manually submitting values to the Azure Metering API ## Requirements In this script, we're using `curl` and `jq`. `curl` certainly is on your system, for `jq`, you might need to fetch it: ```shell #!/bin/bash curl --silent \ --url https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \ --location \ --output jq && \ chmod +x jq && \ sudo mv jq /usr/local/bin && \ sudo chown root.root /usr/local/bin/jq ``` ## Credentials I assume you have 2 service principal credentials in the publisher/ISV AAD tenant: 1. We need a *management* credential which is authorized to manage customer deployments, using with the ARM API. * I created an AAD group, called `managed-app-admins`, and made the service principal a member of that group. * In `Partner Center -> Offer -> Plan -> Technical Configuration -> Authorizations`, I granted the AAD group `Owner` of managed apps. * As a result, the `isv_management_cliend_id` below can interact with managed apps. * ![image-20230127212906222](/files/Hvn5M03PSMhE5eSN7pZE) * So `8cdea44d-38fe-4e3c-bf2a-6b96809d1d27` above is the object ID of the group, and the service principal with app id `8eff18b7-aeeb-4a88-b19c-bf60813a587c` is member of that group. 2. We need a *metering* credential, which is authorized to submit usage to the metering API. * We do *not* use an identity in the managed app, i.e. we do *not* SSH/RDP into a VM in the managed resource group to do our work there. Having a VM just for that is too complicated (and expensive). * This `isv_metering_client_id` (app id) service principal is configured under `Partner Center -> Offer -> Technical Configuration`: * ![image-20230127212946875](/files/rIo8rWocTDlRzfYNE65R) ## The script ```shell #!/bin/bash # This is the ISV/publisher app which is also in the partner center # All of these 3 below work (for me) isv_tenant_id="5f9e748d-300b-48f1-85f5-3aa96d6260cb" isv_tenant_id="geuerpollmannde.onmicrosoft.com" isv_tenant_id="geuer-pollmann.de" # This service principal on the ISV AAD tenant has permissions to manage "managed apps" in customer environments isv_management_client_id="8eff18b7-aeeb-4a88-b19c-bf60813a587c" isv_management_client_secret="$( cat "/mnt/c/Users/chgeuer/.secrets/8eff18b7-aeeb-4a88-b19c-bf60813a587c-admin.txt" )" # This service principal is the one that is configured in partner center, under the offer's technical plan isv_metering_client_id="3eb78160-e434-4713-8374-f40174d64348" isv_metering_client_secret="$( cat "/mnt/c/Users/chgeuer/.secrets/3eb78160-e434-4713-8374-f40174d64348-metering.txt" )" isv_metering_client_id="${AZURE_METERING_MARKETPLACE_CLIENT_ID}" isv_metering_client_secret="=${AZURE_METERING_MARKETPLACE_CLIENT_SECRET}" # # You must now what you want to submit: # # Something like "2023-01-27T15:00:00Z". I'm always submitting for the top of the hour # timestamp="$( date --utc --date='-20 hour' '+%Y-%m-%dT%H:00:00Z' )" timestamp="$( date --utc '+%Y-%m-%dT%H:00:00Z' )" # Current hour timestamp="2023-01-27T15:00:00Z" # Specific hour echo "${timestamp}" planName="plan1" dimensionName="dimension-payg" quantity=5 # We now need to determine the resourceUri of the of the managed app (in the customer subscription). When submitting to the metering API, your JSON body must either contain # a resourceUri, or a resourceId, or both. In case of a managed app, resourceUri is the way to go. # # Please check my blog articles, one of those # - https://cookbook.geuer-pollmann.de/azure/marketplace-metering-ids # - https://techcommunity.microsoft.com/t5/fasttrack-for-azure/azure-marketplace-metered-billing-picking-the-correct-id-when/ba-p/3542373 # # The resourceUri we want to determine is the `managedBy` property of the managed resource group. In this sample, we go the long way, i.e. talk to ARM # on customer side. We query the **managed resource group**, using our management credential. # # The value we want to retrieve looks like this: # "/subscriptions/724467b5-bee4-484b-bf13-d6a5505d2b51/resourceGroups/managed-app-resourcegroup/providers/Microsoft.Solutions/applications/chgp20230118" # customer_subscription_id="724467b5-bee4-484b-bf13-d6a5505d2b51" customer_managed_resource_group_that_contains_the_resources="mrg-chgp20230118" # As the ISV, grab an access token for the ARM API management_access_token="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${isv_tenant_id}/oauth2/v2.0/token" \ --data-urlencode "response_type=token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_id=${isv_management_client_id}" \ --data-urlencode "client_secret=${isv_management_client_secret}" \ --data-urlencode "scope=https://management.azure.com/.default" \ | jq -r ".access_token" )" # Look at the token contents, if you like echo "${management_access_token}" | jq -R 'split(".") | .[1] | @base64d | fromjson' # managedBy="/subscriptions/724467b5-bee4-484b-bf13-d6a5505d2b51/resourceGroups/managed-app-resourcegroup/providers/Microsoft.Solutions/applications/chgp20230118" # Now query the properties of the managed resource group, and grab the managedBy property managedBy="$( curl --silent --get \ --url "https://management.azure.com/subscriptions/${customer_subscription_id}/resourcegroups/${customer_managed_resource_group_that_contains_the_resources}" \ --data-urlencode "api-version=2019-07-01" \ --header "Authorization: Bearer ${management_access_token}" \ | jq -r '.managedBy' )" echo "managedBy: ${managedBy}" # Now we can compose the JSON body for the marketplace call. # As you can see, the managedBy value goes as resourceUri. # meteringPayloadJson="$( echo "{}" | \ jq --arg x "${managedBy}" '.resourceUri=$x' | \ jq --arg x "${planName}" '.planId=$x' | \ jq --arg x "${dimensionName}" '.dimension=$x' | \ jq --arg x "${quantity}" '.quantity=($x | fromjson)' | \ jq --arg x "${timestamp}" '.effectiveStartTime=$x' \ )" echo "${meteringPayloadJson}" > meteringPayload.json # Now use the ISV's metering credential (the one in partner center), to fetch a token to talk to the metering API. # The ID 20e940b3-4c77-4b0b-9a53-9e16a1b010a7 is effectively the metering API. # isv_metering_access_token="$( curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${isv_tenant_id}/oauth2/token" \ --data-urlencode "response_type=token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_id=${isv_metering_client_id}" \ --data-urlencode "client_secret=${isv_metering_client_secret}" \ --data-urlencode "resource=20e940b3-4c77-4b0b-9a53-9e16a1b010a7" \ | jq -r ".access_token" )" # Look at the token's content. echo "${isv_metering_access_token}" | jq -R 'split(".") | .[1] | @base64d | fromjson' > isv_metering_access_token.json # Now finally send the data to Azure Marketplace... marketplace_response="$( curl \ --silent \ --request POST \ --url "https://marketplaceapi.microsoft.com/api/usageEvent?api-version=2018-08-31" \ --header "Authorization: Bearer ${isv_metering_access_token}" \ --header "Content-Type: application/json" \ --data "${meteringPayloadJson}" )" echo "${marketplace_response}" | jq . > marketplace_response.json ``` So the initial metering request looks like this: ```http POST https://saasapi.azure.com/api/batchUsageEvent?api-version=2018-08-31 HTTP/1.1 Host: saasapi.azure.com Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ii1LSTNROW5OUjdiUm9meG1lWm9YcWJIWkdldyIsImtpZCI6Ii1LSTNROW5OUjdiUm9meG1lWm9YcWJIWkdldyJ9.eyJhdWQiOiIyMGU5NDBiMy00Yzc3LTRiMGItOWE1My05ZTE2YTFiMDEwYTciLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC81ZjllNzQ4ZC0zMDBiLTQ4ZjEtODVmNS0zYWE5NmQ2MjYwY2IvIiwiaWF0IjoxNjc0ODQ2MzAxLCJuYmYiOjE2NzQ4NDYzMDEsImV4cCI6MTY3NDg1MDIwMSwiYWlvIjoiRTJaZ1lJaXkvSmtwMWlreFFmaGhzNDdWMFU5Y0FBPT0iLCJhcHBpZCI6IjNlYjc4MTYwLWU0MzQtNDcxMy04Mzc0LWY0MDE3NGQ2NDM0OCIsImFwcGlkYWNyIjoiMSIsImlkcCI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzVmOWU3NDhkLTMwMGItNDhmMS04NWY1LTNhYTk2ZDYyNjBjYi8iLCJvaWQiOiIyZTgyMjA4ZS1lYWQ5LTQwMTYtOTY1OC03NTM4NTUxYTE0MWYiLCJyaCI6IjAuQVJFQWpYU2VYd3N3OFVpRjlUcXBiV0pneTdOQTZTQjNUQXRMbWxPZUZxR3dFS2NSQUFBLiIsInN1YiI6IjJlODIyMDhlLWVhZDktNDAxNi05NjU4LTc1Mzg1NTFhMTQxZiIsInRpZCI6IjVmOWU3NDhkLTMwMGItNDhmMS04NWY1LTNhYTk2ZDYyNjBjYiIsInV0aSI6Ik5XdVVOWE5uOWtPWUtRQkZUMkV1QUEiLCJ2ZXIiOiIxLjAifQ.ecjxnxfuckoffsignaturetOtG8kuXyMr0uEExxxx Content-Type: application/json; charset=utf-8 Content-Length: 285 { "resourceUri":"/subscriptions/724467b5-bee4-484b-bf13-d6a5505d2b51/resourceGroups/managed-app-resourcegroup/providers/microsoft.solutions/applications/chgp20230118", "effectiveStartTime":"2023-01-27T18:00:00Z", "planId":"plan1", "dimension":"dimension-payg", "quantity":5.0 } HTTP/1.1 200 OK Content-Length: 464 Content-Type: application/json; charset=utf-8 Date: Fri, 27 Jan 2023 19:10:03 GMT mise-correlation-id: 505bdb68-7806-4ab7-9b1c-12bee2a3b5b8 x-ms-requestid: fa209de8-f599-4b49-9a25-fc48c3920fd9 x-ms-correlationid: fa209de8-f599-4b49-9a25-fc48c3920fd9 { "usageEventId":"5e1876e2-8b31-48fe-8810-864aeb67625b", "status":"Accepted", "messageTime":"2023-01-27T19:10:02.527833Z", "resourceId":"ffb5220f-0876-492c-b63b-26b73a1ad74f", "resourceUri":"/subscriptions/724467b5-bee4-484b-bf13-d6a5505d2b51/resourceGroups/managed-app-resourcegroup/providers/microsoft.solutions/applications/chgp20230118", "quantity":5.0, "dimension":"dimension-payg", "effectiveStartTime":"2023-01-27T18:00:00Z", "planId":"plan1" } ``` You can see the `"status":"Accepted"`, and a Microsoft-issued `"usageEventId"`. If you try to emit the same usage a 2nd time, you'll get a different response: ```json { "message": "This usage event already exist.", "code": "Conflict", "additionalInfo": { "acceptedMessage": { "usageEventId": "5e1876e2-8b31-48fe-8810-864aeb67625b", "status": "Duplicate", "messageTime": "2023-01-27T19:10:02.527833Z", "resourceId": "ffb5220f-0876-492c-b63b-26b73a1ad74f", "resourceUri": "/subscriptions/724467b5-bee4-484b-bf13-d6a5505d2b51/resourceGroups/managed-app-resourcegroup/providers/microsoft.solutions/applications/chgp20230118", "quantity": 5.0, "dimension": "dimension-payg", "effectiveStartTime": "2023-01-27T18:00:00Z", "planId": "plan1" } } } ``` Marketplace here tells us that the submission is `Conflict / Duplicate` of a previously submitted event. ## Links * **"Azure Marketplace Metered Billing- Picking the correct ID when submitting usage events":** To understand the difference between the `resourceUri` and `resourceId` in Azure Marketplace, please read my blog article , either in my [private](https://cookbook.geuer-pollmann.de/azure/marketplace-metering-ids) or the [official](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/azure-marketplace-metered-billing-picking-the-correct-id-when/ba-p/3542373) blog. * * # How can a publisher/ISV access the data plane of an Azure managed application? ![](/files/jFuZA3Pgn6PxOhSmET2x) ISVs can package and sell their solution as a managed application via Azure Marketplace, i.e. the ISV solution gets provisioned (via an ARM template) into the customer's environment. It's called 'managed' application because the publisher/ISV can indicate which identities from the publisher's Entra ID tenant are authorized to manage resources in the managed application. Authorized employees (or applications) in the publisher's tenant can 'see' the customer's resources in their own Azure portal (and ARM REST API). One way is to configure (in partner center) a group in the publisher's Entra ID tenant to be an `Owner` on managed resource groups. As a result, if the managed resource group for example contains a virtual machine, members of that group can turn the VM on or off. ## Control plane versus Data plane The `Owner` privileges configured in Azure partner center *only* apply to *control plane* operation, that is, to calls against managed resource group resource via the ARM API. For example, if the managed resource group contains a storage account, and the storage account allows access via storage account access keys, then authorized members at the publisher side can reveal the storage account's access keys (technically a `POST` call against the ARM API). > Please note that the access tokens used by publisher employees to hit the ARM API are *issued* by the **publisher's** Entra ID tenant. For managed apps, the ARM API 'knows' that even if the token is not coming from the customer's Entra ID tenant, the access is still OK if it comes from the publisher. If the service access should happen using Entra ID authentication (a Bearer / access token), or the service only supports Enta ID authN, then the previously mentioned 'access key' mechanism doesn't work, so Entra ID it is. However, for data plane operations (like updating a secret in KeyVault or uploading a blob to blob storage), the access token must be issued by the **customer's** Entra ID tenant. And the publisher has no accounts in the customer's tenant. During the provisioning of the managed application, you also cannot create a service principal with client\_id and client\_secret in the customer's tenant. ## How to get a access token from the customer's Entra ID tenant? There are 3 ways how this can be solved: ### Option 1 (highly recommended): Use the ARM control plane to list an access token In Azure Marketplace managed applications, you must distinguish between the "managed application" object, and the "managed resource group" that belongs to the managed application: The managed application object if of ARM resource type `Microsoft.Solutions/applications`. This is the object that is fully customer-manageable; while the customer might not be allowed to delete resources within the managed resource group, they can delete the managed application object itself (which results in deletion of the corresponding resource group). A 'managed application object' can have a managed identity ([see the docs](https://learn.microsoft.com/en-us/azure/azure-resource-manager/managed-applications/publish-managed-identity)). In the ARM template, the ISV can create an RBAC assignment to make the managed application a Secrets Officer on the Key Vault resource. This managed identity lives in the right tenant (the customer tenant), and is authorized to access Key Vault (or other data planes). But how to get a token? The [docs](https://learn.microsoft.com/en-us/azure/azure-resource-manager/managed-applications/publish-managed-identity#accessing-the-managed-identity-token) show the necessary API call: ```http POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Solutions/applications/{applicationName}/listTokens?api-version=2018-09-01-preview HTTP/1.1 Authorization: Bearer {{token issued by the publisher tenant, to hit the ARM API, audience https://management.azure.com/}} { "authorizationAudience": "https://vault.azure.net/" } ``` On the ISV side, an authorized identity first fetches a token for the ARM control plane (audience `https://management.azure.com/`), and then POSTs against the `Microsoft.Solutions/applications` resource at the customer side, with the `listTokens` operation, to fetch a token of the managed app. Alternatively, if there is a user-assigned managed identity in the managed resource group that is authorized to do whatever you want, you can also supply the `"userAssignedIdentities": [...]` identity for which you'd like a token. ### Option 2 (for the adventurous): Workload identity federation In addition to previously described approach of POSTing to the ARM API to list an access token (e.g. for a user-assigned managed identity), you can use workload identity federation to sign-in directly to the UAMI *from a trusted IdP*. I have a small demo for this approach in my GitHub repo [chgeuer/metering\_cloudshell](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/azure/chgeuer/metering_cloudshell/README.md). In this example, the UAMI is configured to allow federated sign-in from an identity provider. So one first has to get an access token from that trusted IdP, and then do a client credentials grant against the customer's Entra tenant, with a `"client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer"`. ```shell curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/{customer tenant}/oauth2/token" \ --data-urlencode "resource=20e940b3-4c77-4b0b-9a53-9e16a1b010a7" \ --data-urlencode "response_type=token" \ --data-urlencode "grant_type=client_credentials" \ --data-urlencode "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \ --data-urlencode "client_id={client id of the UAMI}" \ --data-urlencode "client_assertion={access token from the IdP}" ``` > Up until December 2024, you **could not** configure the publisher's Entra ID tenant to be a trusted issuer for a federated credential in the customer's Entra ID tenant. That would've resulted in an `AADSTS700222` error. That's why in my sample [chgeuer/metering\_cloudshell](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/azure/chgeuer/metering_cloudshell/README.md) I created a 'fake' IdP by storing the IdP metadata in a storage account. > > However, since 18. December 2024, there is a public preview of "[Public Preview of Managed Identities as Federated Identity Credentials for Microsoft Entra](https://devblogs.microsoft.com/identity/access-cloud-resources-across-tenants-without-secrets/)", in which one can specify another Entra ID tenant as trusted issuer for workload identity. ### Option 3 (legacy, bad): Managed identity attached to a VM In the past, publishers attached an authorized managed identity in the managed resource group to a virtual machine in the managed resource group. They then remotely signed in the the VM (via SSH), and fetched a token for the managed identity by talking the the VM's instance metadata service' endpoint (IMDS), on IP 169.254.169.254. However, this is very inflexible (having to have a running VM), hard to automate, expensive and cumbersome (need a running VM), and certainly created security issues (remote connections to a VM might not give cozy feelings). # The checkZonePeers API: Is your availability zone "1" equal to my "1"? > This is a local copy of my article on the [FTA blog](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/is-your-availability-zone-quot-1-quot-equal-to-my-quot-1-quot/ba-p/3562349)... ![availability zones](https://docs.microsoft.com/en-us/azure/availability-zones/media/availability-zones.png) ## Introduction Since March 2018, [Azure Availability Zones](https://docs.microsoft.com/en-us/azure/availability-zones/az-region) (AZs) are generally available. *"Availability Zones are physically separate locations within an Azure region. Each Availability Zone consists of one or more datacenters equipped with independent power, cooling, and networking."* Take for example the Azure Region '*West Europe*', which is in the 'Geography Europe', and is the Microsoft'ish region name for Amsterdam (in the Netherlands (in Europe (on Earth))). West Europe is equipped with 3 availability zones. Roughly speaking, these availability zones are 1 milli-second network latency away from each other, or something like 30-40 km (if you have to walk), so if a local disaster renders resources in one AZ unavailable, you hopefully have some resiliency baked into your solution architecture, with failover resources in the second and third AZ. When you spin up resources in Azure, some of these are so-called "zonal services": > A [zonal service](https://docs.microsoft.com/en-us/azure/availability-zones/az-region#highly-available-services) or resource *"supports AZs, and can be deployed to a specific, self-selected availability zone, to achieve more stringent latency or performance requirements."* A zonal resource's ARM JSON representation contains a `zones` array. For example, let's look at a (zonal) IP address: ```json { "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/...", "type": "Microsoft.Network/publicIPAddresses", "name": "ipproductionweb", "resourceGroup": "webfrontend", "location": "westeurope", "zones": [ "1" ], ... } ``` Great, we know now that our IP address 'lives' in West Europe (Amsterdam), in availability zone "1" (whatever that might mean). This information allows us to place other resources in the same AZ. For example, the network interface card or load balancer that you attach that IP address to, certainly has to be in AZ 1 as well. When you place multiple resources into AZ 1, you know they'll be closer together as if one of them would be in a different AZ. The AZ number helps you plan out a deployment, by deliberately placing resources in the same, or in different AZs. The availability zone numbers are meant to be used only within a single Azure region: AZ1 in West Europe (Amsterdam) has absolutely nothing to do with AZ1 in North Europe (Dublin in Ireland), except that they might be cooled by the same chilly North Sea air. **Assuming you have a single Azure subscription, that's enough you need to know, you can stop reading here. I appreciate your time and attention, check our official docs, and have a pleasant day.** If you're still with me, chances are that you or your company have multiple Azure subscriptions. You might even have multiple Azure AD tenants. Or you might work with subscriptions from customers or partners, who 'live' underneath the partner's Azure AD tenant. And this is where things get more complex: **The "availability zone 1 in West Europe" in subscription 'X' is NOT necessarily equivalent to "availability zone 1 in West Europe" in another subscription 'Y'!** When naming things like 1, 2, 3 and so forth, we humans have the tendency that start using the thingie called "1" first, and once we need a second thing, we use thingie "2", etc. If the Azure team would have made Availability Zone "1" (in a certain region) the same for all of us, then that AZ would quickly demonstrate that "cloud is just the illusion of infinite resources"; all customers would deploy their main workload into AZ1, and only gradually leverage AZ2, and maybe, just maybe, deploy something into AZ3. AZ1 would run hot quickly, while AZ2 would have modest load. And AZ3 would be wondering which business justification made it being built in the first place. Therefore, the Azure team decided to 'shuffle' the mapping of the availability zone identifiers to 'real/physical' availability zone, on a per-subscription basis. Once a subscription is created, that mapping will remain unchanged for the lifetime of that subscription. So now imagine you want to deploy a resource (like a database VM) in your own subscription, but you want it to be physically placed in the very same availability zone which *my subscription* calls "AZ 2". How can you find out which AZ number (in *your* own subscription) you have to pick, to deploy close to my workload? You can find out using the Azure `checkZonePeers` API, which gives you a mapping table. ## How does the `checkZonePeers` API work? The [`checkZonePeers` API](https://docs.microsoft.com/en-us/rest/api/resources/subscriptions/check-zone-peers) lets you retrieve a mapping table, which tells you how other subscriptions call an AZ, which your subscription knows under a certain name. Let's say 'our' reference subscription ID, relative to which we want to determine AZ names (in the `westeurope` region) is `11111111-1111-1111-1111-111111111111`. We ask the API: > Hey, I am `11111111-1111-1111-1111-111111111111`, and I am interested how two other subscription IDs (`22222222-2222-2222-2222-222222222222` and `33333333-3333-3333-3333-333333333333`), call 'my' AZs in `westeurope`... The underlying REST API call looks something like this: ```http POST /subscriptions/11111111-1111-1111-1111-111111111111/providers/Microsoft.Resources/checkZonePeers?api-version=2020-01-01 Host: management.azure.com Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IjJaUXBKM1V... Content-Type: application/json { "location": "westeurope", "subscriptionIds": [ "/subscriptions/22222222-2222-2222-2222-222222222222", "/subscriptions/33333333-3333-3333-3333-333333333333" ] } ``` In the URL, you have the subscription ID which is your 'north star', which defines the baseline of what AZ names are normative for you. In the request body, you specify the Azure region and the other subscriptions you want to have a mapping for. The API's response looks like this: ```json { "subscriptionId": "11111111-1111-1111-1111-111111111111", "location": "westeurope", "availabilityZonePeers": [ { "availabilityZone": "1", "peers": [ { "subscriptionId": "22222222-2222-2222-2222-222222222222", "availabilityZone": "1" }, { "subscriptionId": "33333333-3333-3333-3333-333333333333", "availabilityZone": "3" } ] }, { "availabilityZone": "2", "peers": [ { "subscriptionId": "22222222-2222-2222-2222-222222222222", "availabilityZone": "3" }, { "subscriptionId": "33333333-3333-3333-3333-333333333333", "availabilityZone": "2" } ] }, { "availabilityZone": "3", "peers": [ { "subscriptionId": "22222222-2222-2222-2222-222222222222", "availabilityZone": "2" }, { "subscriptionId": "33333333-3333-3333-3333-333333333333", "availabilityZone": "1" } ] } ] } ``` Therefore, you have the following mapping: | Subscription | AZ X | AZ Y | AZ Z | | ------------------------------------ | :--: | :--: | :--: | | 11111111-1111-1111-1111-111111111111 | 1 | 2 | 3 | | 22222222-2222-2222-2222-222222222222 | 1 | 3 | 2 | | 33333333-3333-3333-3333-333333333333 | 3 | 2 | 1 | AZ 1 in our main sub (`11111111-...`) is AZ "1" as well in `22222222...`, but it's called AZ "3" in `33333333-...`. ## Subscriptions in different Azure ADs, but only a single `Authorization` header? The aforementioned approach works - if all your subscriptions are governed by the very same Azure AD tenant; you fetch the access token, do the lookup for all your subscriptions, and be done. However, if these subscriptions are hooked up to different AAD tenants, you need to demonstrate to the `checkZonePeers` API that you are authorized to access all of them. Unfortunately, the API only accepts a single bearer token in the HTTP Authorization header. How can we supply the other tokens? For such tenant-boundary-spanning API calls, the Azure Resource Manager API has a custom HTTP header, called `x-ms-authorization-auxiliary`, which can hold up to three 'auxiliary' tokens. If the primary token (from the base subscription's AAD tenant) goes into `Authorization: Bearer ...`, you can put three additional AAD tokens from other AAD tenants into the `x-ms-authorization-auxiliary`. > Please note that all of these tokens need to belong to the same subject, or user. Your Azure AD user must have an account (or be a guest) in all relevant AAD tenants, and of course you need to be a Reader on the underlying subscriptions. Otherwise, you will get an `AuxiliaryTokensInvalidUserIdentity` error, indicating that `Authentication failed for auxiliary token: The '1' auxiliary tokens are from the client(s) 'live.com#foobar@hotmail.com' which are different from the client of primary identity 'chgeuer@microsoft.com'.` ## Enable the `Microsoft.Resources/AvailabilityZonePeering` feature for your subs You might have to register usage of that API first. You can check that the feature is enabled, using this command: ```shell az feature show \ --namespace Microsoft.Resources \ --name AvailabilityZonePeering \ --subscription chgeuer-work \ | jq -r '.properties.state' ``` If you don't get back `"Registered"`, make sure you register for the feature in all relevant subscriptions. ```shell az feature register \ --namespace Microsoft.Resources \ --name AvailabilityZonePeering ``` ## A practical demo script (pure `bash`, `curl` and `jq`) Below, you can find a demo script, which you of course need to adapt to your environment, i.e. filling in the appropriate Azure AD tenant IDs and Azure subscription IDs. In my environment, I have three Azure subscriptions and each of them is hooked up to a different Azure AD tenant. | Azure Subscription ID | Azure Active Directory Tenant ID | | -------------------------------------- | -------------------------------------- | | `11111111-1111-1111-1111-111111111111` | `aadaadaa-1111-1111-1111-111111111111` | | `22222222-2222-2222-2222-222222222222` | `aadaadaa-2222-2222-2222-222222222222` | | `33333333-3333-3333-3333-333333333333` | `aadaadaa-3333-3333-3333-333333333333` | In order to make it simple, I'm running this in a Linux shell, under WSL, on my Windows laptop. For each of the (in my case three) required access tokens, I'm doing a device login, so I can simply authenticate in my Windows web browser. If that script detects that it's running within WSL, then I copy the device login's user code into the Windows clipboard, and kick the (Windows-side) default web browser to the device login page, so I just need to Ctrl-V paste the device user code, and do my authentication dance. In addition to the actual response from the REST API, I output some claim contents from the various access tokens, which looks something like this: ``` access_token_1 Issuer: iss="https://sts.windows.net/aadaadaa-1111-1111-1111-111111111111/" access_token_1 Subject: sub="djwNuWxvH-6cUIHWIwRBjVanUvsrG5Ty6eJMqcK722U" access_token_2 Issuer: iss="https://sts.windows.net/aadaadaa-2222-2222-2222-222222222222/" access_token_2 Subject: sub="djwNuWxvH-6cUIHWIwRBjVanUvsrG5Ty6eJMqcK722U" access_token_3 Issuer: iss="https://sts.windows.net/aadaadaa-3333-3333-3333-333333333333/" access_token_3 Subject: sub="djwNuWxvH-6cUIHWIwRBjVanUvsrG5Ty6eJMqcK722U" ``` As you can see, the Subject (`sub`) is the same in all of them (as mentioned earlier, this is needed)... ```sh #!/bin/bash aad1="aadaadaa-1111-1111-1111-111111111111" sub1="11111111-1111-1111-1111-111111111111" aad2="aadaadaa-2222-2222-2222-222222222222" sub2="22222222-2222-2222-2222-222222222222" aad3="aadaadaa-3333-3333-3333-333333333333" sub3="33333333-3333-3333-3333-333333333333" function deviceLogin { local tenant="$1" ; local resource="$2" ; deviceResponse="$(curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/${tenant}/oauth2/v2.0/devicecode" = --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \ --data-urlencode "scope=${resource}" \ )" ; device_code="$(echo "${deviceResponse}" | jq -r ".device_code")" ; sleep_duration="$(echo "${deviceResponse}" | jq -r ".interval")" ; access_token="" ; if [[ $(grep --ignore-case Microsoft /proc/version) ]]; then # On WSL, copy response code to clipboard, and launch user's web browser echo "$( echo "${deviceResponse}" | jq -r ".user_code" )" | iconv -f utf-8 -t utf-16le | clip.exe cmd.exe /C "start $( echo "${deviceResponse}" | jq -r ".verification_uri" )" fi while [[ "${access_token}" == "" ]] do tokenResponse="$(curl \ --silent \ --request POST \ --url "https://login.microsoftonline.com/{aadTenant}/oauth2/v2.0/token" \ --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:device_code" \ --data-urlencode "client_id=04b07795-8ddb-461a-bbee-02f9e1bf7b46" \ --data-urlencode "device_code=${device_code}" \ )" ; if [ "$(echo "${tokenResponse}" | jq -r ".error")" == "authorization_pending" ]; then >&2 echo "$(echo "${deviceResponse}" | jq -r ".message")" ; sleep "${sleep_duration}" ; else access_token="$(echo "${tokenResponse}" | jq -r ".access_token")" ; fi done ; echo "${access_token}" } function showToken { local name="$1" ; local access_token="$2" ; claims="$( jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "${access_token}" )" echo "${name} Issuer: iss=$( echo "${claims}" | jq .iss )" # echo "${name} Audience: aud=$( echo "${claims}" | jq .aud )" echo "${name} Subject: sub=$( echo "${claims}" | jq .sub )" } echo "Please login to the tenants using the same user ID..." arm_api="https://management.azure.com/.default" access_token_1="$( deviceLogin "${aad1}" "${arm_api}" )" access_token_2="$( deviceLogin "${aad2}" "${arm_api}" )" access_token_3="$( deviceLogin "${aad3}" "${arm_api}" )" # # Show selected contents of the access tokens, like issuer and subject. # showToken "access_token_1" "${access_token_1}" showToken "access_token_2" "${access_token_2}" showToken "access_token_3" "${access_token_3}" location="westeurope" checkZonePeersBody="$( echo "{}" \ | jq --arg x "${location}" '.location=$x' \ | jq '.subscriptionIds=[]' \ | jq --arg x "/subscriptions/${sub2}" '.subscriptionIds[.subscriptionIds | length] |= .+ $x' \ | jq --arg x "/subscriptions/${sub3}" '.subscriptionIds[.subscriptionIds | length] |= .+ $x' \ )" curl \ --silent \ --request POST \ --url "https://management.azure.com/subscriptions/${sub1}/providers/Microsoft.Resources/checkZonePeers?api-version=2020-01-01" \ --header "Authorization: Bearer ${access_token_1}" \ --header "x-ms-authorization-auxiliary: Bearer ${access_token_2}, Bearer ${access_token_3}" \ --header "Content-Type: application/json" \ --data "${checkZonePeersBody}" \ | jq "." ``` ## Relevant links * [Availability Zones Overview | Azure](https://docs.microsoft.com/en-us/azure/availability-zones/az-overview) * [Check Zone Peers | Subscriptions | Azure REST API](https://docs.microsoft.com/en-us/rest/api/resources/subscriptions/check-zone-peers) * [Authenticate requests across tenants | Azure Resource Manager API](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/authenticate-multi-tenant) # Token authentication with "Azure Verizon Premium CDN" This walkthrough demonstrates how to use "Azure CDN Premium from Verizon" in front of a private blob storage container as origin. ## Overview In this system, the client's browser should securely access an asset stored in a private container in blob storage, via CDN. *Secure* in this context means that access to the cached asset in CDN should only be granted to authorized clients, and access from the CDN to the origin should be authenticated as well. The term 'origin' usually refers to the service 'behind' the CDN, in our case an Azure Storage Account. To implement this, the client signs in 'somehow' to the web application, using whatever is in place (step 1 below). The web application generates a 'CDN auth token', and tells the browser to fetch the asset from CDN (step 2 below). That CDN URL contains the CDN auth token in the URL's query string. The CDN endpoint (strictly speaking the CDN's *point of presence* (PoP) or *edge node*) validates the CDN auth token, and (when allowed) either serves the cached copy of the asset, or fetches the contents from the origin (step 3 below). ![](/files/X0JsyDY2z2WBJMNV3KXt) ## Create a storage account * Config * Storage account name: `cdnorigi` * Force HTTPS * Enable storage account keys ### Create a private container named `cdnfiles` * Inside the container, create an "Stored access policy" * Policy identifier: `cdn` * Permissions: `[Read]` * Start time: Yesterday * Expiry time: Something far in the future * After creating the policy, go the the container's "Shared access tokens" tab * Signing key: Key 1 or Key 2 * Stored access policy == `cdn` * Click "Generate SAS token and URL" * The Blob SAS token looks like this: `si=cdn&spr=https&sv=2021-06-08&sr=c&sig=Eod7p4YFtfp2rQX2OWT9ZJbNogCNa3Y%2FoH7O82%2F4UdI%3D` * The Blob SAS URL looks like `https://cdnorigi.blob.core.windows.net/cdnfiles?si=cdn&spr=https&sv=2021-06-08&sr=c&sig=Eod7p4YFtfp2rQX2OWT9ZJbNogCNa3Y%2FoH7O82%2F4UdI%3D` * Upload a sample file to the container * After uploading the file, check that the SAS actually works, by putting the filename into the middle of the SAS URL, such as * `https://cdnorigi.blob.core.windows.net/cdnfiles/2021-01-11-Captain-Haddock-02.jpg?si=cdn&spr=https&sv=2021-06-08&sr=c&sig=Eod7p4YFtfp2rQX2OWT9ZJbNogCNa3Y%2FoH7O82%2F4UdI%3D` ### Create a new "Front Door and CDN profiles" offer via the Azure Portal * Select "Explore other offerings" (instead of the default "Azure Front Door") * In "Choose other offerings", pick "Azure CDN Premium from Verizon" * In Profile Details * Name: `cdnchgeuer` * Region: `Global`, Pricing Tier: `Premium Verizon` * Tick the "Create a new CDN endpoint" box * CDN Endpoint name: `cdnchgeuer20230119` * which means our public hostname will be `https://cdnchgeuer20230119.azureedge.net`, unless we later enable a custom domain name. * Origin type: "Storage" * Origin hostname: Pick your storage account, in my case `cdnorigi.blob.core.windows.net` * Click `Create`, and take a cup of coffee. * What now happens is that Azure in the back calls out to APIs of Edgecast/Verizon to provision a CDN profile for you. That can take a while, because Verizon under the hood will roll out your chosen hostname globally to all the various edge nodes (also called 'points of presence') * Configure the endpoint: Once provisioning is finished, in the resource group you should see the Storage account, the CDN profile, and the CDN endpoint: ![Azure Portal Resources](/files/LwaUQ6AdCwb4yIWvHLE5) In the endpoint's "Origin" settings, you might consider disabling the `http` port ![Azure Portal CDN Origin](/files/bG9WNDYW2zsCgRgh9lKY) All the rest of the fine-grained configuration happens by navigating to "Advanced Features" and clicking the "Manage" link at the top of the page: ![Azure Portal Advanced CDN Features](/files/a56OO7YCS4Fm0q9y4Lxg) You now are authenticated and redirected to `cdn.windowsazure.com/http/token/default.aspx`, which is the place where we can configure token authentication: ![Verizon Portal empty token auth configuration](/files/x3XoP4F7ftBhcIoH40WW) ## Configure the `HTTP Large Object Token-Based Authentication` * In the Linux shell (for example in a bash shell in WSL), we'll be using OpenSSL to generate to high-entropy cryptographic keys: * Run `openssl rand -hex 160` twice and note down the results. This command generates hexadecimal encoded cryptographically strong random numbers, and the number is the number of bytes. 160 bytes would be 1280 bit, encoded in 320 hex characters. * Prefix the first value with `primary` and the second one with `backup`, so it's easier to see which key you're looking at. ``` primary4c5604fb8f76524cd99cf65700fd550a92f083d297c327a50cae5891f1f6fcb6965cadaaa56682e3998e376f1cb3db757348356f73a936bfdae12c4eab07e02e70d5108593a97f267cdb2c16c87c58832270a7cfd0199ecd0808169f54238be400e7950f44bce9f801395061cf741ac1f12ea2341b8887c2b6692b6074695854834047c8bff52b3a284a51290483a985f78ad0c548d98fe306c9be3cd12605b8 backupf876ca22ccca6cf4587137251e44490a131b8cca12ab65bbdd8ffa33edeb9862f850d2050e9d3df1b2bcc72ca5ddbf3f32145c846464ce8cfadd6f36881371f88ca6942f1a685211e3eac25cb9b79bc50aad65c7b04d754637de2ee7e8429fdc25a7556e3628899a43ca94178c6d3777229b345132cada7aef4bf7f2c68bb0b04c109ece2a301486c61d5a85b5b2eaaf55ae4e06b3198ec1c9c2606f9b33c2a0 ``` * You can also bump up the "Minimum Encryption Version" from `V2` to `V3` ![Verizon Portal populated token auth configuration](/files/oCDUt1DFYPnhzDc2pi4X) ### Configure custom rules By clicking on the "HTTP Large" menu on the top, and selecting "Rules Engine V4.0", you come to the graphical rule editing experience. ![Verizon Portal empty rules](/files/QjkXNdkEPznl7mKntuCT) Click the `+New` button, and type in a new name for the draft rule, such as `v1`, and click "Continue" #### Create a first rule, to **require token authentication** * Click the `+Rule` button * Rule Description: `Require CDN Token` * Click the `+` dropdown and select `Match` * The `Select Category` dropdown should be `URL` * The second dropdown should be `URL Path Directory Wildcard` * The `Result` dropdown should be `Match` * The `Value` is the path in the CDN URL, something like `/cdnfiles/`. * The `Relative To` dropdown should be `Origin` * **Enforce token auth:** Next, click the indented `+` and select `Feature` * ![Verizon Portal Rule 1 creation](/files/IP3HTAJZH97URNriF5DL) * `Select Category` dropdown == `Access` * `Select Feature` dropdown == `Token Auth` * `Enabled` == `yes` ![image-20230119162214767](/files/KhcljmiFkhdCf6OpT3Ld) * **Name the token parameter**: * Add another `Feature`, this time `Access` and `Token Auth Parameter`, and `Enabled=yes` * In the `Name` text box, pick the name under which your web app will convey CDN tokens to the CDN. That could be `cdntoken` ![Verizon Portal Rule 1 finished](/files/7dNt6bWxb3xkl0xcSti0) Finally, save our first rule. #### Create a second rule, using URL Rewriting to **configure access to the private storage account container, using our SAS** * Click `+Rule` to add a second rule, with a rule description of `Append SAS Token` or similar * Click the `+` button and again select `Match`, this time with `General` / `Always` * On the indented `+` button, add a `Feature` in the `URL` category, with feature being `URL Rewrite` * > The `Source` and `Destination` text boxes have a little resize handle in the bottom right, you might want to make these text boxes larger to see what you're putting here. ![image-20230119163159094](/files/TBxMnTpCJDuBJ9H4burQ) * Now we need to tell the CDN how incoming URLs should be re-written to the external origin server (Azure Blob storage in our case). * To know the string we need to know, temporarily click the bottom `+` button to add another `Match`, and select `Origin` / `Customer Origin`. In the Value dropdown, you should see something like this: ![Verizon Portal customer ID](/files/Fb7bTIxLvyl5NjCxHxAv) * In the dropdown (marked with a `1` in the screenshot), you can see a customer origin identifier, in our case `/801A0567/cdnchgeuer20230119/`. Please type this text into the `Source` and `Destination`. The `cdnchgeuer20230119` part in this string is the name of the endpoint which you configured earlier in the Azure portal. * However, the `/801A0567/` looks slightly random. In practice, this is your Verizon customer ID, which you can see in the upper righthand corner of the portal (`1A0567`), prefixed with the `80`. So in theory, you could just look at your user account info on screen, but in case that `80` prefix changes in the future, temporarily creating a customer origin rule makes it easy to lookup. * Now please delete the temporarily created match, by clicking the little recycle bin icon (`2`). Our screen should now look like this: ![Verizon Portal Rule 2 creation step 2](/files/UMiHatuVMdTptxroriN8) * Now that our `Source` and `Destination` textboxes have the base information, we need to extend it with a mechanism to inject our shared access signature token. * In the `Source` textbox, append the name of your container, and the regular expression `(.*)\?(.*)` to the content, so it looks like this: `/801A0567/cdnchgeuer20230119/cdnfiles/(.*)\?(.*)` * The request coming on to the CDN looks something like this: `https://cdnchgeuer20230119.azureedge.net/cdnfiles/somefile.zip?cdntoken=xxx`, i.e. the path prefix `/cdnfiles/`, a file (`somefile.zip`), and the query string containing our CDN token (`?cdntoken=xxx`). * The regular expression cracks up the path around the questionmark (`\?`), into the file name (`somefile.zip`) in the first regex capture group, and the query string (`cdntoken=xxx`) in the second regex capture group. * In the `Destination` text box, we concatenate the following strings * Our base `/801A0567/cdnchgeuer20230119/` * Our storage account container name `cdnfiles/` * Then we put the file name, which is captured in the 1st capture group, by adding `$1` * The `?` to start the query string again * Our SAS string (`si=cdn&spr=https&sv=2021-06-08&sr=c&sig=Eod7p4YFtfp2rQX2OWT9ZJbNogCNa3Y%2FoH7O82%2F4UdI%3D`) * An additional `&` for additional parameters * The second capture group `$2` * So the overall result is `/801A0567/cdnchgeuer20230119/cdnfiles/$1?si=cdn&spr=https&sv=2021-06-08&sr=c&sig=Eod7p4YFtfp2rQX2OWT9ZJbNogCNa3Y%2FoH7O82%2F4UdI%3D&$2` ![Verizon Portal Rule 2 finished](/files/4qenjljJsPy3feIFlvlJ) * Save the second rule, and click "Lock Draft as Policy" * In the `Raw XML` window, you can also see an XML representation of the rules ```xml Require CDN Token Append SAS Token ``` * Now, click the "Deploy Request" Button, select the "Production" environment, type in a deployment message (like "Deploying v1"), and "Create Deploy Request" ![Lets push to prod](/files/n8YiNziVn3GJ9cL40bnM) * Once submitted, the Verizon system will validate the rules, after a couple of seconds, you should be seeing an "Approved" mark in the Activity log. Overall, the deployment of the rules to the various edge / PoP nodes might take a few minutes. ## Create a CDN token * Finally, we can create a CDN token, to test our solution. We navigate back to the "HTTP Large / Token Auth" page. Further down on that page is an "Encrypt Tool" section, a small web-based UI to create a CDN token. For a simple test, we need two properties in the token, an expiration date (`ec_expire`) and a part of the URL we're granting access to (`ec_url_allow`). * The `ec_expire` header must be an epoch value, an integer which you can create using web sites such as [unixtimestamp.com](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/.gitbook/assets/202301-verizoncdn/https:/www.unixtimestamp.com) or [epochconverter.com](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/.gitbook/assets/202301-verizoncdn/https:/www.epochconverter.com). For example, the timestamp "Wed Dec 31 2025 23:59:59 UTC" corresponds to the epoch value `1767225599`. * The `ec_url_ allow` can contain values such as `/cdnfiles`. * Encrypting these two parameters under my primary key gives me a generated token of `is9St4wZ7gTK8tuPXxKQtcmh7Bi3aMJ8b-yHxvJgyDdcJOyrePUmVIj9HXVFl1WbLWRDiuS-3RaeEgpg-JIsJT9ChNKfqis` In your solution, you (of course) won't visit the Verizon portal to generate a CDN token. You will be using your web application to (protected) generate web links pointing to CDN. So in your web solution, you need to generate these CDN tokens programatically, you certainly want to make these short-lived, limit them to a specific client, and use all the other good features. On the .NET side, I created a small library [Azure-Samples/edgecast-cdn-token-fsharp: A .NET sample to create tokens for Edgecast Azure CDN](https://github.com/Azure-Samples/edgecast-cdn-token-fsharp/) which can help you generating such tokens. In a C# solution for example, you could generate tokens like this: ```csharp using System; using EdgecastCryptoExtensions; var key = "primary4c5604fb8f76524cd99cf65700fd550a92f083d297c327a50cae5891f1f6fcb6965cadaaa56682e3998e376f1cb3db757348356f73a936bfdae12c4eab07e02e70d5108593a97f267cdb2c16c87c58832270a7cfd0199ecd0808169f54238be400e7950f44bce9f801395061cf741ac1f12ea2341b8887c2b6692b6074695854834047c8bff52b3a284a51290483a985f78ad0c548d98fe306c9be3cd12605b8"; var token1 = EdgecastCrypto .createTokenValidUntil( new DateTime( year: 2025, month: 12, day: 31, hour: 23, minute: 59, second: 59)) .AddAllowedUrl("/cdnfiles/") .Encrypt(key); var token2 = EdgecastCrypto .createTokenValidFor(TimeSpan.FromDays(365.0)) //.AddAllowedCountry("DE") // more extension methods available .AddAllowedUrl("/cdnfiles/") .Encrypt(key); Console.Out.WriteLine($"Token1: {token1}"); // Token1: sU5Hu8WjfvysnYVmy6i4nsdNbQGrlnc7snQN075VgHJAj_XVnehjA - xGohmvDwCdJHGFirgBVoAoUmSdrtajA76KwB_5Hsw3 Console.Out.WriteLine($"Token2: {token2}"); // Token2: sU5Hu8WjfvysnYVmy6i4nsdNbQGrlnc7tHYI0bpZj3lAj_XVnehjA - xGohmvDwCdJHGFirgBVoD1w7xarONEKKhIGWp9nrT2 ``` ## Accessing a file. With all that information, we can now hit the CDN endpoint and access a file in our private blob storage container. As a quick reminder, the direct URL to hit our origin server (blob storage) contained the shared access signature: ``` https://cdnorigi.blob.core.windows.net/cdnfiles/2021-01-11-Captain-Haddock-02.jpg?si=cdn&spr=https&sv=2021-06-08&sr=c&sig=Eod7p4YFtfp2rQX2OWT9ZJbNogCNa3Y%2FoH7O82%2F4UdI%3D ``` Instead of the SAS, the CDN URL contains our `cdntoken`: ``` https://cdnchgeuer20230119.azureedge.net/cdnfiles/2021-01-11-Captain-Haddock-02.jpg?cdntoken=is9St4wZ7gTK8tuPXxKQtcmh7Bi3aMJ8b-yHxvJgyDdcJOyrePUmVIj9HXVFl1WbLWRDiuS-3RaeEgpg-JIsJT9ChNKfqis ``` When we hit the CDN without token, or a modified one, we get a `403 - Forbidden` page. Otherwise, you can enjoy this beautiful image: ![](/files/QItxV20UqOxkFg6gNFCj) # Getting the right storage container name in a Bicep template When creating a blob storage container in an ARM/Bicep template, as a non-nested resource, you create a top-level resource like `Microsoft.Storage/storageAccounts/blobServices/containers` . However, the `name` property of that container cannot simply be something like `somecontainer`, because it needs to indicate the storage account, and the blob service. So the `name` must be something like `mystorageaccount/default/somecontaine`. If you want to use the container in other places of the Bicep template, you certainly want to refer to the container object and take the name from there, rather then using the literal container name string (and them preventing Bicep from creating the appropriate `dependsOn`). But referring to `container.name` unfortunately gives us the 3-segment name, and we only want the last segment. Therefore, we need to use ARM string functions to retrieve the last segment, i.e. the text after the last `/` character in the name. Assuming you have a string `x`, you could use this Bicep expression to extract the name, i.e. for `a/b/c` get `c`: > `substring(x, lastIndexOf(x,'/') + 1, length(x) - lastIndexOf(x, '/') - 1)` See below for a quick Bicep sample: ```bicep @description('The location for the deployment') param location string = resourceGroup().location var names = { uniqueHostPrefix: 'x${uniqueString(resourceGroup().id)}' containerName: 'SomeContainer' } resource storageAccount 'Microsoft.Storage/storageAccounts@2022-05-01' = { name: names.uniqueHostPrefix location: location kind: 'StorageV2' sku: { name: 'Standard_LRS' } } resource container 'Microsoft.Storage/storageAccounts/blobServices/containers@2022-05-01' = { name: '${storageAccount.name}/default/${toLower(names.containerName)}' } output theContainerNameVariable string = names.containerName output naiveContainerName string = container.name output containerName string = substring(container.name, lastIndexOf(container.name, '/') + 1, length(container.name) - lastIndexOf(container.name, '/') - 1) ``` Deploying this template would give you the following outputs (just with a different storage account name): ![2023-01-30--storage-container-names](/files/75rdyQi4rxfgV5ycCiil) # Event-sourcing into working memory to improve data access latency ![Cover Illustration](/files/7ZKGJ2I1bUYESRam4UQv) In this article, we describe an architecture modernization, moving from a database-centric access pattern, towards event-sourcing data changes directly into the application's working memory (RAM). ## Modernization scenario ### Overview The starting point for our modernization is a 'traditional' web application. The application depends on both internal data (such as a database) and data from external third party APIs. For example, the internal data could represent configuration information necessary for the business logic in the application to handle the responses from the external APIs. That application could be an e-commerce site, which calls various external catalogs for inventory, queries the external data sources, and applies business rules to the responses and the aggregated result. An example of such internal configuration data could be business process rules, or the list of currently active external API endpoints. When the business onboards a new catalog provider, the admin team would update the internal configuration database with the new catalog configuration. During request handling, the web app needs to query the internal configuration database to retrieve the current list of external dependencies such as catalogs, business rule configurations, etc. The configuration information can change during normal operations, so the application developer has to work on a strategy to determine how often to run queries against the underlying config database. In the worst case, the web application must query the configuration database multiple times, for example to determine the different external providers and to retrieve per-provider business rules. Querying the configuration source on a per-request basis has the advantage to always work on the most recent configuration. Unfortunately, that adds considerable load to the configuration database and increases the end-to-end latency of each request. As a result, it reduces the overall capacity of a web application node. The following illustration outlines our starting point: * Step (a): The client sends requests to the load balancer * Step (b): The load balancer distributes incoming requests to the application servers running the web application * Step (c): The web application queries both the configuration source * Step (d): and the external dependencies. The *configuration source* abstractly represents the part of the system that changes the configuration, such as business administrators or automated processes. ![Architecture showing the system using a regular relational database](/files/mQxKdGVdzv7kWMTLf5y8) ### Caching in the ORM Applications typically access a relational database via technology like an object relational mapper (ORM), such as .NET Entity Framework (EF). ORMs can cache query results in memory, to reduce the number of outgoing database queries. The following diagram represents the ORM as a 'local cache' inside the web application: ![Slightly updated architecture showing application-level caching](/files/Fkry29sVUbjlstepSPmi) #### Going to extremes - Caching the whole database Some customers choose a rather aggressive caching strategy: When their application servers start, they 'pre-warm' the node by **completely** querying all tables in the configuration database and pull the complete data set into the application's working memory. This approach removes the need to query the configuration database during the request/response lifecycle, with the web app caching all configuration information locally. Unfortunately, it also brings significant downsides: * **Application start times:** It takes too long to launch the application, due to the pre-warming process, which has to download the full database. During that phase, the application isn't ready to respond to incoming requests. This startup delay can become a problem in scale-out situations: imagine an unforeseen spike in the number of incoming requests, for example due to a TV commercial. When a scaling logic increases the number of web server instances, it would take quite some time until these new nodes can help handling incoming traffic; the spike might already have created problems overloading the other nodes. * **Database load:** Consider how the load pattern affects the database: During such a scale-out event, a potentially large number of servers intensely queries the database at the same time, potentially overloading the database. A '[thundering herd](https://en.wikipedia.org/wiki/Thundering_herd_problem)' of customers in the web tier leads to a scale-out action and results in a thundering herd of web servers bringing down the database. * **Configuration updates:** Runtime updates to the configuration database aren't reaching the web application any longer: When the application caches everything once in RAM on startup, and the ORM no longer queries the database, it doesn't pull updated data from the configuration database into the application. In the past, we have seen customers who 'solved' this problem by rebooting all web servers, one after the other, to force each rebooting web server to fetch the entire database *again*, this time with the updated configuration. ## Introducing Event Sourcing into the architecture Ideally, the web application should keep the configuration information in the working memory. However, we have to avoid the aforementioned disadvantages: The application should ... 1. ... start quickly (load the configuration data into RAM), 2. ... do so without bringing down the configuration system, and 3. ... configuration changes should be visible as fast as possible (without having to restart the application). The following approach will help with the second and third requirement (we handle quick startup in the next step): ### Event sourcing The original definition of Event Sourcing is to > *capture all changes to an application state as a sequence of events.* ([by Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html)) In this article, we're using the term 'event sourcing' liberally: Simply speaking, all configuration change events in the system change a certain part of the overall state. To understand event sourcing better, let's consider the analogy of a bank account: When a customer opens a new account (which is the first event), the account has a zero balance. When the customer receives money (a second event), the system increments the account balance by the given amount. When the customer wires money to a friend, this third event results in having a lower account balance again. The different events (account creation, credit and debit transactions, account closure) represent the banking-specific (domain) events, and the bank records these deltas in the bank's ledger. The account's current balance represents the 'state' of the account. A system replaying (sourcing) all the events from the beginning, in the order in which they appeared, always comes to the same result. The bank's ledger is an 'append-only log'. ### An "append-only log" data structure and service As first step in refactoring the architecture, we 'replace' the configuration database with an 'append-only log' data structure. > The term *'log'* does not refer to log file entries (like an HTTP request log), but should be interpreted like the "captain's log" on a ship, in which all important events are written down sequentially, and historic records (the past) is not modified. 'Append-only' means that newly arriving events don't modify previously written events. Instead, the system appends new events at the end of the log structure. Given that all the web application servers must read the append-only log, some service must expose that structure over the network. Various services implement this pattern, including Apache Kafka, RabbitMQ Streams, or Azure Event Hubs. > Such an event stream could loosely be compared to a database's transaction log, in which all state changes to the various database tables are recorded sequentially as well. Replaying the transaction log allows the reconstruction of the database state, like in event sourcing. ### Event Hubs: partitions, sequence numbers and offsets An Event Hubs instance internally has one or more '[partitions](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability#partitions)'. A partition is a unit of compute that hosts a single append-only commit log. When a sender publishes a message to the Event Hubs endpoint, they can specify a 'partition key'. The partition key serves as input to a hashing function that maps the message to one of the partitions. All messages with the same partition key end up in the same partition. Event Hubs strictly orders messages when a partition receives them: Each message in a partition gets assigned a unique **sequence number**, a strictly monotonic increasing integer that uniquely identifies the message. For example, the first message ever in a given partition would have sequence number #0, and the next message would have sequence number #1, and so on. ### Pulling events into the application The illustration demonstrates the concept: * Step (c): The configuration source emits state update events into event hub * Step (d): All running web app nodes receive (pull) their individual copy of these changes We augment the application with an active component, which that pulls a copy of the stream from the append-only log structure (Azure Event Hubs service). The component then locally applies these events / updates / deltas to the local copy of the configuration state. ![Architecture introducing event sourcing](/files/HxGUIKqwkXjZLuPnFiGV) The application's active component tracks the events' sequence numbers, enforces correct processing order and ensures exactly once processing. The following illustration demonstrates the foundational principle: * The system starts with an 'empty' (null) state, represented by the document with the `⌀` symbol. * We have a function `f()`, which takes in the previous state (`s`) and an event (`e'`), and generates the next version of the state (`s'`). So the equation is `s' := f(s, e')`. For example, `s_500 = f(s_499, e_500)`, that is, the event #500 would transform state #499 into state #500. * The logic applies the first event in the partition (`#0`) to the empty `⌀` state, to generate `state #0`, and so forth. Each new event creates a corresponding newer version of the state. ![Algorithm illustration for event sourcing from an empty state](/files/3TqHzIskIicgG285Zkym) ### Continuously following new messages Append-only log structures usually offer a mechanism to continuously notify consumers about new messages arriving at the end of the log. In a Unix-based system, you can use `tail -f /var/log/messages` to 'follow the tail' of the log file, that is, to continuously see when some producer appends new lines to the log file. In Azure Event Hubs, the consumers remain connected to the Event Hubs via a protocol such as 'AMQP' or 'AMQP over Web Sockets', so that they receive newly arriving messages with minimal latency. ## Resuming operations The state of the system depends on all events, ever received in the past. To compute the most recent state, we have to start with an empty state store, and then replay all events from the beginning of time until today. Replaying all events is an unacceptable approach for practical purposes: when a new node in the web app joins the cluster, it would have to reprocess configuration changes from months ago, just to have an up-to-date understanding of the latest configuration. In practice, there's a simple optimization: create state snapshots on a regular cadence. A snapshot is a (versioned) copy of the serialized state. It might say > This file contains 'state #314', and describes how the state looked like after applying all events #0--#314. To bring snapshot generation into the architecture, we introduce the "snapshot generation" component: ![Architecture adding snapshot generation](/files/zEvM5ZIyrkifBcZhiyeh) That component regularly computes the most recent snapshot, and serializes the state into a file in object storage, such as Azure Blob storage. When a new (uninitialized) web app node starts (or the snapshot generator itself), it first reads the most-recent state snapshot from snapshot storage. That state file carries metadata about which *sequence number* the snapshot corresponds to. After de-serializing the state, the service positions its 'read pointer' at the right sequence number in the event hub partition. The following diagram describes that process. After the component reads state #314, it starts reading the events #315 onwards, and applies them as well, to continuously compute to the most up-to-date representation of the state. ![Algorithm illustration for event sourcing resuming from an existing state](/files/ImKopY1jbPuPoZsQsSci) ### Implementation internals Think of the internal component running in the web app like this: The function `f()` initially receives the most recent state snapshot, and continuously feeds back the most recent state into itself, alongside with the events coming off the event hub. The most recent state is available within the memory of the application, and the application can access it via a global read-only property or a function call. Applying a state update event to the state might touch upon multiple areas of the state data structure. For example, a single update might change two or three data locations within the state type. In the database world that would correspond to a transaction that updates multiple tables or rows. It's important that the web application 'sees' a consistent view of the world. For example, the state must correspond to event #314, or to event #315, but nothing in-between, such as a partially applied event #315. > Using a functional programming language with immutable (unmodifiable) data structures can be of great help here. Examples of such languages could be F#, Scala, Rust or Elixir. While the term "immutable data structure" might sound wasteful or not very useful, it refers to the programming language's ability to represent state transitions. One can say "Give me a copy of this immutable object, with that property here having a different value". Such property modifications, alongside with the fact that large parts of the state might not change, allow to reuse large parts of the object graph. ![The internals of the data pump](/files/shaaxoTsWtKMUM1iS7nt) ### Historic events and Event Hubs Capture The number of events going into an append-only log data structure can potentially be huge. Therefore, the service operator needs to consider how to deal with this data growth. For example, in an Apache Kafka cluster, one might choose to 'just let it grow' and attach some more hard drives, keeping all the information around. Another approach to handle this challenge is 'log compaction', in which older messages get deleted from a partition. Azure Event Hubs addresses this challenge by setting a 'message retention' or '[event retention](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-retention)' period. Event Hubs removes messages that are older than this period from the event hub partition. A reader can no longer 'seek' to them and read them off the partition. The shortest duration is 24 hours, that is, all messages that arrived within that time period are readable from the partition. Older messages (like last week's ones) aren't available here. The maximum retention period varies between 7 days (for Event Hubs Standard) and 90 days (for the Premium and Dedicated SKUs). In our event sourcing system, it might be necessary to read messages that are older than the retention period. For example, if the snapshot generator didn't run for a longer period of time, the stored snapshots might be too old to resume without missing some events. To help with that problem, you can enable **Event Hubs Capture** to fully keep track of the entire history. Enabling the Event Hubs Capture feature forces the Event Hubs service to regularly write older messages as [Apache Avro](https://avro.apache.org/) or [Apache Parquet](https://parquet.apache.org/)-formatted files into Azure Blob storage. The following diagram illustrates the practical use: ![Architecture showing how to resume event source stream from Event Hubs Capture](/files/O38vYaIyKlhIbL7EGGBI) In this situation, the most recent state snapshot corresponds to event #409, so that the system must process event #410 and following next. Unfortunately, the oldest event in the event hub partition is event #412, so that events #410 and #411 aren't available from the partition directly. However, the storage container configured for Event Hubs Capture contains Avro (or Parquet) files containing these events. After the logic feeds the two rather old events into the pipeline, it can 'flip over' to the Event Hubs endpoint for the more recent events. ## When to use this pattern The pattern in this document can be helpful in the following situations: * You have an application that needs low-latency access to an internal data set. * You want to keep some information local in the application and reduce the number of database queries. * The data set is small enough to fit into working memory of your application (without degrading the regular functionality of the application). * The application needs near-realtime access to changes in the data set. * Ideally, you can model the data set using a (functional) programming language that offers immutable data types. The advantage of immutable data types is that different threads or tasks in the application have a consistent view into different versions of the data set. ## A concrete sample implementation A sample solution that implements this pattern is in my GitHub repository [chgeuer/distributed-search](https://github.com/chgeuer/distributed-search). This .NET-based sample in F# implements a distributed search system for an e-commerce site. Customers search for fashion, and the site fans out the search query across multiple third-party search APIs. The system aggregates the external responses into a combined response and applies local business rules. ### The local state The system's configuration data ([`FashionBusinessData`](https://github.com/chgeuer/distributed-search/blob/23c2f20c366505b085ecdd3a26f7105e4dd64848/customer/Customer.Domain.Fashion/FashionBusinessData.fs#L8-L11)) represents the markup by type of the fashion item: ```fsharp type FashionType = string let TShirt: FashionType = "T-Shirt" let Pullover: FashionType = "Pullover" let Throusers: FashionType = "Throusers" type FashionBusinessData = { Markup: Map Brands: Map DefaultMarkup: decimal } ``` For example, the e-commerce site might add a 2% markup on T-shirts, and 4.3% on trousers. ### The update events The F# type [`FashionBusinessDataUpdate`](https://github.com/chgeuer/distributed-search/blob/23c2f20c366505b085ecdd3a26f7105e4dd64848/customer/Customer.Domain.Fashion/FashionBusinessData.fs#L18-L21) is a discriminated union that represents all possible configuration updates coming through Event Hubs: ```fsharp type FashionBusinessDataUpdate = | MarkupUpdate of FashionType: FashionType * MarkupPrice: decimal | BrandUpdate of BrandAcronym: string * Name: string | SetDefaultMarkup of DefaultMarkupPrice: decimal ``` For example, a `MarkupUpdate` event would set a new markup value for a certain `FashionType`, such as increasing the markup for T-shirts to 3%. ### The update function `f()` The [`update`](https://github.com/chgeuer/distributed-search/blob/510724cf02da83e230ba863edf37d3b7523a9a2c/customer/Customer.Domain.Fashion/FashionBusinessData.fs#L25-L40) function implements the process to transform the previous state into the new state, given an update event: ```fsharp let update (businessData: FashionBusinessData) (update: FashionBusinessDataUpdate) : FashionBusinessData = match update with | MarkupUpdate(fashionType, markupPrice) -> match markupPrice with | price when price <= 0m -> { businessData with Markup = businessData.Markup.Remove(fashionType) } | price -> { businessData with Markup = businessData.Markup.Add(fashionType, price) } | BrandUpdate(key, value) -> { businessData with Brands = businessData.Brands.Add(key, value) } | SetDefaultMarkup newDefaultPrice -> { businessData with DefaultMarkup = newDefaultPrice } ``` For example, a `MarkupUpdate` event with a negative price transforms the state so that the `Markup` dictionary no longer contains the entry. This model shows how a sequence of multiple updates can transform the state: ```fsharp let ApplyFashionUpdates (businessData: FashionBusinessData) (updates: IEnumerable) : FashionBusinessData = updates |> Seq.fold ApplyFashionUpdate businessData ``` ### Putting it all together The [`BusinessDataPump`](https://github.com/chgeuer/distributed-search/blob/510724cf02da83e230ba863edf37d3b7523a9a2c/framework/BusinessDataPump/BusinessDataPump.cs) finally brings all moving pieces together: The application starts a `BusinessDataPump` instance, which internally runs a .NET task to pull events from an Event Hubs partition, and exposes the events as an observable sequence (using Reactive Extensions / Rx.NET). The public API to the application is just a [simple property](https://github.com/chgeuer/distributed-search/blob/510724cf02da83e230ba863edf37d3b7523a9a2c/framework/BusinessDataPump/BusinessDataPump.cs#L44): ```csharp public class BusinessDataPump { ... public BusinessData BusinessData { get; private set; } ... } ``` The generic type `TBusinessData` in our example is an instance of the immutable `FashionBusinessData` type. This immutability gives each thread in the running application the assurance that the object is a consistent view into the state. ## Next steps To learn more about event sourcing, you might consider exploring a related few areas: * Domain Driven Design (DDD) * Event Driven Architecture (EDA) * Command and Query Responsibility Segregation (CQRS) ## Contributors *Microsoft maintains this article. The following contributors wrote it:* Principal author: * [Dr. Christian Geuer-Pollmann](https://www.linkedin.com/in/chgeuer/) ([@chgeuer](https://github.com/chgeuer)) | Principal Customer Engineer ## Related resources * Azure Architecture Center * [Partitioning in Event Hubs and Kafka](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/reference-architectures/event-hubs/partitioning-in-event-hubs-and-kafka.yml) * The [Event Sourcing pattern](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/patterns/event-sourcing.yml) * The [CQRS pattern](https://github.com/chgeuer/gitbook-chgeuertips/blob/master/patterns/cqrs.yml) * Free e-book download: [Exploring CQRS and Event Sourcing- A journey into high scalability, availability, and maintainability with Windows Azure (PDF)](https://www.microsoft.com/en-us/download/details.aspx?id=34774) * [Versioning in an Event Sourced System, Gregory Young](https://leanpub.com/esversioning/read) * Video: [Event Sourcing, Gregory Young, GOTO 2014 Conference (YouTube)](https://www.youtube.com/watch?v=8JKjvY4etTY) * [Event Sourcing, Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html) # Postgrex on Azure - Connecting to Azure PostgreSQL from Elixir > Written 2024-NOV-14 by Christian Geuer-Pollmann NOTE: If you want to open this whole article as a Livebook, head over to [`https://gist.github.com/chgeuer/387537c47e48c4c084ac9c6dfba41bba`](https://gist.github.com/chgeuer/387537c47e48c4c084ac9c6dfba41bba) A few weeks ago, I read an [article](https://techcommunity.microsoft.com/blog/fasttrackforazureblog/connecting-to-azure-sql-database-using-sqlalchemy-and-microsoft-entra-authentica/4259772) on how to connect to an Azure SQL DB using Python's SQLAlchemy library, featuring Microsoft Entra ID authentication. My personal interest is more on the Elixir/Erlang side of the house, so Elixir (and [Postgrex](https://github.com/elixir-ecto/postgrex)) it is. Right on time were questions in the Elixir Forum where a customer had challenges to get the Elixir-side TLS config right to establish a connection, so that it's the right time for the topic. **tl;dr:** There are 2 technically interesting things to learn from this article: 1. To get [postgrex](https://github.com/elixir-ecto/postgrex) library to talk to the TLS-protected Azure PostgreSQL endpoint, the client's SSL options must ensure everything for validating the X.509 certificate chain is available, and you communicate the proper database name via SNI. The [Microsoft documentation](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-networking-ssl-tls#download-root-ca-certificates-and-update-application-clients-in-certificate-pinning-scenarios) points to the proper root CA certificates, which allows us to do certificate pinning. 2. The other question is how to get the Microsoft Entra token properly supplied, when establishing the connection to the database. Here it matters to have a token with the right scope/audience, *and* to have the right username. While I expected the username to be the client\_id/application\_id or something, it's actually the friendly name of the application in Entra. > For the sake of this article, I include screenshots and IDs of stuff, so we can fully follow along, but I'll delete the actual resources and identities, so no reason to try whether the creds still work 😐. ## How to authenticate to PostgreSQL? In this article, I demonstrate how you can authenticate to Azure PostgreSQL using two different mechanisms, via Microsoft Entra authentication, and via traditional PostgreSQL authentication. * **Entra ID:** Signing in via Entra means you setup a new application in Entra to represent your web application and create a password for that application. When you want to establish a database connection, you first must fetch an access token from Entra to talk to the database. * **PostgreSQL authentication:** In this case, you access the database using a username/password combination you chose during database creation. > Microsoft strongly recommends to go with the Entra ID option. Entra ID allows you to use a single credential to access many different services, like Azure PostgreSQL, storage, KeyVault, Event Hub, and many more. Otherwise you would have to manage (store/protect/rotate) service-specific credentials for each of these services within your application. Having said that, if for whatever reason you say "Thanks mate, no Entra ID for me right now", then feel free to skip all the Entra-related bits in this article (but don't complain later 🙄). ## Azure setup ### Create the Entra application registration Assuming you want to follow along with this article, first lets setup the Azure side. In [Microsoft Entra's "App registrations" blade](https://portal.azure.com/#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/RegisteredApps), create yourself a new application, and in the app's "Certificates & secrets" tab create a "Client secret". ![Setting up the Entra application](/files/MPHpUrgAe20p0Oldoxue) In the screenshot above, you can see I created an application called `postgrex`, which will be our username when we attempt to authenticate with a JWT access token issued by Entra. Fire off vim or Notepad.exe and make yourself a little cheat sheet, like this (obviously whatever values apply to your environment): ``` TENANT_ID=81c45207-40a0-4d7d-a8f3-feeca7c918fd CLIENT_NAME=postgrex CLIENT_ID=dc604136-1d05-4be5-b6c1-5e251c8fd25d CLIENT_SECRET=... ``` ### Create your Azure PostgreSQL DB Next, get yourself an "Azure Database for PostgreSQL". I created a tiny B1ms version for testing purposes. In authentication options, I chose to use both PostgreSQL authentication *and* Microsoft Entra authentication, so I could try both together. During the setup, you specify the admin username and password (for vanilla PostgreSQL authentication), but you also add your previously created app registration to the list of authorized admins: ![Azure PostgreSQL configuration](/files/SvCdn8dX0Jaq1l2Lp7xO) Enrich your cheat sheet with the rest of the data (name of your instance, and admin user / pass): ``` AZURE_POSTGRES_HOST=elixirdemo1 AZURE_POSTGRES_ADMIN_USER=adminpostgrex AZURE_POSTGRES_ADMIN_PASS=SuperSecret123.- ``` Now please store your key/value file with the config values on your computer, we'll be using it in the next step. > **Firewall rules...** *For demo/testing purposes only*, I also opened up the PostgreSQL endpoint to the whole Internet (Public access), and opened my instance to the IP range `0.0.0.0 -- 255.255.255.255`. Doing this would be a really BAD idea for any production system, so please don't shoot yourselves in the foot here. ![Azure PostgreSQL networking configuration](/files/TpRMi4pHlDF5IDQXHmTs) ## Client environment For our little demo, you might use the awesome Livebook application, an interactive notebook environment for Elixir and Erlang, that allows you to run code step-by-step. Head over to [livebook.dev](https://livebook.dev/) and get the installer for your platform. ### Installing dependencies In Livebook, I install the current dependencies for [`postgrex`](https://hexdocs.pm/postgrex/readme.html), [`req`](https://hexdocs.pm/req/readme.html) and [`livebook_env`](https://hexdocs.pm/livebook_env/readme.html). * `:x509` to simplify certificate handling. (in a previous version of this article, I did it all by hand 🙄) * `:postgrex` is the Elixir-side library to connect to the database. * `:req` is a nice HTTP client. I could've used Erlang's native `:httpc` client, but Req is cool. * `:livebook_env` is a small helper that reads our 'cheat sheet' with config values, and injects them as environment variables. So your "Notebook dependencies and setup" section should look like this: ```elixir Mix.install( [ {:x509, "~> 0.8.10"}, {:postgrex, "~> 0.19.3"}, {:req, "~> 0.5.7"}, {:livebook_env, "~> 1.0"} ] ) ``` And everything compiles nicely: ![Elixir libraries](/files/F1rct5G8bwRrRvUfq9Zf) In the next cell, we load our configuration from the file into the environment. Running that code should report that our 7 values (`TENANT_ID`, `CLIENT_ID`, `CLIENT_SECRET`, `CLIENT_NAME`, `AZURE_POSTGRES_HOST`, `AZURE_POSTGRES_ADMIN_USER`, `AZURE_POSTGRES_ADMIN_PASS`) are read into the environment: ```elixir LivebookEnv.import_dotenv("/home/joe/Desktop/postgrex_cheat_sheet.txt") ``` ![LivebookEnv.import\_dotenv("C:/Users/chgeuer/Desktop/postgrex\_cheat\_sheet.txt")](/files/VvexNpkdjenmjvLuAI2c) Next, we create an Elixir module that helps us with the X.509 certificate pinning and TLS configuration. Copy/paste this into a new cell: ```elixir defmodule MicrosoftCerts do defmodule CompileHelpers do defp http_get(url) do %Req.Response{status: 200, body: body} = Req.get!(url: url) body end def download_certs_for_pinning() do # https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-networking-ssl-tls#download-root-ca-certificates-and-update-application-clients-in-certificate-pinning-scenarios """ https://www.microsoft.com/pkiops/certs/microsoft%20azure%20rsa%20tls%20issuing%20ca%2004%20-%20xsign.crt https://www.microsoft.com/pkiops/certs/Microsoft%20RSA%20Root%20Certificate%20Authority%202017.crt https://cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem https://dl.cacerts.digicert.com/DigiCertGlobalRootG2.crt.pem https://cacerts.digicert.com/DigiCertGlobalRootCA.crt """ |> String.split(["\n"], trim: true) |> Enum.map(fn url -> {url, http_get(url)} end) |> Enum.map(fn {url, data} -> cond do url |> String.ends_with?(".crt") -> data url |> String.ends_with?(".pem") -> data |> X509.Certificate.from_pem!() |> X509.Certificate.to_der() end end) |> Enum.uniq() end end @certs MicrosoftCerts.CompileHelpers.download_certs_for_pinning() def ssl_opts(hostname) do [ protocol: :tls, protocol_version: :"tlsv1.3", verify: :verify_peer, cacerts: @certs, server_name_indication: String.to_charlist(hostname), depth: 3 ] end end MicrosoftCerts.CompileHelpers.download_certs_for_pinning() ``` When you now run a new cell with the command `MicrosoftCerts.CompileHelpers.download_certs_for_pinning()`, you should see a list with 4 binaries (which are 4 root CA certificates). ![Downloading the certificates](/files/JQH8VQnjqHinhInHrUp2) This module essentially downloads root CA certificates from Microsoft and DigiCert (I took the URLs from our [documentation](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-networking-ssl-tls#download-root-ca-certificates-and-update-application-clients-in-certificate-pinning-scenarios). It then transforms the `.crt` files to base64-encoded PEM syntax, uses Erlang's `:public_key.pem.decode/1` function to parse everything, extracts the certificates, and at compile time puts it into the `@certs` module attribute. This simply means these X.509 certs are only downloaded once when the application is compiled and embedded in the Erlang byte code. The only thing we will need later is the `MicrosoftCerts.ssl_opts/1` function, that creates the `:ssl` config we need for postgrex. ### Fetch a token the 'hard way' Rather than bringing in other dependencies, let's just grab an access token from Microsoft Entra ID, by using the Req HTTP client and POSTing a token issuance request to Entra. As you see, we're using the environment variables for Entra tenant ID, our app's client\_id and client\_secrets. The scope `"https://ossrdbms-aad.database.windows.net/.default"` is the one you need to talk to Azure PostgreSQL and MySQL... ```elixir {:ok, %Req.Response{status: 200, body: %{"access_token" => access_token}}} = Req.request( method: :post, url: "https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token", path_params_style: :curly, path_params: [ tenant_id: System.get_env("TENANT_ID") ], form: [ grant_type: "client_credentials", client_id: System.get_env("CLIENT_ID"), client_secret: System.get_env("CLIENT_SECRET"), scope: "https://ossrdbms-aad.database.windows.net/.default" ] ) IO.puts("https://jwt.ms/#access_token=#{access_token}") ``` This last `IO.puts(...)` statement prints out a link which you can open in your web browser, to inspect the access\_token... > The access token is just a fragment path, i.e. doesn't get transferred to an external server. [jwt.ms](https://jwt.ms/) is a static web site by Microsoft, which parses JWT tokens using JavaScript exclusively in the user's browser. ![Fetching the access token](/files/0pmAqYl91gyCGaV6d06v) With a valid access token in our hands, we're finally able to connect to the database: ```elixir db_host = System.get_env("AZURE_POSTGRES_HOST") <> ".postgres.database.azure.com" {:ok, conn} = Postgrex.start_link( hostname: db_host, port: 5432, database: "postgres", ssl: MicrosoftCerts.ssl_opts(db_host), # The username here is the friendly name of our app... username: System.get_env("CLIENT_NAME"), password: access_token ) ``` Assuming the `Postgrex.start_link/1` call doesn't fail, the `conn` variable now contains a process ID, representing the connection to the database: ![Connecting to the database](/files/IRL2MxtXFUgpanHnlCes) So the last step here is to run a query against the database, like this one: ```elixir create_table_query = """ CREATE TABLE IF NOT EXISTS users ( id SERIAL PRIMARY KEY, username VARCHAR(50) NOT NULL UNIQUE, email VARCHAR(255) NOT NULL UNIQUE, password_hash VARCHAR(255) NOT NULL, created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ); """ # Execute the create table query Postgrex.query!(conn, create_table_query, []) ``` ![Running a SQL statement](/files/OCHRyf21jQJLLtFpMJCm) ### Using PostgreSQL authentication Last but not least, of course you can also just use plain vanilla PostgreSQL authN with the username/password approach. But the strong recommendation is to use Entra authN (instead of username/passwords), as this is much better manageable. ```elixir {:ok, conn2} = Postgrex.start_link( hostname: db_host, port: 5432, database: "postgres", ssl: MicrosoftCerts.ssl_opts(db_host), username: System.get_env("AZURE_POSTGRES_ADMIN_USER"), password: System.get_env("AZURE_POSTGRES_ADMIN_PASS") ) ``` # Excel ![](/files/WAWhWOJH0jkjl2IprAXy) The only Excel tutorial you need in your life: {% embed url="" %} ![Using the INDEX() and the MATCH() functions](/files/rk0hQjO7MxXpT5PxTirJ) ``` =INDEX(Bonuses[Bonus]; MATCH([@Type]; Bonuses[Type]; 0)) ``` # Desktop Setup ## Windows 11 How-To: Disable new context menu, Explorer command bar ### Disable new context menu: ``` reg.exe add "HKCU\Software\Classes\CLSID\{86ca1aa0-34aa-4e8b-a509-50c905bae2a2}\InprocServer32" /f /ve ``` ### Restore new context menu: ``` reg.exe delete "HKCU\Software\Classes\CLSID\{86ca1aa0-34aa-4e8b-a509-50c905bae2a2}" /f ``` ### Disable Explorer command bar ``` reg.exe add "HKCU\Software\Classes\CLSID\{d93ed569-3b3e-4bff-8355-3c44f6a52bb5}\InprocServer32" /f /ve ``` ### Restore Explorer command bar ``` reg.exe delete "HKCU\Software\Classes\CLSID\{d93ed569-3b3e-4bff-8355-3c44f6a52bb5}" /f ``` Source ### Show Taskbar on the left ```shell reg.exe add HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v TaskbarAl /t REG_DWORD /d 0 /f ``` ### Show Taskbar in the center ```shell reg.exe add HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced /v TaskbarAl /t REG_DWORD /d 1 /f ``` ## Powershell * [Powerline Setup](https://www.hanselman.com/blog/HowToMakeAPrettyPromptInWindowsTerminalWithPowerlineNerdFontsCascadiaCodeWSLAndOhmyposh.aspx) ## Install Powerline ```powershell Install-Module posh-git -Scope CurrentUser Install-Module oh-my-posh -Scope CurrentUser ``` ### `code $PROFILE` ```powershell Set-PSReadlineOption -EditMode Emacs Set-PSReadlineKeyHandler -Chord Ctrl+D -Function DeleteCharOrExit Import-Module posh-git Import-Module oh-my-posh Set-Theme Sorin #Set-Theme Paradox ``` ## Chrome Setup * [Extension to always access `docs.microsoft.com` in `en-us` English](https://chrome.google.com/webstore/detail/english-docsmicrosoftcom/ggkanifnckjfjdmeclcakoboheakicgk) ## Enable "long path on Windows" ``` Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem] "LongPathsEnabled"=dword:00000001 ``` ## Docker ### Make Windows Docker visible on WSL * Have a machine environment variable `DOCKER_HOST=tcp://localhost:2375` * Propagate that value down to WSL using `WSLENV=DOCKER_HOST/u:...`, which makes Windows environment variables visible in WSL. * Check ### Enable drive sharing Seems like Azure AD accounts on a Windows 10 box cannot simply enable Volume sharing with Docker CE for Windows. Therefore, do the following: * In computer management, create a local user "dockerhost" with a strong password, which never expires. * Add that dockerhost user to the "Administrators" and "docker-users" group * Turn on drive sharing in the Docker for Windows settings, and instead of "EUROPE\foobar", use ".\dockerhost" as user. * Grant "dockerhost" user full access on `~/.nuget` and `~/vsdbg`. ## Path ``` %SystemRoot%\system32 %SystemRoot% %SystemRoot%\System32\Wbem %SystemRoot%\System32\WindowsPowerShell\v1.0 %SystemRoot%\System32\OpenSSH %ProgramFiles%\SysinternalsSuite %ProgramFiles(x86)%\Vim\vim82 %ProgramFiles%\Git\cmd %ProgramFiles%\TortoiseGit\bin %ProgramFiles%\Python38\ %ProgramFiles%\Python38\Scripts\ %ProgramFiles%\nodejs %ProgramFiles%\erl10.6\bin %ProgramFiles(x86)%\Elixir\bin %ProgramFiles%\Go\bin %ProgramFiles%\dotnet %ProgramFiles%\PowerShell\6 %ProgramFiles(x86)%\Microsoft SDKs\Azure\CLI2\wbin %ProgramFiles%\Microsoft SQL Server\130\Tools\Binn %ProgramFiles%\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn %ProgramFiles(x86)%\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\FSharp %ProgramFiles(x86)%\Graphviz2.38\bin %ProgramFiles%\dotnet %ProgramFiles%\Docker\Docker\resources\bin %ProgramData%\DockerDesktop\version-bin %UserProfile%\AppData\Local\Microsoft\WindowsApps %UserProfile%\AppData\Local\Programs\Microsoft VS Code %UserProfile%\.cargo\bin %UserProfile%\.mix\escripts %UserProfile%\.dotnet\tools %UserProfile%\bin %UserProfile%\bin\SysinternalsSuite ``` ## Disable Microsoft Compatibility Telemetry ```batch sc delete DiagTrack sc delete dmwappushservice sc config dmwappushservice start= disabled echo "" > %ProgramData%\Microsoft\Diagnosis\ETLLogs\AutoLogger\AutoLogger-DiagTrack-Listener.etl reg add HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection /v AllowTelemtery /t REG_DWORD /d 0 /f ``` ## Disable web search from Win10 ```batch reg add HKCU\Software\Microsoft\Windows\CurrentVersion\Search /v CortanaConsent /t REG_DWORD /d 0 /f reg add HKCU\Software\Microsoft\Windows\CurrentVersion\Search /v BingSearchEnabled /t REG_DWORD /d 0 /f ``` ## vim setup ```bash export VISUAL=vim export EDITOR="${VISUAL}" export GIT_EDITOR="${VISUAL}" git config --global core.editor "vim" ``` ### `~/.vimrc` ``` function! StatusLine(current) return (a:current ? crystalline#mode() . '%#Crystalline#' : '%#CystallineInactive#') . ' %f%h%w%m%r ' endfunction " https://jovicailic.org/2017/04/vim-persistent-undo/ set undofile set undodir=~/.vim/undodir let g:crystalline_statusline_fn='StatusLine' let g:crystalline_theme='default' set laststatus=2 set nocompatible set path+=** set wildmenu filetype off set rtp+=$HOME/.vim/bundle/Vundle.vim/ call vundle#begin('$HOME/.vim/bundle/') Plugin 'VundleVim/Vundle.vim' Plugin 'slashmili/alchemist.vim' Plugin 'elixir-editors/vim-elixir' Plugin 'jalvesaq/vimcmdline' Plugin 'fatih/vim-go' Plugin 'morhetz/gruvbox' Plugin 'arcticicestudio/nord-vim' Plugin 'flrnprz/plastic.vim' Plugin 'rbong/vim-crystalline' Plugin 'segeljakt/vim-silicon' Plugin 'preservim/nerdtree' " Plugin 'alok/notational-fzf-vim' Plugin 'vim/killersheep' call vundle#end() filetype plugin indent on set langmenu=en_US.UTF-8 language messages en_US.UTF-8 if has("gui_win32") " set guifont=Fira_Code:h20 set guifont=Cascadia_Code:h20 " set guifont=CascadiaCode_Nerd_Font:h20 " set guifont=Pragmata_Pro_Mono:h20 " set renderoptions=type:directx set encoding=utf-8 endif " highlight Normal guifg=#d7d7d7 guibg=#1e1e1e " colorscheme darkblue " colorscheme gruvbox colorscheme nord set encoding=utf-8 set listchars=eol:�,tab:>-,trail:�,extends:>,precedes:<,nbsp:� highlight SpecialKey term=standout ctermbg=yellow guibg=yellow set list noremap :set list! inoremap :set list! cnoremap :set list! " http://vim.wikia.com/wiki/Automatic_word_wrapping set wrap linebreak nolist " source $VIMRUNTIME/mswin.vim " behave mswin if has("win32") execute pathogen#infect() " syntax on syntax enable filetype plugin indent on endif syntax enable set number set background=dark set mouse=a if &term =~ '256color' set t_ut= endif let mapleader="\" "inoremap ( () "inoremap { {} "inoremap [ [] "inoremap " "" " copy selected text to clipboard set guioptions+=a " nnoremap y "+Y " nnoremap p "+gP " nmap gt nmap :tabnext nmap :tabprev inoremap :tabnexta inoremap :tabpreva " let g:nv_search_paths = ['C:\Users\chgeuer\Documents\SAP'] " highlight ColorColumn ctermbg=red ctermfg=blue " exec 'set colorcolumn=' . join(range(2, 80, 3), ',') ``` ## Useful `bash` aliases ```bash # # `cdr` brings you to the root of your git repo (kudos to https://twitter.com/thorstenball/status/1223218245592911878) # alias cdr='cd $(git rev-parse --show-toplevel)' ``` ## Windows Terminal Settings `%USERPROFILE%\AppData\Local\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState\profiles.json` ````json { "$schema": "https://aka.ms/terminal-profiles-schema", "defaultProfile": "{07b52e3e-de2c-5db4-bd2d-ba144ed6c273}", "copyOnSelect": true, "copyFormatting": false, "alwaysShowTabs": true, "initialCols": 120, "initialRows": 30, "requestedTheme": "dark", "showTabsInTitlebar": true, "showTerminalTitleInTitlebar": true, "wordDelimiters": " ./\\()\"\u0027-:,.;\u003c\u003e~!@#$%^\u0026*|+=[]{}~?�", "profiles": { "defaults": { "fontFace": "Delugia Nerd Font", "colorScheme": "Nord" }, "list": [ { // Make changes here to the powershell.exe profile. "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}", "name": "Windows PowerShell", "commandline": "powershell.exe", "hidden": false }, { // Make changes here to the cmd.exe profile. "guid": "{0caa0dad-35be-5f56-a8ff-afceeeaa6101}", "name": "Command Prompt", "commandline": "cmd.exe", "hidden": false }, { "guid": "{7d04ce37-c00f-43ac-ba47-992cb1393215}", "name": "yori", "commandline": "C:\\Users\\chgeuer\\AppData\\Local\\Yori\\yori.exe", "startingDirectory": "C:\\Users\\chgeuer", "hidden": false }, { "guid": "{b453ae62-4e3d-5e58-b989-0a998ec441b8}", "hidden": true, "name": "Azure Cloud Shell", "source": "Windows.Terminal.Azure" }, { "guid": "{07b52e3e-de2c-5db4-bd2d-ba144ed6c273}", "hidden": false, "name": "Ubuntu-20.04", // "startingDirectory": "//wsl$/Ubuntu-20.04/home/chgeuer", "source": "Windows.Terminal.Wsl" } ] }, "schemes": [ { "name": "Nord", "foreground":"#d8dee9","background":"#2e3440","black":"#3b4252","blue":"#81a1c1","brightBlack":"#4c566a","brightBlue":"#81a1c1","brightCyan":"#8fbcbb","brightGreen":"#a3be8c","brightPurple":"#b48ead","brightRed":"#bf616a","brightWhite":"#eceff4","brightYellow":"#ebcb8b","cyan":"#88c0d0","green":"#a3be8c","purple":"#b48ead","red":"#bf616a","white":"#e5e9f0","yellow":"#ebcb8b" }, { "name": "Solarized Dark", "foreground":"#FDF6E3","background":"#073642","black":"#073642","blue":"#268BD2","brightBlack":"#002B36","brightBlue":"#839496","brightCyan":"#93A1A1","brightGreen":"#586E75","brightPurple":"#6C71C4","brightRed":"#CB4B16","brightWhite":"#FDF6E3","brightYellow":"#657B83","cyan":"#2AA198","green":"#859900","purple":"#D33682","red":"#D30102","white":"#EEE8D5","yellow":"#B58900" }, { "name": "Solarized Light", "foreground":"#073642","background":"#FDF6E3","black":"#073642","blue":"#268BD2","brightBlack":"#002B36","brightBlue":"#839496","brightCyan":"#93A1A1","brightGreen":"#586E75","brightPurple":"#6C71C4","brightRed":"#CB4B16","brightWhite":"#FDF6E3","brightYellow":"#657B83","cyan":"#2AA198","green":"#859900","purple":"#D33682","red":"#D30102","white":"#EEE8D5","yellow":"#B58900" }, { "name": "Ubuntu", "foreground":"#EEEEEC","background":"#2C001E","black":"#EEEEEC","blue":"#268BD2","brightBlack":"#002B36","brightBlue":"#839496","brightCyan":"#93A1A1","brightGreen":"#586E75","brightPurple":"#6C71C4","brightRed":"#CB4B16","brightWhite":"#FDF6E3","brightYellow":"#657B83","cyan":"#2AA198","green":"#729FCF","purple":"#D33682","red":"#16C60C","white":"#EEE8D5","yellow":"#B58900" }, { "name": "Ubuntuz", "foreground":"#EEEEEC","background":"#300A24","black":"#2E3436","blue":"#3465A4","brightBlack":"#555753","brightBlue":"#729FCF","brightCyan":"#34E2E2","brightGreen":"#8AE234","brightPurple":"#AD7FA8","brightRed":"#EF2929","brightWhite":"#EEEEEC","brightYellow":"#FCE94F","cyan":"#06989A","green":"#4E9A06","purple":"#75507B","red":"#CC0000","white":"#D3D7CF","yellow":"#C4A000" }, { "name": "Pandora", "foreground":"#E1E1E1","background":"#141E43","black":"#000000","blue":"#338F86","brightBlack":"#3F5648","brightBlue":"#23D7D7","brightCyan":"#00EDE1","brightGreen":"#74CD68","brightPurple":"#FF37FF","brightRed":"#FF3242","brightWhite":"#FFFFFF","brightYellow":"#FFB929","cyan":"#23D7D7","green":"#74AF68","purple":"#9414E6","red":"#FF4242","white":"#E2E2E2","yellow":"#FFAD29" }, { "name": "Treehouse", "foreground":"#786B53","background":"#191919","black":"#321300","blue":"#58859A","brightBlack":"#433626","brightBlue":"#85CFED","brightCyan":"#F07D14","brightGreen":"#55F238","brightPurple":"#E14C5A","brightRed":"#ED5D20","brightWhite":"#FFC800","brightYellow":"#F2B732","cyan":"#B25A1E","green":"#44A900","purple":"#97363D","red":"#B2270E","white":"#786B53","yellow":"#AA820C" }, { "name": "Treehousez", "foreground":"#786B53","background":"#191919","black":"#321300","blue":"#58859A","brightBlack":"#433626","brightBlue":"#85CFED","brightCyan":"#F07D14","brightGreen":"#55F238","brightPurple":"#E14C5A","brightRed":"#ED5D20","brightWhite":"#FFC800","brightYellow":"#F2B732","cyan":"#B25A1E","green":"#44A900","purple":"#97363D","red":"#B2270E","white":"#786B53","yellow":"#AA820C" }, { "name": "Symfonicz", "foreground":"#FFFFFF","background":"#000000","black":"#000000","blue":"#0084D4","brightBlack":"#1B1D21","brightBlue":"#0084D4","brightCyan":"#CCCCFF","brightGreen":"#56DB3A","brightPurple":"#B729D9","brightRed":"#DC322F","brightWhite":"#FFFFFF","brightYellow":"#FF8400","cyan":"#CCCCFF","green":"#56DB3A","purple":"#B729D9","red":"#DC322F","white":"#FFFFFF","yellow":"#FF8400" }, { "name": "Campbell", "foreground":"#F2F2F2","background":"#0C0C0C","black":"#0C0C0C","blue":"#0037DA","brightBlack":"#767676","brightBlue":"#3B78FF","brightCyan":"#61D6D6","brightGreen":"#16C60C","brightPurple":"#B4009E","brightRed":"#E74856","brightWhite":"#F2F2F2","brightYellow":"#F9F1A5","cyan":"#3A96DD","green":"#13A10E","purple":"#881798","red":"#C50F1F","white":"#CCCCCC","yellow":"#C19C00" }, { "name": "Chalkboardz", "foreground":"#D9E6F2","background":"#29262F","black":"#000000","blue":"#7372C3","brightBlack":"#323232","brightBlue":"#AAAADB","brightCyan":"#AADADB","brightGreen":"#AADBAA","brightPurple":"#DBAADA","brightRed":"#DBAAAA","brightWhite":"#FFFFFF","brightYellow":"#DADBAA","cyan":"#72C2C3","green":"#72C373","purple":"#C372C2","red":"#C37372","white":"#D9D9D9","yellow":"#C2C372" }, { "name": "Chesterz", "foreground":"#FFFFFF","background":"#2C3643","black":"#080200","blue":"#288AD6","brightBlack":"#6F6B68","brightBlue":"#278AD6","brightCyan":"#27DEDE","brightGreen":"#16C98D","brightPurple":"#D34590","brightRed":"#FA5E5B","brightWhite":"#FFFFFF","brightYellow":"#FEEF6D","cyan":"#28DDDE","green":"#16C98D","purple":"#D34590","red":"#FA5E5B","white":"#E7E7E7","yellow":"#FFC83F" }, { "name": "Lovelace", "foreground":"#FDFDFD","background":"#1D1F28","black":"#282A36","blue":"#8897F4","brightBlack":"#414458","brightBlue":"#556FFF","brightCyan":"#3FDCEE","brightGreen":"#18E3C8","brightPurple":"#B043D1","brightRed":"#FF4971","brightWhite":"#BEBEC1","brightYellow":"#FF8037","cyan":"#79E6F3","green":"#5ADECD","purple":"#C574DD","red":"#F37F97","white":"#FDFDFD","yellow":"#F2A272" } ], "keybindings": [ { "command": "paste", "keys": "ctrl+v" }, { "command": "find", "keys": "ctrl+shift+f" }, { "command": {"action": "copy", "singleLine": false }, "keys": "ctrl+c" }, { "command": {"action": "splitPane", "split": "auto", "splitMode": "duplicate" }, "keys": "alt+shift+d" }, { "command": {"action": "splitPane", "split": "vertical", "profile": "{c6eaf9f4-32a7-5fdc-b5cf-066e8a4b1e40}", "tabTitle": "github","startingDirectory": "C:\\github\\chgeuer" }, "keys": ["ctrl+m"] }, { "command": "nextTab", "keys": [ "ctrl+pgdn" ] }, { "command": "prevTab", "keys": [ "ctrl+pgup" ] }, { "command": "newTab", "keys": [ "ctrl+t" ] }, { "command": "closeTab", "keys": [ "ctrl+w" ] }, { "command": "scrollDown", "keys": [ "ctrl+shift+down" ] }, { "command": "scrollDownPage", "keys": [ "ctrl+shift+pgdn" ] }, { "command": "scrollUp", "keys": [ "ctrl+shift+up" ] }, { "command": "scrollUpPage", "keys": [ "ctrl+shift+pgup" ] } ] }``` ### Windows Terminal Color Scheme Consolidation - https://github.com/mbadolato/iTerm2-Color-Schemes/tree/master/windowsterminal - https://terminalsplash.com/ ```json { "schemes": [ {"name": "Nord", "foreground":"#d8dee9","background":"#2e3440","black":"#3b4252","blue":"#81a1c1","brightBlack":"#4c566a","brightBlue":"#81a1c1","brightCyan":"#8fbcbb","brightGreen":"#a3be8c","brightPurple":"#b48ead","brightRed":"#bf616a","brightWhite":"#eceff4","brightYellow":"#ebcb8b","cyan":"#88c0d0","green":"#a3be8c","purple":"#b48ead","red":"#bf616a","white":"#e5e9f0","yellow":"#ebcb8b" }, {"name": "Solarized Dark", "foreground":"#FDF6E3","background":"#073642","black":"#073642","blue":"#268BD2","brightBlack":"#002B36","brightBlue":"#839496","brightCyan":"#93A1A1","brightGreen":"#586E75","brightPurple":"#6C71C4","brightRed":"#CB4B16","brightWhite":"#FDF6E3","brightYellow":"#657B83","cyan":"#2AA198","green":"#859900","purple":"#D33682","red":"#D30102","white":"#EEE8D5","yellow":"#B58900" }, {"name": "Solarized Light", "foreground":"#073642","background":"#FDF6E3","black":"#073642","blue":"#268BD2","brightBlack":"#002B36","brightBlue":"#839496","brightCyan":"#93A1A1","brightGreen":"#586E75","brightPurple":"#6C71C4","brightRed":"#CB4B16","brightWhite":"#FDF6E3","brightYellow":"#657B83","cyan":"#2AA198","green":"#859900","purple":"#D33682","red":"#D30102","white":"#EEE8D5","yellow":"#B58900" }, {"name": "Ubuntu", "foreground":"#EEEEEC","background":"#2C001E","black":"#EEEEEC","blue":"#268BD2","brightBlack":"#002B36","brightBlue":"#839496","brightCyan":"#93A1A1","brightGreen":"#586E75","brightPurple":"#6C71C4","brightRed":"#CB4B16","brightWhite":"#FDF6E3","brightYellow":"#657B83","cyan":"#2AA198","green":"#729FCF","purple":"#D33682","red":"#16C60C","white":"#EEE8D5","yellow":"#B58900" }, {"name": "Ubuntuz", "foreground":"#EEEEEC","background":"#300A24","black":"#2E3436","blue":"#3465A4","brightBlack":"#555753","brightBlue":"#729FCF","brightCyan":"#34E2E2","brightGreen":"#8AE234","brightPurple":"#AD7FA8","brightRed":"#EF2929","brightWhite":"#EEEEEC","brightYellow":"#FCE94F","cyan":"#06989A","green":"#4E9A06","purple":"#75507B","red":"#CC0000","white":"#D3D7CF","yellow":"#C4A000" }, {"name": "Pandora", "foreground":"#E1E1E1","background":"#141E43","black":"#000000","blue":"#338F86","brightBlack":"#3F5648","brightBlue":"#23D7D7","brightCyan":"#00EDE1","brightGreen":"#74CD68","brightPurple":"#FF37FF","brightRed":"#FF3242","brightWhite":"#FFFFFF","brightYellow":"#FFB929","cyan":"#23D7D7","green":"#74AF68","purple":"#9414E6","red":"#FF4242","white":"#E2E2E2","yellow":"#FFAD29" }, {"name": "Treehouse", "foreground":"#786B53","background":"#191919","black":"#321300","blue":"#58859A","brightBlack":"#433626","brightBlue":"#85CFED","brightCyan":"#F07D14","brightGreen":"#55F238","brightPurple":"#E14C5A","brightRed":"#ED5D20","brightWhite":"#FFC800","brightYellow":"#F2B732","cyan":"#B25A1E","green":"#44A900","purple":"#97363D","red":"#B2270E","white":"#786B53","yellow":"#AA820C" }, {"name": "Treehousez", "foreground":"#786B53","background":"#191919","black":"#321300","blue":"#58859A","brightBlack":"#433626","brightBlue":"#85CFED","brightCyan":"#F07D14","brightGreen":"#55F238","brightPurple":"#E14C5A","brightRed":"#ED5D20","brightWhite":"#FFC800","brightYellow":"#F2B732","cyan":"#B25A1E","green":"#44A900","purple":"#97363D","red":"#B2270E","white":"#786B53","yellow":"#AA820C" }, {"name": "Symfonicz", "foreground":"#FFFFFF","background":"#000000","black":"#000000","blue":"#0084D4","brightBlack":"#1B1D21","brightBlue":"#0084D4","brightCyan":"#CCCCFF","brightGreen":"#56DB3A","brightPurple":"#B729D9","brightRed":"#DC322F","brightWhite":"#FFFFFF","brightYellow":"#FF8400","cyan":"#CCCCFF","green":"#56DB3A","purple":"#B729D9","red":"#DC322F","white":"#FFFFFF","yellow":"#FF8400" }, {"name": "Campbell", "foreground":"#F2F2F2","background":"#0C0C0C","black":"#0C0C0C","blue":"#0037DA","brightBlack":"#767676","brightBlue":"#3B78FF","brightCyan":"#61D6D6","brightGreen":"#16C60C","brightPurple":"#B4009E","brightRed":"#E74856","brightWhite":"#F2F2F2","brightYellow":"#F9F1A5","cyan":"#3A96DD","green":"#13A10E","purple":"#881798","red":"#C50F1F","white":"#CCCCCC","yellow":"#C19C00" }, {"name": "Chalkboardz", "foreground":"#D9E6F2","background":"#29262F","black":"#000000","blue":"#7372C3","brightBlack":"#323232","brightBlue":"#AAAADB","brightCyan":"#AADADB","brightGreen":"#AADBAA","brightPurple":"#DBAADA","brightRed":"#DBAAAA","brightWhite":"#FFFFFF","brightYellow":"#DADBAA","cyan":"#72C2C3","green":"#72C373","purple":"#C372C2","red":"#C37372","white":"#D9D9D9","yellow":"#C2C372" }, {"name": "Chesterz", "foreground":"#FFFFFF","background":"#2C3643","black":"#080200","blue":"#288AD6","brightBlack":"#6F6B68","brightBlue":"#278AD6","brightCyan":"#27DEDE","brightGreen":"#16C98D","brightPurple":"#D34590","brightRed":"#FA5E5B","brightWhite":"#FFFFFF","brightYellow":"#FEEF6D","cyan":"#28DDDE","green":"#16C98D","purple":"#D34590","red":"#FA5E5B","white":"#E7E7E7","yellow":"#FFC83F" }, {"name": "Lovelace", "foreground":"#FDFDFD","background":"#1D1F28","black":"#282A36","blue":"#8897F4","brightBlack":"#414458","brightBlue":"#556FFF","brightCyan":"#3FDCEE","brightGreen":"#18E3C8","brightPurple":"#B043D1","brightRed":"#FF4971","brightWhite":"#BEBEC1","brightYellow":"#FF8037","cyan":"#79E6F3","green":"#5ADECD","purple":"#C574DD","red":"#F37F97","white":"#FDFDFD","yellow":"#F2A272" }, {"name":"Dracula", "foreground":"#F8F8F2","background":"#282A36","brightBlack":"#6272A4","brightWhite":"#FFFFFF","brightBlue":"#D6ACFF","brightCyan":"#A4FFFF","brightGreen":"#69FF94","brightPurple":"#FF92DF","brightRed":"#FF6E6E","brightYellow":"#FFFFA5","black":"#21222C","white":"#F8F8F2","blue":"#BD93F9","cyan":"#8BE9FD","green":"#50FA7B","purple":"#FF79C6","red":"#FF5555","yellow":"#F1FA8C"}, {"name":"Night Owl", "foreground":"#D6DEEB","background":"#011627","brightBlack":"#575656","brightWhite":"#FFFFFF","brightBlue":"#82AAFF","brightCyan":"#7FDBCA","brightGreen":"#22DA6E","brightPurple":"#C792EA","brightRed":"#EF5350","brightYellow":"#FFEB95","black":"#011627","white":"#FFFFFF","blue":"#82AAFF","cyan":"#21C7A8","green":"#22DA6E","purple":"#C792EA","red":"#EF5350","yellow":"#ADDB67"}, {"name":"Light Owl", "foreground":"#403F53","background":"#FBFBFB","brightBlack":"#403F53","brightWhite":"#979797","brightBlue":"#288ED7","brightCyan":"#2AA298","brightGreen":"#08916A","brightPurple":"#D6438A","brightRed":"#DE3D3B","brightYellow":"#DAAA01","black":"#403F53","white":"#F0F0F0","blue":"#288ED7","cyan":"#2AA298","green":"#08916A","purple":"#D6438A","red":"#DE3D3B","yellow":"#E0AF02"}, {"name":"Polygone", "foreground":"#ffffff","background":"#101116","brightBlack":"#2b2b2b","brightWhite":"#FFFFFF","brightBlue":"#0095ff","brightCyan":"#397199","brightGreen":"#00ff6a","brightPurple":"#9940ff","brightRed":"#ff4040","brightYellow":"#ffe873","black":"#006aff","white":"#ffffff","blue":"#6272a4","cyan":"#0d4259","green":"#2fff24","purple":"#6a00ff","red":"#ff2424","yellow":"#ffc400"}, {"name":"Cobalt2", "foreground":"#c7c7c7","background":"#193549","brightBlack":"#808080","brightWhite":"#ffffff","brightBlue":"#1478DB","brightCyan":"#00ffff","brightGreen":"#33ff00","brightPurple":"#cc00ff","brightRed":"#ff0000","brightYellow":"#ffff00","black":"#000000","white":"#c7c7c7","blue":"#1478DB","cyan":"#00c5c7","green":"#3AD900","purple":"#ff2c70","red":"#ff2600","yellow":"#ffc600"}, {"name":"SMYCK", "foreground":"#F8F8F8","background":"#1B1B1B","brightBlack":"#5D5D5D","brightWhite":"#F7F7F7","brightBlue":"#9CD9F0","brightCyan":"#77DFD8","brightGreen":"#CDEE69","brightPurple":"#FBB1F9","brightRed":"#E09690","brightYellow":"#FFE377","black":"#000000","white":"#B0B0B0","blue":"#4E90A7","cyan":"#218693","green":"#8EB33B","purple":"#C8A0D1","red":"#C75646","yellow":"#D0B03C"}, {"name":"3024 Day", "foreground":"#4a4543","background":"#f7f7f7","brightBlack":"#5c5855","brightWhite":"#f7f7f7","brightBlue":"#807d7c","brightCyan":"#cdab53","brightGreen":"#3a3432","brightPurple":"#d6d5d4","brightRed":"#e8bbd0","brightYellow":"#4a4543","black":"#090300","white":"#a5a2a2","blue":"#01a0e4","cyan":"#b5e4f4","green":"#01a252","purple":"#a16a94","red":"#db2d20","yellow":"#fded02"}, {"name":"3024 Night", "foreground":"#a5a2a2","background":"#090300","brightBlack":"#5c5855","brightWhite":"#f7f7f7","brightBlue":"#807d7c","brightCyan":"#cdab53","brightGreen":"#3a3432","brightPurple":"#d6d5d4","brightRed":"#e8bbd0","brightYellow":"#4a4543","black":"#090300","white":"#a5a2a2","blue":"#01a0e4","cyan":"#b5e4f4","green":"#01a252","purple":"#a16a94","red":"#db2d20","yellow":"#fded02"}, {"name":"AdventureTime", "foreground":"#f8dcc0","background":"#1f1d45","brightBlack":"#4e7cbf","brightWhite":"#f6f5fb","brightBlue":"#1997c6","brightCyan":"#c8faf4","brightGreen":"#9eff6e","brightPurple":"#9b5953","brightRed":"#fc5f5a","brightYellow":"#efc11a","black":"#050404","white":"#f8dcc0","blue":"#0f4ac6","cyan":"#70a598","green":"#4ab118","purple":"#665993","red":"#bd0013","yellow":"#e7741e"}, {"name":"Afterglow", "foreground":"#d0d0d0","background":"#212121","brightBlack":"#505050","brightWhite":"#f5f5f5","brightBlue":"#6c99bb","brightCyan":"#7dd6cf","brightGreen":"#7e8e50","brightPurple":"#9f4e85","brightRed":"#ac4142","brightYellow":"#e5b567","black":"#151515","white":"#d0d0d0","blue":"#6c99bb","cyan":"#7dd6cf","green":"#7e8e50","purple":"#9f4e85","red":"#ac4142","yellow":"#e5b567"}, {"name":"AlienBlood", "foreground":"#637d75","background":"#0f1610","brightBlack":"#3c4812","brightWhite":"#73fa91","brightBlue":"#00aae0","brightCyan":"#00e0c4","brightGreen":"#18e000","brightPurple":"#0058e0","brightRed":"#e08009","brightYellow":"#bde000","black":"#112616","white":"#647d75","blue":"#2f6a7f","cyan":"#327f77","green":"#2f7e25","purple":"#47587f","red":"#7f2b27","yellow":"#717f24"}, {"name":"Andromeda", "foreground":"#e5e5e5","background":"#262a33","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#2472c8","brightCyan":"#0fa8cd","brightGreen":"#05bc79","brightPurple":"#bc3fbc","brightRed":"#cd3131","brightYellow":"#e5e512","black":"#000000","white":"#e5e5e5","blue":"#2472c8","cyan":"#0fa8cd","green":"#05bc79","purple":"#bc3fbc","red":"#cd3131","yellow":"#e5e512"}, {"name":"Argonaut", "foreground":"#fffaf4","background":"#0e1019","brightBlack":"#444444","brightWhite":"#ffffff","brightBlue":"#0092ff","brightCyan":"#67fff0","brightGreen":"#abe15b","brightPurple":"#9a5feb","brightRed":"#ff2740","brightYellow":"#ffd242","black":"#232323","white":"#ffffff","blue":"#008df8","cyan":"#00d8eb","green":"#8ce10b","purple":"#6d43a6","red":"#ff000f","yellow":"#ffb900"}, {"name":"Arthur", "foreground":"#ddeedd","background":"#1c1c1c","brightBlack":"#554444","brightWhite":"#ddccbb","brightBlue":"#87ceeb","brightCyan":"#b0c4de","brightGreen":"#88aa22","brightPurple":"#996600","brightRed":"#cc5533","brightYellow":"#ffa75d","black":"#3d352a","white":"#bbaa99","blue":"#6495ed","cyan":"#b0c4de","green":"#86af80","purple":"#deb887","red":"#cd5c5c","yellow":"#e8ae5b"}, {"name":"AtelierSulphurpool", "foreground":"#979db4","background":"#202746","brightBlack":"#6b7394","brightWhite":"#f5f7ff","brightBlue":"#898ea4","brightCyan":"#9c637a","brightGreen":"#293256","brightPurple":"#dfe2f1","brightRed":"#c76b29","brightYellow":"#5e6687","black":"#202746","white":"#979db4","blue":"#3d8fd1","cyan":"#22a2c9","green":"#ac9739","purple":"#6679cc","red":"#c94922","yellow":"#c08b30"}, {"name":"Atom", "foreground":"#c5c8c6","background":"#161719","brightBlack":"#000000","brightWhite":"#e0e0e0","brightBlue":"#96cbfe","brightCyan":"#85befd","brightGreen":"#94fa36","brightPurple":"#b9b6fc","brightRed":"#fd5ff1","brightYellow":"#f5ffa8","black":"#000000","white":"#e0e0e0","blue":"#85befd","cyan":"#85befd","green":"#87c38a","purple":"#b9b6fc","red":"#fd5ff1","yellow":"#ffd7b1"}, {"name":"AtomOneLight", "foreground":"#2a2c33","background":"#f9f9f9","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#2f5af3","brightCyan":"#3f953a","brightGreen":"#3f953a","brightPurple":"#a00095","brightRed":"#de3e35","brightYellow":"#d2b67c","black":"#000000","white":"#bbbbbb","blue":"#2f5af3","cyan":"#3f953a","green":"#3f953a","purple":"#950095","red":"#de3e35","yellow":"#d2b67c"}, {"name":"ayu", "foreground":"#e6e1cf","background":"#0f1419","brightBlack":"#323232","brightWhite":"#ffffff","brightBlue":"#68d5ff","brightCyan":"#c7fffd","brightGreen":"#eafe84","brightPurple":"#ffa3aa","brightRed":"#ff6565","brightYellow":"#fff779","black":"#000000","white":"#ffffff","blue":"#36a3d9","cyan":"#95e6cb","green":"#b8cc52","purple":"#f07178","red":"#ff3333","yellow":"#e7c547"}, {"name":"ayu_light", "foreground":"#5c6773","background":"#fafafa","brightBlack":"#323232","brightWhite":"#ffffff","brightBlue":"#73d8ff","brightCyan":"#7ff1cb","brightGreen":"#b8e532","brightPurple":"#ffa3aa","brightRed":"#ff6565","brightYellow":"#ffc94a","black":"#000000","white":"#ffffff","blue":"#41a6d9","cyan":"#4dbf99","green":"#86b300","purple":"#f07178","red":"#ff3333","yellow":"#f29718"}, {"name":"Banana Blueberry", "foreground":"#cccccc","background":"#191323","brightBlack":"#495162","brightWhite":"#ffffff","brightBlue":"#91fff4","brightCyan":"#bcf3ff","brightGreen":"#98c379","brightPurple":"#da70d6","brightRed":"#fe9ea1","brightYellow":"#f9e46b","black":"#17141f","white":"#f1f1f1","blue":"#22e8df","cyan":"#56b6c2","green":"#00bd9c","purple":"#dc396a","red":"#ff6b7f","yellow":"#e6c62f"}, {"name":"Batman", "foreground":"#6f6f6f","background":"#1b1d1e","brightBlack":"#505354","brightWhite":"#dadbd6","brightBlue":"#919495","brightCyan":"#a3a3a6","brightGreen":"#fff27d","brightPurple":"#9a9a9d","brightRed":"#fff78e","brightYellow":"#feed6c","black":"#1b1d1e","white":"#c6c5bf","blue":"#737174","cyan":"#62605f","green":"#c8be46","purple":"#747271","red":"#e6dc44","yellow":"#f4fd22"}, {"name":"Belafonte Day", "foreground":"#45373c","background":"#d5ccba","brightBlack":"#5e5252","brightWhite":"#d5ccba","brightBlue":"#426a79","brightCyan":"#989a9c","brightGreen":"#858162","brightPurple":"#97522c","brightRed":"#be100e","brightYellow":"#eaa549","black":"#20111b","white":"#968c83","blue":"#426a79","cyan":"#989a9c","green":"#858162","purple":"#97522c","red":"#be100e","yellow":"#eaa549"}, {"name":"Belafonte Night", "foreground":"#968c83","background":"#20111b","brightBlack":"#5e5252","brightWhite":"#d5ccba","brightBlue":"#426a79","brightCyan":"#989a9c","brightGreen":"#858162","brightPurple":"#97522c","brightRed":"#be100e","brightYellow":"#eaa549","black":"#20111b","white":"#968c83","blue":"#426a79","cyan":"#989a9c","green":"#858162","purple":"#97522c","red":"#be100e","yellow":"#eaa549"}, {"name":"BirdsOfParadise", "foreground":"#e0dbb7","background":"#2a1f1d","brightBlack":"#9b6c4a","brightWhite":"#fff9d5","brightBlue":"#b8d3ed","brightCyan":"#93cfd7","brightGreen":"#95d8ba","brightPurple":"#d19ecb","brightRed":"#e84627","brightYellow":"#d0d150","black":"#573d26","white":"#e0dbb7","blue":"#5a86ad","cyan":"#74a6ad","green":"#6ba18a","purple":"#ac80a6","red":"#be2d26","yellow":"#e99d2a"}, {"name":"Blazer", "foreground":"#d9e6f2","background":"#0d1926","brightBlack":"#262626","brightWhite":"#ffffff","brightBlue":"#bdbddb","brightCyan":"#bddbdb","brightGreen":"#bddbbd","brightPurple":"#dbbddb","brightRed":"#dbbdbd","brightYellow":"#dbdbbd","black":"#000000","white":"#d9d9d9","blue":"#7a7ab8","cyan":"#7ab8b8","green":"#7ab87a","purple":"#b87ab8","red":"#b87a7a","yellow":"#b8b87a"}, {"name":"Blue Matrix", "foreground":"#00a2ff","background":"#101116","brightBlack":"#686868","brightWhite":"#ffffff","brightBlue":"#6871ff","brightCyan":"#60fdff","brightGreen":"#5ffa68","brightPurple":"#d682ec","brightRed":"#ff6e67","brightYellow":"#fffc67","black":"#101116","white":"#c7c7c7","blue":"#00b0ff","cyan":"#76c1ff","green":"#00ff9c","purple":"#d57bff","red":"#ff5680","yellow":"#fffc58"}, {"name":"BlueBerryPie", "foreground":"#babab9","background":"#1c0c28","brightBlack":"#201637","brightWhite":"#0a6c7e","brightBlue":"#39173d","brightCyan":"#5e6071","brightGreen":"#0a6c7e","brightPurple":"#bc94b7","brightRed":"#c87272","brightYellow":"#7a3188","black":"#0a4c62","white":"#f0e8d6","blue":"#90a5bd","cyan":"#7e83cc","green":"#5cb1b3","purple":"#9d54a7","red":"#99246e","yellow":"#eab9a8"}, {"name":"BlulocoDark", "foreground":"#abb2bf","background":"#1e2127","brightBlack":"#61697a","brightWhite":"#ffffff","brightBlue":"#199ffd","brightCyan":"#50acae","brightGreen":"#37bd58","brightPurple":"#fc58f6","brightRed":"#fc4a6d","brightYellow":"#f6be48","black":"#4a505d","white":"#ccd5e5","blue":"#285bff","cyan":"#366f9a","green":"#23974a","purple":"#8c62fd","red":"#f81141","yellow":"#fd7e57"}, {"name":"BlulocoLight", "foreground":"#2a2c33","background":"#f7f7f7","brightBlack":"#dedfe8","brightWhite":"#1d1d22","brightBlue":"#1085d9","brightCyan":"#5b80ad","brightGreen":"#34b354","brightPurple":"#c00db3","brightRed":"#fc4a6d","brightYellow":"#b89427","black":"#cbccd5","white":"#000000","blue":"#1e44dd","cyan":"#1f4d7a","green":"#21883a","purple":"#6d1bed","red":"#c90e42","yellow":"#d54d17"}, {"name":"Borland", "foreground":"#ffff4e","background":"#0000a4","brightBlack":"#7c7c7c","brightWhite":"#ffffff","brightBlue":"#b5dcff","brightCyan":"#dfdffe","brightGreen":"#ceffac","brightPurple":"#ff9cfe","brightRed":"#ffb6b0","brightYellow":"#ffffcc","black":"#4f4f4f","white":"#eeeeee","blue":"#96cbfe","cyan":"#c6c5fe","green":"#a8ff60","purple":"#ff73fd","red":"#ff6c60","yellow":"#ffffb6"}, {"name":"Breeze", "foreground":"#eff0f1","background":"#31363b","brightBlack":"#7f8c8d","brightWhite":"#fcfcfc","brightBlue":"#3daee9","brightCyan":"#16a085","brightGreen":"#1cdc9a","brightPurple":"#8e44ad","brightRed":"#c0392b","brightYellow":"#fdbc4b","black":"#31363b","white":"#eff0f1","blue":"#1d99f3","cyan":"#1abc9c","green":"#11d116","purple":"#9b59b6","red":"#ed1515","yellow":"#f67400"}, {"name":"Bright Lights", "foreground":"#b3c9d7","background":"#191919","brightBlack":"#191919","brightWhite":"#c2c8d7","brightBlue":"#76d5ff","brightCyan":"#6cbfb5","brightGreen":"#b7e876","brightPurple":"#ba76e7","brightRed":"#ff355b","brightYellow":"#ffc251","black":"#191919","white":"#c2c8d7","blue":"#76d4ff","cyan":"#6cbfb5","green":"#b7e876","purple":"#ba76e7","red":"#ff355b","yellow":"#ffc251"}, {"name":"Broadcast", "foreground":"#e6e1dc","background":"#2b2b2b","brightBlack":"#323232","brightWhite":"#ffffff","brightBlue":"#9fcef0","brightCyan":"#a0cef0","brightGreen":"#83d182","brightPurple":"#ffffff","brightRed":"#ff7b6b","brightYellow":"#ffff7c","black":"#000000","white":"#ffffff","blue":"#6d9cbe","cyan":"#6e9cbe","green":"#519f50","purple":"#d0d0ff","red":"#da4939","yellow":"#ffd24a"}, {"name":"Brogrammer", "foreground":"#d6dbe5","background":"#131313","brightBlack":"#d6dbe5","brightWhite":"#ffffff","brightBlue":"#1081d6","brightCyan":"#0f7ddb","brightGreen":"#1dd361","brightPurple":"#5350b9","brightRed":"#de352e","brightYellow":"#f3bd09","black":"#1f1f1f","white":"#d6dbe5","blue":"#2a84d2","cyan":"#1081d6","green":"#2dc55e","purple":"#4e5ab7","red":"#f81118","yellow":"#ecba0f"}, {"name":"Builtin Dark", "foreground":"#bbbbbb","background":"#000000","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#55ffff","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#000000","white":"#bbbbbb","blue":"#0000bb","cyan":"#00bbbb","green":"#00bb00","purple":"#bb00bb","red":"#bb0000","yellow":"#bbbb00"}, {"name":"Builtin Light", "foreground":"#000000","background":"#ffffff","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#55ffff","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#000000","white":"#bbbbbb","blue":"#0000bb","cyan":"#00bbbb","green":"#00bb00","purple":"#bb00bb","red":"#bb0000","yellow":"#bbbb00"}, {"name":"Builtin Pastel Dark", "foreground":"#bbbbbb","background":"#000000","brightBlack":"#7c7c7c","brightWhite":"#ffffff","brightBlue":"#b5dcff","brightCyan":"#dfdffe","brightGreen":"#ceffac","brightPurple":"#ff9cfe","brightRed":"#ffb6b0","brightYellow":"#ffffcc","black":"#4f4f4f","white":"#eeeeee","blue":"#96cbfe","cyan":"#c6c5fe","green":"#a8ff60","purple":"#ff73fd","red":"#ff6c60","yellow":"#ffffb6"}, {"name":"Builtin Solarized Dark", "foreground":"#839496","background":"#002b36","brightBlack":"#002b36","brightWhite":"#fdf6e3","brightBlue":"#839496","brightCyan":"#93a1a1","brightGreen":"#586e75","brightPurple":"#6c71c4","brightRed":"#cb4b16","brightYellow":"#657b83","black":"#073642","white":"#eee8d5","blue":"#268bd2","cyan":"#2aa198","green":"#859900","purple":"#d33682","red":"#dc322f","yellow":"#b58900"}, {"name":"Builtin Solarized Light", "foreground":"#657b83","background":"#fdf6e3","brightBlack":"#002b36","brightWhite":"#fdf6e3","brightBlue":"#839496","brightCyan":"#93a1a1","brightGreen":"#586e75","brightPurple":"#6c71c4","brightRed":"#cb4b16","brightYellow":"#657b83","black":"#073642","white":"#eee8d5","blue":"#268bd2","cyan":"#2aa198","green":"#859900","purple":"#d33682","red":"#dc322f","yellow":"#b58900"}, {"name":"Builtin Tango Dark", "foreground":"#ffffff","background":"#000000","brightBlack":"#555753","brightWhite":"#eeeeec","brightBlue":"#729fcf","brightCyan":"#34e2e2","brightGreen":"#8ae234","brightPurple":"#ad7fa8","brightRed":"#ef2929","brightYellow":"#fce94f","black":"#000000","white":"#d3d7cf","blue":"#3465a4","cyan":"#06989a","green":"#4e9a06","purple":"#75507b","red":"#cc0000","yellow":"#c4a000"}, {"name":"Builtin Tango Light", "foreground":"#000000","background":"#ffffff","brightBlack":"#555753","brightWhite":"#eeeeec","brightBlue":"#729fcf","brightCyan":"#34e2e2","brightGreen":"#8ae234","brightPurple":"#ad7fa8","brightRed":"#ef2929","brightYellow":"#fce94f","black":"#000000","white":"#d3d7cf","blue":"#3465a4","cyan":"#06989a","green":"#4e9a06","purple":"#75507b","red":"#cc0000","yellow":"#c4a000"}, {"name":"C64", "foreground":"#7869c4","background":"#40318d","brightBlack":"#000000","brightWhite":"#f7f7f7","brightBlue":"#40318d","brightCyan":"#67b6bd","brightGreen":"#55a049","brightPurple":"#8b3f96","brightRed":"#883932","brightYellow":"#bfce72","black":"#090300","white":"#ffffff","blue":"#40318d","cyan":"#67b6bd","green":"#55a049","purple":"#8b3f96","red":"#883932","yellow":"#bfce72"}, {"name":"Calamity", "foreground":"#d5ced9","background":"#2f2833","brightBlack":"#7e6c88","brightWhite":"#ffffff","brightBlue":"#3b79c7","brightCyan":"#74d3de","brightGreen":"#a5f69c","brightPurple":"#f92672","brightRed":"#fc644d","brightYellow":"#e9d7a5","black":"#2f2833","white":"#d5ced9","blue":"#3b79c7","cyan":"#74d3de","green":"#a5f69c","purple":"#f92672","red":"#fc644d","yellow":"#e9d7a5"}, {"name":"Chalk", "foreground":"#d2d8d9","background":"#2b2d2e","brightBlack":"#888888","brightWhite":"#d2d8d9","brightBlue":"#4196ff","brightCyan":"#53cdbd","brightGreen":"#80c470","brightPurple":"#fc5275","brightRed":"#f24840","brightYellow":"#ffeb62","black":"#7d8b8f","white":"#d2d8d9","blue":"#2a7fac","cyan":"#44a799","green":"#789b6a","purple":"#bd4f5a","red":"#b23a52","yellow":"#b9ac4a"}, {"name":"Chalkboard", "foreground":"#d9e6f2","background":"#29262f","brightBlack":"#323232","brightWhite":"#ffffff","brightBlue":"#aaaadb","brightCyan":"#aadadb","brightGreen":"#aadbaa","brightPurple":"#dbaada","brightRed":"#dbaaaa","brightYellow":"#dadbaa","black":"#000000","white":"#d9d9d9","blue":"#7372c3","cyan":"#72c2c3","green":"#72c373","purple":"#c372c2","red":"#c37372","yellow":"#c2c372"}, {"name":"ChallengerDeep", "foreground":"#cbe1e7","background":"#1e1c31","brightBlack":"#565575","brightWhite":"#cbe3e7","brightBlue":"#91ddff","brightCyan":"#aaffe4","brightGreen":"#95ffa4","brightPurple":"#c991e1","brightRed":"#ff8080","brightYellow":"#ffe9aa","black":"#141228","white":"#a6b3cc","blue":"#65b2ff","cyan":"#63f2f1","green":"#62d196","purple":"#906cff","red":"#ff5458","yellow":"#ffb378"}, {"name":"Chester", "foreground":"#ffffff","background":"#2c3643","brightBlack":"#6f6b68","brightWhite":"#ffffff","brightBlue":"#278ad6","brightCyan":"#27dede","brightGreen":"#16c98d","brightPurple":"#d34590","brightRed":"#fa5e5b","brightYellow":"#feef6d","black":"#080200","white":"#e7e7e7","blue":"#288ad6","cyan":"#28ddde","green":"#16c98d","purple":"#d34590","red":"#fa5e5b","yellow":"#ffc83f"}, {"name":"Ciapre", "foreground":"#aea47a","background":"#191c27","brightBlack":"#555555","brightWhite":"#f4f4f4","brightBlue":"#3097c6","brightCyan":"#f3dbb2","brightGreen":"#a6a75d","brightPurple":"#d33061","brightRed":"#ac3835","brightYellow":"#dcdf7c","black":"#181818","white":"#aea47f","blue":"#576d8c","cyan":"#5c4f4b","green":"#48513b","purple":"#724d7c","red":"#810009","yellow":"#cc8b3f"}, {"name":"CLRS", "foreground":"#262626","background":"#ffffff","brightBlack":"#555753","brightWhite":"#eeeeec","brightBlue":"#1670ff","brightCyan":"#3ad5ce","brightGreen":"#2cc631","brightPurple":"#e900b0","brightRed":"#fb0416","brightYellow":"#fdd727","black":"#000000","white":"#b3b3b3","blue":"#135cd0","cyan":"#33c3c1","green":"#328a5d","purple":"#9f00bd","red":"#f8282a","yellow":"#fa701d"}, {"name":"Cobalt Neon", "foreground":"#8ff586","background":"#142838","brightBlack":"#fff688","brightWhite":"#8ff586","brightBlue":"#3c7dd2","brightCyan":"#6cbc67","brightGreen":"#8ff586","brightPurple":"#8230a7","brightRed":"#d4312e","brightYellow":"#e9f06d","black":"#142631","white":"#ba46b2","blue":"#8ff586","cyan":"#8ff586","green":"#3ba5ff","purple":"#781aa0","red":"#ff2320","yellow":"#e9e75c"}, {"name":"Cobalt2", "foreground":"#ffffff","background":"#132738","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#6ae3fa","brightGreen":"#3bd01d","brightPurple":"#ff55ff","brightRed":"#f40e17","brightYellow":"#edc809","black":"#000000","white":"#bbbbbb","blue":"#1460d2","cyan":"#00bbbb","green":"#38de21","purple":"#ff005d","red":"#ff0000","yellow":"#ffe50a"}, {"name":"coffee_theme", "foreground":"#000000","background":"#f5deb3","brightBlack":"#686868","brightWhite":"#ffffff","brightBlue":"#6871ff","brightCyan":"#60fdff","brightGreen":"#5ffa68","brightPurple":"#ff77ff","brightRed":"#ff6e67","brightYellow":"#fffc67","black":"#000000","white":"#c7c7c7","blue":"#0225c7","cyan":"#00c5c7","green":"#00c200","purple":"#ca30c7","red":"#c91b00","yellow":"#c7c400"}, {"name":"CrayonPonyFish", "foreground":"#68525a","background":"#150707","brightBlack":"#3d2b2e","brightWhite":"#b0949d","brightBlue":"#cfc9ff","brightCyan":"#ffceaf","brightGreen":"#8dff57","brightPurple":"#fc6cba","brightRed":"#c5255d","brightYellow":"#c8381d","black":"#2b1b1d","white":"#68525a","blue":"#8c87b0","cyan":"#e8a866","green":"#579524","purple":"#692f50","red":"#91002b","yellow":"#ab311b"}, {"name":"Cyberdyne", "foreground":"#00ff92","background":"#151144","brightBlack":"#2e2e2e","brightWhite":"#ffffff","brightBlue":"#c2e3ff","brightCyan":"#e6e7fe","brightGreen":"#d6fcba","brightPurple":"#ffb2fe","brightRed":"#ffc4be","brightYellow":"#fffed5","black":"#080808","white":"#f1f1f1","blue":"#0071cf","cyan":"#6bffdd","green":"#00c172","purple":"#ff90fe","red":"#ff8373","yellow":"#d2a700"}, {"name":"cyberpunk", "foreground":"#e5e5e5","background":"#332a57","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#1bccfd","brightCyan":"#99d6fc","brightGreen":"#21f6bc","brightPurple":"#e6aefe","brightRed":"#ff8aa4","brightYellow":"#fff787","black":"#000000","white":"#ffffff","blue":"#00bfff","cyan":"#86cbfe","green":"#00fbac","purple":"#df95ff","red":"#ff7092","yellow":"#fffa6a"}, {"name":"Dark Pastel", "foreground":"#ffffff","background":"#000000","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#55ffff","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#000000","white":"#bbbbbb","blue":"#5555ff","cyan":"#55ffff","green":"#55ff55","purple":"#ff55ff","red":"#ff5555","yellow":"#ffff55"}, {"name":"Dark+", "foreground":"#cccccc","background":"#0e0e0e","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#3b8eea","brightCyan":"#29b8db","brightGreen":"#23d18b","brightPurple":"#d670d6","brightRed":"#f14c4c","brightYellow":"#f5f543","black":"#000000","white":"#e5e5e5","blue":"#2472c8","cyan":"#11a8cd","green":"#0dbc79","purple":"#bc3fbc","red":"#cd3131","yellow":"#e5e510"}, {"name":"Darkside", "foreground":"#bababa","background":"#222324","brightBlack":"#000000","brightWhite":"#bababa","brightBlue":"#387cd3","brightCyan":"#3d97e2","brightGreen":"#77b869","brightPurple":"#957bbe","brightRed":"#e05a4f","brightYellow":"#efd64b","black":"#000000","white":"#bababa","blue":"#1c98e8","cyan":"#1c98e8","green":"#68c256","purple":"#8e69c9","red":"#e8341c","yellow":"#f2d42c"}, {"name":"deep", "foreground":"#cdcdcd","background":"#090909","brightBlack":"#535353","brightWhite":"#ffffff","brightBlue":"#9fa9ff","brightCyan":"#8df9ff","brightGreen":"#22ff18","brightPurple":"#e09aff","brightRed":"#fb0007","brightYellow":"#fedc2b","black":"#000000","white":"#e0e0e0","blue":"#5665ff","cyan":"#50d2da","green":"#1cd915","purple":"#b052da","red":"#d70005","yellow":"#d9bd26"}, {"name":"Desert", "foreground":"#ffffff","background":"#333333","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#87ceff","brightCyan":"#ffd700","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#4d4d4d","white":"#f5deb3","blue":"#cd853f","cyan":"#ffa0a0","green":"#98fb98","purple":"#ffdead","red":"#ff2b2b","yellow":"#f0e68c"}, {"name":"DimmedMonokai", "foreground":"#b9bcba","background":"#1f1f1f","brightBlack":"#888987","brightWhite":"#fdffb9","brightBlue":"#186de3","brightCyan":"#2e706d","brightGreen":"#0f722f","brightPurple":"#fb0067","brightRed":"#fb001f","brightYellow":"#c47033","black":"#3a3d43","white":"#b9bcba","blue":"#4f76a1","cyan":"#578fa4","green":"#879a3b","purple":"#855c8d","red":"#be3f48","yellow":"#c5a635"}, {"name":"DoomOne", "foreground":"#bbc2cf","background":"#282c34","brightBlack":"#000000","brightWhite":"#bfbfbf","brightBlue":"#a9a1e1","brightCyan":"#51afef","brightGreen":"#99bb66","brightPurple":"#c678dd","brightRed":"#ff6655","brightYellow":"#ecbe7b","black":"#000000","white":"#bbc2cf","blue":"#a9a1e1","cyan":"#51afef","green":"#98be65","purple":"#c678dd","red":"#ff6c6b","yellow":"#ecbe7b"}, {"name":"DotGov", "foreground":"#ebebeb","background":"#262c35","brightBlack":"#191919","brightWhite":"#ffffff","brightBlue":"#17b2e0","brightCyan":"#8bd2ed","brightGreen":"#3d9751","brightPurple":"#7830b0","brightRed":"#bf091d","brightYellow":"#f6bb34","black":"#191919","white":"#ffffff","blue":"#17b2e0","cyan":"#8bd2ed","green":"#3d9751","purple":"#7830b0","red":"#bf091d","yellow":"#f6bb34"}, {"name":"Dracula", "foreground":"#f8f8f2","background":"#1e1f29","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#bd93f9","brightCyan":"#8be9fd","brightGreen":"#50fa7b","brightPurple":"#ff79c6","brightRed":"#ff5555","brightYellow":"#f1fa8c","black":"#000000","white":"#bbbbbb","blue":"#bd93f9","cyan":"#8be9fd","green":"#50fa7b","purple":"#ff79c6","red":"#ff5555","yellow":"#f1fa8c"}, {"name":"Duotone Dark", "foreground":"#b7a1ff","background":"#1f1d27","brightBlack":"#353147","brightWhite":"#eae5ff","brightBlue":"#ffc284","brightCyan":"#2488ff","brightGreen":"#2dcd73","brightPurple":"#de8d40","brightRed":"#d9393e","brightYellow":"#d9b76e","black":"#1f1d27","white":"#b7a1ff","blue":"#ffc284","cyan":"#2488ff","green":"#2dcd73","purple":"#de8d40","red":"#d9393e","yellow":"#d9b76e"}, {"name":"Earthsong", "foreground":"#e5c7a9","background":"#292520","brightBlack":"#675f54","brightWhite":"#f6f7ec","brightBlue":"#5fdaff","brightCyan":"#84f088","brightGreen":"#98e036","brightPurple":"#ff9269","brightRed":"#ff645a","brightYellow":"#e0d561","black":"#121418","white":"#e5c6aa","blue":"#1398b9","cyan":"#509552","green":"#85c54c","purple":"#d0633d","red":"#c94234","yellow":"#f5ae2e"}, {"name":"Elemental", "foreground":"#807a74","background":"#22211d","brightBlack":"#555445","brightWhite":"#fff1e9","brightBlue":"#79d9d9","brightCyan":"#59d599","brightGreen":"#61e070","brightPurple":"#cd7c54","brightRed":"#e0502a","brightYellow":"#d69927","black":"#3c3c30","white":"#807974","blue":"#497f7d","cyan":"#387f58","green":"#479a43","purple":"#7f4e2f","red":"#98290f","yellow":"#7f7111"}, {"name":"Elementary", "foreground":"#efefef","background":"#181818","brightBlack":"#4b4b4b","brightWhite":"#8c00ec","brightBlue":"#0955ff","brightCyan":"#3ea8fc","brightGreen":"#6bc219","brightPurple":"#fb0050","brightRed":"#fc1c18","brightYellow":"#fec80e","black":"#242424","white":"#efefef","blue":"#063b8c","cyan":"#2595e1","green":"#5aa513","purple":"#e40038","red":"#d71c15","yellow":"#fdb40c"}, {"name":"ENCOM", "foreground":"#00a595","background":"#000000","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#0000ff","brightCyan":"#00cdcd","brightGreen":"#00ee00","brightPurple":"#ff00ff","brightRed":"#ff0000","brightYellow":"#ffff00","black":"#000000","white":"#bbbbbb","blue":"#0081ff","cyan":"#008b8b","green":"#008b00","purple":"#bc00ca","red":"#9f0000","yellow":"#ffd000"}, {"name":"Espresso Libre", "foreground":"#b8a898","background":"#2a211c","brightBlack":"#555753","brightWhite":"#eeeeec","brightBlue":"#43a8ed","brightCyan":"#34e2e2","brightGreen":"#9aff87","brightPurple":"#ff818a","brightRed":"#ef2929","brightYellow":"#fffb5c","black":"#000000","white":"#d3d7cf","blue":"#0066ff","cyan":"#06989a","green":"#1a921c","purple":"#c5656b","red":"#cc0000","yellow":"#f0e53a"}, {"name":"Espresso", "foreground":"#ffffff","background":"#323232","brightBlack":"#535353","brightWhite":"#ffffff","brightBlue":"#8ab7d9","brightCyan":"#dcf4ff","brightGreen":"#c2e075","brightPurple":"#efb5f7","brightRed":"#f00c0c","brightYellow":"#e1e48b","black":"#353535","white":"#eeeeec","blue":"#6c99bb","cyan":"#bed6ff","green":"#a5c261","purple":"#d197d9","red":"#d25252","yellow":"#ffc66d"}, {"name":"Fahrenheit", "foreground":"#ffffce","background":"#000000","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#cb4a05","brightCyan":"#fed04d","brightGreen":"#cc734d","brightPurple":"#4e739f","brightRed":"#fecea0","brightYellow":"#fd9f4d","black":"#1d1d1d","white":"#ffffce","blue":"#720102","cyan":"#979797","green":"#9e744d","purple":"#734c4d","red":"#cda074","yellow":"#fecf75"}, {"name":"Fideloper", "foreground":"#dbdae0","background":"#292f33","brightBlack":"#092028","brightWhite":"#fcf4df","brightBlue":"#7c85c4","brightCyan":"#819090","brightGreen":"#d4605a","brightPurple":"#5c5db2","brightRed":"#d4605a","brightYellow":"#a86671","black":"#292f33","white":"#eae3ce","blue":"#2e78c2","cyan":"#309186","green":"#edb8ac","purple":"#c0236f","red":"#cb1e2d","yellow":"#b7ab9b"}, {"name":"FirefoxDev", "foreground":"#7c8fa4","background":"#0e1011","brightBlack":"#001e27","brightWhite":"#e2e2e2","brightBlue":"#006fc0","brightCyan":"#005794","brightGreen":"#1d9000","brightPurple":"#a200da","brightRed":"#e1003f","brightYellow":"#cd9409","black":"#002831","white":"#dcdcdc","blue":"#359ddf","cyan":"#4b73a2","green":"#5eb83c","purple":"#d75cff","red":"#e63853","yellow":"#a57706"}, {"name":"Firewatch", "foreground":"#9ba2b2","background":"#1e2027","brightBlack":"#585f6d","brightWhite":"#e6e5ff","brightBlue":"#4c89c5","brightCyan":"#44a8b6","brightGreen":"#5ab977","brightPurple":"#d55119","brightRed":"#d95360","brightYellow":"#dfb563","black":"#585f6d","white":"#e6e5ff","blue":"#4d89c4","cyan":"#44a8b6","green":"#5ab977","purple":"#d55119","red":"#d95360","yellow":"#dfb563"}, {"name":"FishTank", "foreground":"#ecf0fe","background":"#232537","brightBlack":"#6c5b30","brightWhite":"#f6ffec","brightBlue":"#b2befa","brightCyan":"#a5bd86","brightGreen":"#dbffa9","brightPurple":"#fda5cd","brightRed":"#da4b8a","brightYellow":"#fee6a9","black":"#03073c","white":"#ecf0fc","blue":"#525fb8","cyan":"#968763","green":"#acf157","purple":"#986f82","red":"#c6004a","yellow":"#fecd5e"}, {"name":"Flat", "foreground":"#2cc55d","background":"#002240","brightBlack":"#212c3c","brightWhite":"#e7eced","brightBlue":"#3c7dd2","brightCyan":"#35b387","brightGreen":"#2d9440","brightPurple":"#8230a7","brightRed":"#d4312e","brightYellow":"#e5be0c","black":"#222d3f","white":"#b0b6ba","blue":"#3167ac","cyan":"#2c9370","green":"#32a548","purple":"#781aa0","red":"#a82320","yellow":"#e58d11"}, {"name":"Flatland", "foreground":"#b8dbef","background":"#1d1f21","brightBlack":"#1d1d19","brightWhite":"#ffffff","brightBlue":"#61b9d0","brightCyan":"#d63865","brightGreen":"#a7d42c","brightPurple":"#695abc","brightRed":"#d22a24","brightYellow":"#ff8949","black":"#1d1d19","white":"#ffffff","blue":"#5096be","cyan":"#d63865","green":"#9fd364","purple":"#695abc","red":"#f18339","yellow":"#f4ef6d"}, {"name":"Floraverse", "foreground":"#dbd1b9","background":"#0e0d15","brightBlack":"#331e4d","brightWhite":"#fff5db","brightBlue":"#40a4cf","brightCyan":"#62caa8","brightGreen":"#b4ce59","brightPurple":"#f12aae","brightRed":"#d02063","brightYellow":"#fac357","black":"#08002e","white":"#f3e0b8","blue":"#1d6da1","cyan":"#42a38c","green":"#5d731a","purple":"#b7077e","red":"#64002c","yellow":"#cd751c"}, {"name":"ForestBlue", "foreground":"#e2d8cd","background":"#051519","brightBlack":"#3d3d3d","brightWhite":"#e2d8cd","brightBlue":"#39a7a2","brightCyan":"#6096bf","brightGreen":"#6bb48d","brightPurple":"#7e62b3","brightRed":"#fb3d66","brightYellow":"#30c85a","black":"#333333","white":"#e2d8cd","blue":"#8ed0ce","cyan":"#31658c","green":"#92d3a2","purple":"#5e468c","red":"#f8818e","yellow":"#1a8e63"}, {"name":"Framer", "foreground":"#777777","background":"#111111","brightBlack":"#414141","brightWhite":"#ffffff","brightBlue":"#33bbff","brightCyan":"#bbecff","brightGreen":"#b6f292","brightPurple":"#cebbff","brightRed":"#ff8888","brightYellow":"#ffd966","black":"#141414","white":"#cccccc","blue":"#00aaff","cyan":"#88ddff","green":"#98ec65","purple":"#aa88ff","red":"#ff5555","yellow":"#ffcc33"}, {"name":"FrontEndDelight", "foreground":"#adadad","background":"#1b1c1d","brightBlack":"#5fac6d","brightWhite":"#8c735b","brightBlue":"#3393ca","brightCyan":"#4fbce6","brightGreen":"#74ec4c","brightPurple":"#e75e4f","brightRed":"#f74319","brightYellow":"#fdc325","black":"#242526","white":"#adadad","blue":"#2c70b7","cyan":"#3ca1a6","green":"#565747","purple":"#f02e4f","red":"#f8511b","yellow":"#fa771d"}, {"name":"FunForrest", "foreground":"#dec165","background":"#251200","brightBlack":"#7f6a55","brightWhite":"#ffeaa3","brightBlue":"#7cc9cf","brightCyan":"#e6a96b","brightGreen":"#bfc65a","brightPurple":"#d26349","brightRed":"#e55a1c","brightYellow":"#ffcb1b","black":"#000000","white":"#ddc265","blue":"#4699a3","cyan":"#da8213","green":"#919c00","purple":"#8d4331","red":"#d6262b","yellow":"#be8a13"}, {"name":"Galaxy", "foreground":"#ffffff","background":"#1d2837","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#589df6","brightCyan":"#3979bc","brightGreen":"#35bb9a","brightPurple":"#e75699","brightRed":"#fa8c8f","brightYellow":"#ffff55","black":"#000000","white":"#bbbbbb","blue":"#589df6","cyan":"#1f9ee7","green":"#21b089","purple":"#944d95","red":"#f9555f","yellow":"#fef02a"}, {"name":"Github", "foreground":"#3e3e3e","background":"#f4f4f4","brightBlack":"#666666","brightWhite":"#ffffff","brightBlue":"#2e6cba","brightCyan":"#1cfafe","brightGreen":"#87d5a2","brightPurple":"#ffa29f","brightRed":"#de0000","brightYellow":"#f1d007","black":"#3e3e3e","white":"#ffffff","blue":"#003e8a","cyan":"#89d1ec","green":"#07962a","purple":"#e94691","red":"#970b16","yellow":"#f8eec7"}, {"name":"Glacier", "foreground":"#ffffff","background":"#0c1115","brightBlack":"#404a55","brightWhite":"#ffffff","brightBlue":"#2a8bc1","brightCyan":"#a0b6d3","brightGreen":"#49e998","brightPurple":"#ea4727","brightRed":"#bd0f2f","brightYellow":"#fddf6e","black":"#2e343c","white":"#ffffff","blue":"#1f5872","cyan":"#778397","green":"#35a770","purple":"#bd2523","red":"#bd0f2f","yellow":"#fb9435"}, {"name":"Grape", "foreground":"#9f9fa1","background":"#171423","brightBlack":"#59516a","brightWhite":"#a288f7","brightBlue":"#a9bcec","brightCyan":"#9de3eb","brightGreen":"#53aa5e","brightPurple":"#ad81c2","brightRed":"#f0729a","brightYellow":"#b2dc87","black":"#2d283f","white":"#9e9ea0","blue":"#487df4","cyan":"#3bdeed","green":"#1fa91b","purple":"#8d35c9","red":"#ed2261","yellow":"#8ddc20"}, {"name":"Grass", "foreground":"#fff0a5","background":"#13773d","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#0000bb","brightCyan":"#55ffff","brightGreen":"#00bb00","brightPurple":"#ff55ff","brightRed":"#bb0000","brightYellow":"#e7b000","black":"#000000","white":"#bbbbbb","blue":"#0000a3","cyan":"#00bbbb","green":"#00bb00","purple":"#950062","red":"#bb0000","yellow":"#e7b000"}, {"name":"Gruvbox Dark", "foreground":"#e6d4a3","background":"#1e1e1e","brightBlack":"#7f7061","brightWhite":"#e6d4a3","brightBlue":"#719586","brightCyan":"#7db669","brightGreen":"#aab01e","brightPurple":"#c77089","brightRed":"#f73028","brightYellow":"#f7b125","black":"#1e1e1e","white":"#978771","blue":"#377375","cyan":"#578e57","green":"#868715","purple":"#a04b73","red":"#be0f17","yellow":"#cc881a"}, {"name":"Hacktober", "foreground":"#c9c9c9","background":"#141414","brightBlack":"#2c2b2a","brightWhite":"#ffffff","brightBlue":"#5389c5","brightCyan":"#ebc587","brightGreen":"#42824a","brightPurple":"#e795a5","brightRed":"#b33323","brightYellow":"#c75a22","black":"#191918","white":"#f1eee7","blue":"#206ec5","cyan":"#ac9166","green":"#587744","purple":"#864651","red":"#b34538","yellow":"#d08949"}, {"name":"Hardcore", "foreground":"#a0a0a0","background":"#121212","brightBlack":"#505354","brightWhite":"#f8f8f2","brightBlue":"#66d9ef","brightCyan":"#a3babf","brightGreen":"#beed5f","brightPurple":"#9e6ffe","brightRed":"#ff669d","brightYellow":"#e6db74","black":"#1b1d1e","white":"#ccccc6","blue":"#66d9ef","cyan":"#5e7175","green":"#a6e22e","purple":"#9e6ffe","red":"#f92672","yellow":"#fd971f"}, {"name":"Harper", "foreground":"#a8a49d","background":"#010101","brightBlack":"#726e6a","brightWhite":"#fefbea","brightBlue":"#489e48","brightCyan":"#f5bfd7","brightGreen":"#7fb5e1","brightPurple":"#b296c6","brightRed":"#f8b63f","brightYellow":"#d6da25","black":"#010101","white":"#a8a49d","blue":"#489e48","cyan":"#f5bfd7","green":"#7fb5e1","purple":"#b296c6","red":"#f8b63f","yellow":"#d6da25"}, {"name":"Highway", "foreground":"#ededed","background":"#222225","brightBlack":"#5d504a","brightWhite":"#ffffff","brightBlue":"#4fc2fd","brightCyan":"#5d504a","brightGreen":"#b1d130","brightPurple":"#de0071","brightRed":"#f07e18","brightYellow":"#fff120","black":"#000000","white":"#ededed","blue":"#006bb3","cyan":"#384564","green":"#138034","purple":"#6b2775","red":"#d00e18","yellow":"#ffcb3e"}, {"name":"Hipster Green", "foreground":"#84c138","background":"#100b05","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#86a93e","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#000000","white":"#bfbfbf","blue":"#246eb2","cyan":"#00a6b2","green":"#00a600","purple":"#b200b2","red":"#b6214a","yellow":"#bfbf00"}, {"name":"Hivacruz", "foreground":"#ede4e4","background":"#132638","brightBlack":"#6b7394","brightWhite":"#f5f7ff","brightBlue":"#898ea4","brightCyan":"#9c637a","brightGreen":"#73ad43","brightPurple":"#dfe2f1","brightRed":"#c76b29","brightYellow":"#5e6687","black":"#202746","white":"#979db4","blue":"#3d8fd1","cyan":"#22a2c9","green":"#ac9739","purple":"#6679cc","red":"#c94922","yellow":"#c08b30"}, {"name":"Homebrew", "foreground":"#00ff00","background":"#000000","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#00d900","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#000000","white":"#bfbfbf","blue":"#0000b2","cyan":"#00a6b2","green":"#00a600","purple":"#b200b2","red":"#990000","yellow":"#999900"}, {"name":"Hopscotch.256", "foreground":"#b9b5b8","background":"#322931","brightBlack":"#797379","brightWhite":"#ffffff","brightBlue":"#1290bf","brightCyan":"#149b93","brightGreen":"#8fc13e","brightPurple":"#c85e7c","brightRed":"#dd464c","brightYellow":"#fdcc59","black":"#322931","white":"#b9b5b8","blue":"#1290bf","cyan":"#149b93","green":"#8fc13e","purple":"#c85e7c","red":"#dd464c","yellow":"#fdcc59"}, {"name":"Hopscotch", "foreground":"#b9b5b8","background":"#322931","brightBlack":"#797379","brightWhite":"#ffffff","brightBlue":"#989498","brightCyan":"#b33508","brightGreen":"#433b42","brightPurple":"#d5d3d5","brightRed":"#fd8b19","brightYellow":"#5c545b","black":"#322931","white":"#b9b5b8","blue":"#1290bf","cyan":"#149b93","green":"#8fc13e","purple":"#c85e7c","red":"#dd464c","yellow":"#fdcc59"}, {"name":"Hurtado", "foreground":"#dbdbdb","background":"#000000","brightBlack":"#262626","brightWhite":"#dbdbdb","brightBlue":"#89beff","brightCyan":"#86eafe","brightGreen":"#a5df55","brightPurple":"#c001c1","brightRed":"#d51d00","brightYellow":"#fbe84a","black":"#575757","white":"#cbcccb","blue":"#496487","cyan":"#86e9fe","green":"#a5e055","purple":"#fd5ff1","red":"#ff1b00","yellow":"#fbe74a"}, {"name":"Hybrid", "foreground":"#b7bcba","background":"#161719","brightBlack":"#1d1f22","brightWhite":"#5a626a","brightBlue":"#4b6b88","brightCyan":"#4d7b74","brightGreen":"#798431","brightPurple":"#6e5079","brightRed":"#8d2e32","brightYellow":"#e58a50","black":"#2a2e33","white":"#b5b9b6","blue":"#6e90b0","cyan":"#7fbfb4","green":"#b3bf5a","purple":"#a17eac","red":"#b84d51","yellow":"#e4b55e"}, {"name":"IC_Green_PPL", "foreground":"#e0f1dc","background":"#2c2c2c","brightBlack":"#035c03","brightWhite":"#e0f1dc","brightBlue":"#2efaeb","brightCyan":"#3cfac8","brightGreen":"#aefb86","brightPurple":"#50fafa","brightRed":"#b4fa5c","brightYellow":"#dafa87","black":"#014401","white":"#e6fef2","blue":"#2ec3b9","cyan":"#3ca078","green":"#41a638","purple":"#50a096","red":"#ff2736","yellow":"#76a831"}, {"name":"IC_Orange_PPL", "foreground":"#ffcb83","background":"#262626","brightBlack":"#6a4f2a","brightWhite":"#fafaff","brightBlue":"#ffbe55","brightCyan":"#c69752","brightGreen":"#f6ff40","brightPurple":"#fc874f","brightRed":"#ff8c68","brightYellow":"#ffe36e","black":"#000000","white":"#ffc88a","blue":"#bd6d00","cyan":"#f79500","green":"#a4a900","purple":"#fc5e00","red":"#c13900","yellow":"#caaf00"}, {"name":"idea", "foreground":"#adadad","background":"#202020","brightBlack":"#ffffff","brightWhite":"#181818","brightBlue":"#6c9ced","brightCyan":"#248887","brightGreen":"#98b61c","brightPurple":"#fc7eff","brightRed":"#fc7072","brightYellow":"#ffff0b","black":"#adadad","white":"#181818","blue":"#437ee7","cyan":"#248887","green":"#98b61c","purple":"#9d74b0","red":"#fc5256","yellow":"#ccb444"}, {"name":"idleToes", "foreground":"#ffffff","background":"#323232","brightBlack":"#535353","brightWhite":"#ffffff","brightBlue":"#5eb7f7","brightCyan":"#dcf4ff","brightGreen":"#9dff91","brightPurple":"#ff9dff","brightRed":"#f07070","brightYellow":"#ffe48b","black":"#323232","white":"#eeeeec","blue":"#4099ff","cyan":"#bed6ff","green":"#7fe173","purple":"#f680ff","red":"#d25252","yellow":"#ffc66d"}, {"name":"IR_Black", "foreground":"#f1f1f1","background":"#000000","brightBlack":"#7b7b7b","brightWhite":"#ffffff","brightBlue":"#b5dcff","brightCyan":"#e0e0fe","brightGreen":"#cfffab","brightPurple":"#fb9cfe","brightRed":"#fcb6b0","brightYellow":"#ffffcc","black":"#4f4f4f","white":"#efedef","blue":"#96cafe","cyan":"#c6c5fe","green":"#a8ff60","purple":"#fa73fd","red":"#fa6c60","yellow":"#fffeb7"}, {"name":"Jackie Brown", "foreground":"#ffcc2f","background":"#2c1d16","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#86a93e","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#2c1d16","white":"#bfbfbf","blue":"#246eb2","cyan":"#00acee","green":"#2baf2b","purple":"#d05ec1","red":"#ef5734","yellow":"#bebf00"}, {"name":"Japanesque", "foreground":"#f7f6ec","background":"#1e1e1e","brightBlack":"#595b59","brightWhite":"#b2b5ae","brightBlue":"#135979","brightCyan":"#76bbca","brightGreen":"#767f2c","brightPurple":"#604291","brightRed":"#d18fa6","brightYellow":"#78592f","black":"#343935","white":"#fafaf6","blue":"#4c9ad4","cyan":"#389aad","green":"#7bb75b","purple":"#a57fc4","red":"#cf3f61","yellow":"#e9b32a"}, {"name":"Jellybeans", "foreground":"#dedede","background":"#121212","brightBlack":"#bdbdbd","brightWhite":"#ffffff","brightBlue":"#b1d8f6","brightCyan":"#1ab2a8","brightGreen":"#bddeab","brightPurple":"#fbdaff","brightRed":"#ffa1a1","brightYellow":"#ffdca0","black":"#929292","white":"#dedede","blue":"#97bedc","cyan":"#00988e","green":"#94b979","purple":"#e1c0fa","red":"#e27373","yellow":"#ffba7b"}, {"name":"JetBrains Darcula", "foreground":"#adadad","background":"#202020","brightBlack":"#555555","brightWhite":"#eeeeee","brightBlue":"#6d9df1","brightCyan":"#60d3d1","brightGreen":"#67ff4f","brightPurple":"#fb82ff","brightRed":"#fb7172","brightYellow":"#ffff00","black":"#000000","white":"#adadad","blue":"#4581eb","cyan":"#33c2c1","green":"#126e00","purple":"#fa54ff","red":"#fa5355","yellow":"#c2c300"}, {"name":"Kibble", "foreground":"#f7f7f7","background":"#0e100a","brightBlack":"#5a5a5a","brightWhite":"#ffffff","brightBlue":"#97a4f7","brightCyan":"#68f2e0","brightGreen":"#6ce05c","brightPurple":"#c495f0","brightRed":"#f01578","brightYellow":"#f3f79e","black":"#4d4d4d","white":"#e2d1e3","blue":"#3449d1","cyan":"#0798ab","green":"#29cf13","purple":"#8400ff","red":"#c70031","yellow":"#d8e30e"}, {"name":"Kolorit", "foreground":"#efecec","background":"#1d1a1e","brightBlack":"#1d1a1e","brightWhite":"#ededed","brightBlue":"#5db4ee","brightCyan":"#57e9eb","brightGreen":"#47d7a1","brightPurple":"#da6cda","brightRed":"#ff5b82","brightYellow":"#e8e562","black":"#1d1a1e","white":"#ededed","blue":"#5db4ee","cyan":"#57e9eb","green":"#47d7a1","purple":"#da6cda","red":"#ff5b82","yellow":"#e8e562"}, {"name":"Lab Fox", "foreground":"#ffffff","background":"#2e2e2e","brightBlack":"#464646","brightWhite":"#ffffff","brightBlue":"#db501f","brightCyan":"#7d53e7","brightGreen":"#53eaa8","brightPurple":"#441090","brightRed":"#ff6517","brightYellow":"#fca013","black":"#2e2e2e","white":"#ffffff","blue":"#db3b21","cyan":"#6e49cb","green":"#3eb383","purple":"#380d75","red":"#fc6d26","yellow":"#fca121"}, {"name":"Laser", "foreground":"#f106e3","background":"#030d18","brightBlack":"#8f8f8f","brightWhite":"#ffffff","brightBlue":"#f92883","brightCyan":"#e6e7fe","brightGreen":"#d6fcba","brightPurple":"#ffb2fe","brightRed":"#ffc4be","brightYellow":"#fffed5","black":"#626262","white":"#f1f1f1","blue":"#fed300","cyan":"#d1d1fe","green":"#b4fb73","purple":"#ff90fe","red":"#ff8373","yellow":"#09b4bd"}, {"name":"Later This Evening", "foreground":"#959595","background":"#222222","brightBlack":"#454747","brightWhite":"#c1c2c2","brightBlue":"#6699d6","brightCyan":"#5fc0ae","brightGreen":"#aabb39","brightPurple":"#ab53d6","brightRed":"#d3232f","brightYellow":"#e5be39","black":"#2b2b2b","white":"#3c3d3d","blue":"#a0bad6","cyan":"#91bfb7","green":"#afba67","purple":"#c092d6","red":"#d45a60","yellow":"#e5d289"}, {"name":"Lavandula", "foreground":"#736e7d","background":"#050014","brightBlack":"#372d46","brightWhite":"#8c91fa","brightBlue":"#8e87e0","brightCyan":"#9ad4e0","brightGreen":"#52e0c4","brightPurple":"#a776e0","brightRed":"#e05167","brightYellow":"#e0c386","black":"#230046","white":"#736e7d","blue":"#4f4a7f","cyan":"#58777f","green":"#337e6f","purple":"#5a3f7f","red":"#7d1625","yellow":"#7f6f49"}, {"name":"LiquidCarbon", "foreground":"#afc2c2","background":"#303030","brightBlack":"#000000","brightWhite":"#bccccc","brightBlue":"#0099cc","brightCyan":"#7ac4cc","brightGreen":"#559a70","brightPurple":"#cc69c8","brightRed":"#ff3030","brightYellow":"#ccac00","black":"#000000","white":"#bccccc","blue":"#0099cc","cyan":"#7ac4cc","green":"#559a70","purple":"#cc69c8","red":"#ff3030","yellow":"#ccac00"}, {"name":"LiquidCarbonTransparent", "foreground":"#afc2c2","background":"#000000","brightBlack":"#000000","brightWhite":"#bccccc","brightBlue":"#0099cc","brightCyan":"#7ac4cc","brightGreen":"#559a70","brightPurple":"#cc69c8","brightRed":"#ff3030","brightYellow":"#ccac00","black":"#000000","white":"#bccccc","blue":"#0099cc","cyan":"#7ac4cc","green":"#559a70","purple":"#cc69c8","red":"#ff3030","yellow":"#ccac00"}, {"name":"LiquidCarbonTransparentInvers", "foreground":"#afc2c2","background":"#000000","brightBlack":"#ffffff","brightWhite":"#000000","brightBlue":"#0099cc","brightCyan":"#7ac4cc","brightGreen":"#559a70","brightPurple":"#cc69c8","brightRed":"#ff3030","brightYellow":"#ccac00","black":"#bccccd","white":"#000000","blue":"#0099cc","cyan":"#7ac4cc","green":"#559a70","purple":"#cc69c8","red":"#ff3030","yellow":"#ccac00"}, {"name":"lovelace", "foreground":"#fdfdfd","background":"#1d1f28","brightBlack":"#414458","brightWhite":"#bebec1","brightBlue":"#556fff","brightCyan":"#3fdcee","brightGreen":"#18e3c8","brightPurple":"#b043d1","brightRed":"#ff4971","brightYellow":"#ff8037","black":"#282a36","white":"#fdfdfd","blue":"#8897f4","cyan":"#79e6f3","green":"#5adecd","purple":"#c574dd","red":"#f37f97","yellow":"#f2a272"}, {"name":"Man Page", "foreground":"#000000","background":"#fef49c","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#00d900","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#000000","white":"#cccccc","blue":"#0000b2","cyan":"#00a6b2","green":"#00a600","purple":"#b200b2","red":"#cc0000","yellow":"#999900"}, {"name":"Material", "foreground":"#232322","background":"#eaeaea","brightBlack":"#424242","brightWhite":"#d9d9d9","brightBlue":"#54a4f3","brightCyan":"#26bbd1","brightGreen":"#7aba3a","brightPurple":"#aa4dbc","brightRed":"#e83b3f","brightYellow":"#ffea2e","black":"#212121","white":"#efefef","blue":"#134eb2","cyan":"#0e717c","green":"#457b24","purple":"#560088","red":"#b7141f","yellow":"#f6981e"}, {"name":"MaterialDark", "foreground":"#e5e5e5","background":"#232322","brightBlack":"#424242","brightWhite":"#d9d9d9","brightBlue":"#54a4f3","brightCyan":"#26bbd1","brightGreen":"#7aba3a","brightPurple":"#aa4dbc","brightRed":"#e83b3f","brightYellow":"#ffea2e","black":"#212121","white":"#efefef","blue":"#134eb2","cyan":"#0e717c","green":"#457b24","purple":"#560088","red":"#b7141f","yellow":"#f6981e"}, {"name":"MaterialOcean", "foreground":"#8f93a2","background":"#0f111a","brightBlack":"#546e7a","brightWhite":"#ffffff","brightBlue":"#82aaff","brightCyan":"#89ddff","brightGreen":"#c3e88d","brightPurple":"#c792ea","brightRed":"#ff5370","brightYellow":"#ffcb6b","black":"#546e7a","white":"#ffffff","blue":"#82aaff","cyan":"#89ddff","green":"#c3e88d","purple":"#c792ea","red":"#ff5370","yellow":"#ffcb6b"}, {"name":"Mathias", "foreground":"#bbbbbb","background":"#000000","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#55ffff","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#000000","white":"#f2f2f2","blue":"#c48dff","cyan":"#67d9f0","green":"#a6e32d","purple":"#fa2573","red":"#e52222","yellow":"#fc951e"}, {"name":"Medallion", "foreground":"#cac296","background":"#1d1908","brightBlack":"#5e5219","brightWhite":"#fed698","brightBlue":"#acb8ff","brightCyan":"#ffbc51","brightGreen":"#b2ca3b","brightPurple":"#ffa0ff","brightRed":"#ff9149","brightYellow":"#ffe54a","black":"#000000","white":"#cac29a","blue":"#616bb0","cyan":"#916c25","green":"#7c8b16","purple":"#8c5a90","red":"#b64c00","yellow":"#d3bd26"}, {"name":"midnight-in-mojave", "foreground":"#ffffff","background":"#1e1e1e","brightBlack":"#1e1e1e","brightWhite":"#ffffff","brightBlue":"#0a84ff","brightCyan":"#5ac8fa","brightGreen":"#32d74b","brightPurple":"#bf5af2","brightRed":"#ff453a","brightYellow":"#ffd60a","black":"#1e1e1e","white":"#ffffff","blue":"#0a84ff","cyan":"#5ac8fa","green":"#32d74b","purple":"#bf5af2","red":"#ff453a","yellow":"#ffd60a"}, {"name":"Misterioso", "foreground":"#e1e1e0","background":"#2d3743","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#23d7d7","brightCyan":"#00ede1","brightGreen":"#74cd68","brightPurple":"#ff37ff","brightRed":"#ff3242","brightYellow":"#ffb929","black":"#000000","white":"#e1e1e0","blue":"#338f86","cyan":"#23d7d7","green":"#74af68","purple":"#9414e6","red":"#ff4242","yellow":"#ffad29"}, {"name":"Molokai", "foreground":"#bbbbbb","background":"#121212","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#00afff","brightCyan":"#51ceff","brightGreen":"#b1e05f","brightPurple":"#af87ff","brightRed":"#f6669d","brightYellow":"#fff26d","black":"#121212","white":"#bbbbbb","blue":"#1080d0","cyan":"#43a8d0","green":"#98e123","purple":"#8700ff","red":"#fa2573","yellow":"#dfd460"}, {"name":"MonaLisa", "foreground":"#f7d66a","background":"#120b0d","brightBlack":"#874228","brightWhite":"#ffe598","brightBlue":"#9eb2b4","brightCyan":"#8acd8f","brightGreen":"#b4b264","brightPurple":"#ff5b6a","brightRed":"#ff4331","brightYellow":"#ff9566","black":"#351b0e","white":"#f7d75c","blue":"#515c5d","cyan":"#588056","green":"#636232","purple":"#9b1d29","red":"#9b291c","yellow":"#c36e28"}, {"name":"Monokai Remastered", "foreground":"#d9d9d9","background":"#0c0c0c","brightBlack":"#625e4c","brightWhite":"#f6f6ef","brightBlue":"#9d65ff","brightCyan":"#58d1eb","brightGreen":"#98e024","brightPurple":"#f4005f","brightRed":"#f4005f","brightYellow":"#e0d561","black":"#1a1a1a","white":"#c4c5b5","blue":"#9d65ff","cyan":"#58d1eb","green":"#98e024","purple":"#f4005f","red":"#f4005f","yellow":"#fd971f"}, {"name":"Monokai Soda", "foreground":"#c4c5b5","background":"#1a1a1a","brightBlack":"#625e4c","brightWhite":"#f6f6ef","brightBlue":"#9d65ff","brightCyan":"#58d1eb","brightGreen":"#98e024","brightPurple":"#f4005f","brightRed":"#f4005f","brightYellow":"#e0d561","black":"#1a1a1a","white":"#c4c5b5","blue":"#9d65ff","cyan":"#58d1eb","green":"#98e024","purple":"#f4005f","red":"#f4005f","yellow":"#fa8419"}, {"name":"Monokai Vivid", "foreground":"#f9f9f9","background":"#121212","brightBlack":"#838383","brightWhite":"#ffffff","brightBlue":"#0443ff","brightCyan":"#51ceff","brightGreen":"#b1e05f","brightPurple":"#f200f6","brightRed":"#f6669d","brightYellow":"#fff26d","black":"#121212","white":"#ffffff","blue":"#0443ff","cyan":"#01b6ed","green":"#98e123","purple":"#f800f8","red":"#fa2934","yellow":"#fff30a"}, {"name":"N0tch2k", "foreground":"#a0a0a0","background":"#222222","brightBlack":"#474747","brightWhite":"#d8c8bb","brightBlue":"#98bd5e","brightCyan":"#dcdcdc","brightGreen":"#8c8c8c","brightPurple":"#a3a3a3","brightRed":"#a97775","brightYellow":"#a99175","black":"#383838","white":"#d0b8a3","blue":"#657d3e","cyan":"#c9c9c9","green":"#666666","purple":"#767676","red":"#a95551","yellow":"#a98051"}, {"name":"Neopolitan", "foreground":"#ffffff","background":"#271f19","brightBlack":"#000000","brightWhite":"#f8f8f8","brightBlue":"#253b76","brightCyan":"#8da6ce","brightGreen":"#61ce3c","brightPurple":"#ff0080","brightRed":"#800000","brightYellow":"#fbde2d","black":"#000000","white":"#f8f8f8","blue":"#253b76","cyan":"#8da6ce","green":"#61ce3c","purple":"#ff0080","red":"#800000","yellow":"#fbde2d"}, {"name":"Neutron", "foreground":"#e6e8ef","background":"#1c1e22","brightBlack":"#23252b","brightWhite":"#ebedf2","brightBlue":"#6a7c93","brightCyan":"#3f94a8","brightGreen":"#5ab977","brightPurple":"#a4799d","brightRed":"#b54036","brightYellow":"#deb566","black":"#23252b","white":"#e6e8ef","blue":"#6a7c93","cyan":"#3f94a8","green":"#5ab977","purple":"#a4799d","red":"#b54036","yellow":"#deb566"}, {"name":"Night Owlish Light", "foreground":"#403f53","background":"#ffffff","brightBlack":"#7a8181","brightWhite":"#989fb1","brightBlue":"#5ca7e4","brightCyan":"#00c990","brightGreen":"#49d0c5","brightPurple":"#697098","brightRed":"#f76e6e","brightYellow":"#dac26b","black":"#011627","white":"#7a8181","blue":"#4876d6","cyan":"#08916a","green":"#2aa298","purple":"#403f53","red":"#d3423e","yellow":"#daaa01"}, {"name":"NightLion v1", "foreground":"#bbbbbb","background":"#000000","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#55ffff","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#4c4c4c","white":"#bbbbbb","blue":"#276bd8","cyan":"#00dadf","green":"#5fde8f","purple":"#bb00bb","red":"#bb0000","yellow":"#f3f167"}, {"name":"NightLion v2", "foreground":"#bbbbbb","background":"#171717","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#62cbe8","brightCyan":"#00ccd8","brightGreen":"#7df71d","brightPurple":"#ff9bf5","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#4c4c4c","white":"#bbbbbb","blue":"#64d0f0","cyan":"#00dadf","green":"#04f623","purple":"#ce6fdb","red":"#bb0000","yellow":"#f3f167"}, {"name":"Nocturnal Winter", "foreground":"#e6e5e5","background":"#0d0d17","brightBlack":"#808080","brightWhite":"#ffffff","brightBlue":"#6096ff","brightCyan":"#0ae78d","brightGreen":"#0ae78d","brightPurple":"#ff78a2","brightRed":"#f16d86","brightYellow":"#fffc67","black":"#4d4d4d","white":"#fcfcfc","blue":"#3182e0","cyan":"#09c87a","green":"#09cd7e","purple":"#ff2b6d","red":"#f12d52","yellow":"#f5f17a"}, {"name":"Novel", "foreground":"#3b2322","background":"#dfdbc3","brightBlack":"#808080","brightWhite":"#ffffff","brightBlue":"#0000cc","brightCyan":"#0087cc","brightGreen":"#009600","brightPurple":"#cc00cc","brightRed":"#cc0000","brightYellow":"#d06b00","black":"#000000","white":"#cccccc","blue":"#0000cc","cyan":"#0087cc","green":"#009600","purple":"#cc00cc","red":"#cc0000","yellow":"#d06b00"}, {"name":"Obsidian", "foreground":"#cdcdcd","background":"#283033","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#a1d7ff","brightCyan":"#55ffff","brightGreen":"#93c863","brightPurple":"#ff55ff","brightRed":"#ff0003","brightYellow":"#fef874","black":"#000000","white":"#bbbbbb","blue":"#3a9bdb","cyan":"#00bbbb","green":"#00bb00","purple":"#bb00bb","red":"#a60001","yellow":"#fecd22"}, {"name":"Ocean", "foreground":"#ffffff","background":"#224fbc","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#00d900","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#000000","white":"#bfbfbf","blue":"#0000b2","cyan":"#00a6b2","green":"#00a600","purple":"#b200b2","red":"#990000","yellow":"#999900"}, {"name":"OceanicMaterial", "foreground":"#c2c8d7","background":"#1c262b","brightBlack":"#777777","brightWhite":"#ffffff","brightBlue":"#54a4f3","brightCyan":"#42c7da","brightGreen":"#70be71","brightPurple":"#aa4dbc","brightRed":"#dc5c60","brightYellow":"#fff163","black":"#000000","white":"#a4a4a4","blue":"#1e80f0","cyan":"#16afca","green":"#40a33f","purple":"#8800a0","red":"#ee2b2a","yellow":"#ffea2e"}, {"name":"Ollie", "foreground":"#8a8dae","background":"#222125","brightBlack":"#5b3725","brightWhite":"#5b6ea7","brightBlue":"#4488ff","brightCyan":"#1ffaff","brightGreen":"#3bff99","brightPurple":"#ffc21d","brightRed":"#ff3d48","brightYellow":"#ff5e1e","black":"#000000","white":"#8a8eac","blue":"#2d57ac","cyan":"#1fa6ac","green":"#31ac61","purple":"#b08528","red":"#ac2e31","yellow":"#ac4300"}, {"name":"OneHalfDark", "foreground":"#dcdfe4","background":"#282c34","brightBlack":"#282c34","brightWhite":"#dcdfe4","brightBlue":"#61afef","brightCyan":"#56b6c2","brightGreen":"#98c379","brightPurple":"#c678dd","brightRed":"#e06c75","brightYellow":"#e5c07b","black":"#282c34","white":"#dcdfe4","blue":"#61afef","cyan":"#56b6c2","green":"#98c379","purple":"#c678dd","red":"#e06c75","yellow":"#e5c07b"}, {"name":"OneHalfLight", "foreground":"#383a42","background":"#fafafa","brightBlack":"#4f525e","brightWhite":"#ffffff","brightBlue":"#61afef","brightCyan":"#56b6c2","brightGreen":"#98c379","brightPurple":"#c678dd","brightRed":"#e06c75","brightYellow":"#e5c07b","black":"#383a42","white":"#fafafa","blue":"#0184bc","cyan":"#0997b3","green":"#50a14f","purple":"#a626a4","red":"#e45649","yellow":"#c18401"}, {"name":"Operator Mono Dark", "foreground":"#c3cac2","background":"#191919","brightBlack":"#9a9b99","brightWhite":"#fdfdf6","brightBlue":"#89d3f6","brightCyan":"#82eada","brightGreen":"#83d0a2","brightPurple":"#ff2c7a","brightRed":"#c37d62","brightYellow":"#fdfdc5","black":"#5a5a5a","white":"#ced4cd","blue":"#4387cf","cyan":"#72d5c6","green":"#4d7b3a","purple":"#b86cb4","red":"#ca372d","yellow":"#d4d697"}, {"name":"Pandora", "foreground":"#e1e1e1","background":"#141e43","brightBlack":"#3f5648","brightWhite":"#ffffff","brightBlue":"#23d7d7","brightCyan":"#00ede1","brightGreen":"#74cd68","brightPurple":"#ff37ff","brightRed":"#ff3242","brightYellow":"#ffb929","black":"#000000","white":"#e2e2e2","blue":"#338f86","cyan":"#23d7d7","green":"#74af68","purple":"#9414e6","red":"#ff4242","yellow":"#ffad29"}, {"name":"Paraiso Dark", "foreground":"#a39e9b","background":"#2f1e2e","brightBlack":"#776e71","brightWhite":"#e7e9db","brightBlue":"#06b6ef","brightCyan":"#5bc4bf","brightGreen":"#48b685","brightPurple":"#815ba4","brightRed":"#ef6155","brightYellow":"#fec418","black":"#2f1e2e","white":"#a39e9b","blue":"#06b6ef","cyan":"#5bc4bf","green":"#48b685","purple":"#815ba4","red":"#ef6155","yellow":"#fec418"}, {"name":"Parasio Dark", "foreground":"#a39e9b","background":"#2f1e2e","brightBlack":"#776e71","brightWhite":"#e7e9db","brightBlue":"#06b6ef","brightCyan":"#5bc4bf","brightGreen":"#48b685","brightPurple":"#815ba4","brightRed":"#ef6155","brightYellow":"#fec418","black":"#2f1e2e","white":"#a39e9b","blue":"#06b6ef","cyan":"#5bc4bf","green":"#48b685","purple":"#815ba4","red":"#ef6155","yellow":"#fec418"}, {"name":"PaulMillr", "foreground":"#f2f2f2","background":"#000000","brightBlack":"#666666","brightWhite":"#ffffff","brightBlue":"#709aed","brightCyan":"#7adff2","brightGreen":"#66ff66","brightPurple":"#db67e6","brightRed":"#ff0080","brightYellow":"#f3d64e","black":"#2a2a2a","white":"#bbbbbb","blue":"#396bd7","cyan":"#66ccff","green":"#79ff0f","purple":"#b449be","red":"#ff0000","yellow":"#e7bf00"}, {"name":"PencilDark", "foreground":"#f1f1f1","background":"#212121","brightBlack":"#424242","brightWhite":"#f1f1f1","brightBlue":"#20bbfc","brightCyan":"#4fb8cc","brightGreen":"#5fd7af","brightPurple":"#6855de","brightRed":"#fb007a","brightYellow":"#f3e430","black":"#212121","white":"#d9d9d9","blue":"#008ec4","cyan":"#20a5ba","green":"#10a778","purple":"#523c79","red":"#c30771","yellow":"#a89c14"}, {"name":"PencilLight", "foreground":"#424242","background":"#f1f1f1","brightBlack":"#424242","brightWhite":"#f1f1f1","brightBlue":"#20bbfc","brightCyan":"#4fb8cc","brightGreen":"#5fd7af","brightPurple":"#6855de","brightRed":"#fb007a","brightYellow":"#f3e430","black":"#212121","white":"#d9d9d9","blue":"#008ec4","cyan":"#20a5ba","green":"#10a778","purple":"#523c79","red":"#c30771","yellow":"#a89c14"}, {"name":"Piatto Light", "foreground":"#414141","background":"#ffffff","brightBlack":"#3f3f3f","brightWhite":"#f2f2f2","brightBlue":"#3c5ea8","brightCyan":"#829429","brightGreen":"#829429","brightPurple":"#a454b2","brightRed":"#db3365","brightYellow":"#cd6f34","black":"#414141","white":"#ffffff","blue":"#3c5ea8","cyan":"#66781e","green":"#66781e","purple":"#a454b2","red":"#b23771","yellow":"#cd6f34"}, {"name":"Pnevma", "foreground":"#d0d0d0","background":"#1c1c1c","brightBlack":"#4a4845","brightWhite":"#efefef","brightBlue":"#a1bdce","brightCyan":"#b1e7dd","brightGreen":"#afbea2","brightPurple":"#d7beda","brightRed":"#d78787","brightYellow":"#e4c9af","black":"#2f2e2d","white":"#d0d0d0","blue":"#7fa5bd","cyan":"#8adbb4","green":"#90a57d","purple":"#c79ec4","red":"#a36666","yellow":"#d7af87"}, {"name":"primary", "foreground":"#000000","background":"#ffffff","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#4285f4","brightCyan":"#0f9d58","brightGreen":"#0f9d58","brightPurple":"#4285f4","brightRed":"#db4437","brightYellow":"#f4b400","black":"#000000","white":"#ffffff","blue":"#4285f4","cyan":"#4285f4","green":"#0f9d58","purple":"#db4437","red":"#db4437","yellow":"#f4b400"}, {"name":"Pro Light", "foreground":"#191919","background":"#ffffff","brightBlack":"#9f9f9f","brightWhite":"#f2f2f2","brightBlue":"#0082ff","brightCyan":"#61f7f8","brightGreen":"#61ef57","brightPurple":"#ff7eff","brightRed":"#ff6640","brightYellow":"#f2f156","black":"#000000","white":"#dcdcdc","blue":"#3b75ff","cyan":"#4ed2de","green":"#50d148","purple":"#ed66e8","red":"#e5492b","yellow":"#c6c440"}, {"name":"Pro", "foreground":"#f2f2f2","background":"#000000","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#00d900","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#000000","white":"#bfbfbf","blue":"#2009db","cyan":"#00a6b2","green":"#00a600","purple":"#b200b2","red":"#990000","yellow":"#999900"}, {"name":"Purple Rain", "foreground":"#fffbf6","background":"#21084a","brightBlack":"#565656","brightWhite":"#ffffff","brightBlue":"#00a6ff","brightCyan":"#74fdf3","brightGreen":"#b8e36e","brightPurple":"#ac7bf0","brightRed":"#ff4250","brightYellow":"#ffd852","black":"#000000","white":"#ffffff","blue":"#00a2fa","cyan":"#00deef","green":"#9be205","purple":"#815bb5","red":"#ff260e","yellow":"#ffc400"}, {"name":"purplepeter", "foreground":"#ece7fa","background":"#2a1a4a","brightBlack":"#100b23","brightWhite":"#b9aed3","brightBlue":"#79daed","brightCyan":"#a0a0d6","brightGreen":"#b4be8f","brightPurple":"#ba91d4","brightRed":"#f99f92","brightYellow":"#f2e9bf","black":"#0a0520","white":"#ffba81","blue":"#66d9ef","cyan":"#ba8cff","green":"#99b481","purple":"#e78fcd","red":"#ff796d","yellow":"#efdfac"}, {"name":"rebecca", "foreground":"#e8e6ed","background":"#292a44","brightBlack":"#666699","brightWhite":"#f4f2f9","brightBlue":"#69c0fa","brightCyan":"#8bfde1","brightGreen":"#01eac0","brightPurple":"#c17ff8","brightRed":"#ff92cd","brightYellow":"#fffca8","black":"#12131e","white":"#e4e3e9","blue":"#7aa5ff","cyan":"#56d3c2","green":"#04dbb5","purple":"#bf9cf9","red":"#dd7755","yellow":"#f2e7b7"}, {"name":"Red Alert", "foreground":"#ffffff","background":"#762423","brightBlack":"#262626","brightWhite":"#ffffff","brightBlue":"#65aaf1","brightCyan":"#b7dfdd","brightGreen":"#aff08c","brightPurple":"#ddb7df","brightRed":"#e02553","brightYellow":"#dfddb7","black":"#000000","white":"#d6d6d6","blue":"#489bee","cyan":"#6bbeb8","green":"#71be6b","purple":"#e979d7","red":"#d62e4e","yellow":"#beb86b"}, {"name":"Red Planet", "foreground":"#c2b790","background":"#222222","brightBlack":"#676767","brightWhite":"#d6bfb8","brightBlue":"#60827e","brightCyan":"#38add8","brightGreen":"#869985","brightPurple":"#de4974","brightRed":"#b55242","brightYellow":"#ebeb91","black":"#202020","white":"#b9aa99","blue":"#69819e","cyan":"#5b8390","green":"#728271","purple":"#896492","red":"#8c3432","yellow":"#e8bf6a"}, {"name":"Red Sands", "foreground":"#d7c9a7","background":"#7a251e","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#0072ae","brightCyan":"#55ffff","brightGreen":"#00bb00","brightPurple":"#ff55ff","brightRed":"#bb0000","brightYellow":"#e7b000","black":"#000000","white":"#bbbbbb","blue":"#0072ff","cyan":"#00bbbb","green":"#00bb00","purple":"#bb00bb","red":"#ff3f00","yellow":"#e7b000"}, {"name":"Relaxed", "foreground":"#d9d9d9","background":"#353a44","brightBlack":"#636363","brightWhite":"#f7f7f7","brightBlue":"#7eaac7","brightCyan":"#acbbd0","brightGreen":"#a0ac77","brightPurple":"#b06698","brightRed":"#bc5653","brightYellow":"#ebc17a","black":"#151515","white":"#d9d9d9","blue":"#6a8799","cyan":"#c9dfff","green":"#909d63","purple":"#b06698","red":"#bc5653","yellow":"#ebc17a"}, {"name":"Rippedcasts", "foreground":"#ffffff","background":"#2b2b2b","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#86bdc9","brightCyan":"#8c9bc4","brightGreen":"#bcee68","brightPurple":"#e500e5","brightRed":"#eecbad","brightYellow":"#e5e500","black":"#000000","white":"#bfbfbf","blue":"#75a5b0","cyan":"#5a647e","green":"#a8ff60","purple":"#ff73fd","red":"#cdaf95","yellow":"#bfbb1f"}, {"name":"Royal", "foreground":"#514968","background":"#100815","brightBlack":"#312d3d","brightWhite":"#9e8cbd","brightBlue":"#90baf9","brightCyan":"#acd4eb","brightGreen":"#2cd946","brightPurple":"#a479e3","brightRed":"#d5356c","brightYellow":"#fde83b","black":"#241f2b","white":"#524966","blue":"#6580b0","cyan":"#8aaabe","green":"#23801c","purple":"#674d96","red":"#91284c","yellow":"#b49d27"}, {"name":"Ryuuko", "foreground":"#ececec","background":"#2c3941","brightBlack":"#5d7079","brightWhite":"#ececec","brightBlue":"#6a8e95","brightCyan":"#88b2ac","brightGreen":"#66907d","brightPurple":"#b18a73","brightRed":"#865f5b","brightYellow":"#b1a990","black":"#2c3941","white":"#ececec","blue":"#6a8e95","cyan":"#88b2ac","green":"#66907d","purple":"#b18a73","red":"#865f5b","yellow":"#b1a990"}, {"name":"Scarlet Protocol", "foreground":"#e41951","background":"#1c153d","brightBlack":"#686868","brightWhite":"#ffffff","brightBlue":"#6871ff","brightCyan":"#60fdff","brightGreen":"#5ffa68","brightPurple":"#bd35ec","brightRed":"#ff6e67","brightYellow":"#fffc67","black":"#101116","white":"#c7c7c7","blue":"#0271b6","cyan":"#00c5c7","green":"#00dc84","purple":"#ca30c7","red":"#ff0051","yellow":"#faf945"}, {"name":"Seafoam Pastel", "foreground":"#d4e7d4","background":"#243435","brightBlack":"#8a8a8a","brightWhite":"#e0e0e0","brightBlue":"#7ac3cf","brightCyan":"#ade0e0","brightGreen":"#98d9aa","brightPurple":"#d6b2a1","brightRed":"#cf937a","brightYellow":"#fae79d","black":"#757575","white":"#e0e0e0","blue":"#4d7b82","cyan":"#729494","green":"#728c62","purple":"#8a7267","red":"#825d4d","yellow":"#ada16d"}, {"name":"SeaShells", "foreground":"#deb88d","background":"#09141b","brightBlack":"#434b53","brightWhite":"#fee4ce","brightBlue":"#1bbcdd","brightCyan":"#87acb4","brightGreen":"#628d98","brightPurple":"#bbe3ee","brightRed":"#d48678","brightYellow":"#fdd39f","black":"#17384c","white":"#deb88d","blue":"#1e4950","cyan":"#50a3b5","green":"#027c9b","purple":"#68d4f1","red":"#d15123","yellow":"#fca02f"}, {"name":"Seti", "foreground":"#cacecd","background":"#111213","brightBlack":"#323232","brightWhite":"#ffffff","brightBlue":"#43a5d5","brightCyan":"#8ec43d","brightGreen":"#8ec43d","brightPurple":"#8b57b5","brightRed":"#c22832","brightYellow":"#e0c64f","black":"#323232","white":"#eeeeee","blue":"#43a5d5","cyan":"#8ec43d","green":"#8ec43d","purple":"#8b57b5","red":"#c22832","yellow":"#e0c64f"}, {"name":"shades-of-purple", "foreground":"#ffffff","background":"#1e1d40","brightBlack":"#686868","brightWhite":"#ffffff","brightBlue":"#6871ff","brightCyan":"#79e8fb","brightGreen":"#43d426","brightPurple":"#ff77ff","brightRed":"#f92a1c","brightYellow":"#f1d000","black":"#000000","white":"#c7c7c7","blue":"#6943ff","cyan":"#00c5c7","green":"#3ad900","purple":"#ff2c70","red":"#d90429","yellow":"#ffe700"}, {"name":"Shaman", "foreground":"#405555","background":"#001015","brightBlack":"#384451","brightWhite":"#58fbd6","brightBlue":"#61d5ba","brightCyan":"#98d028","brightGreen":"#2aea5e","brightPurple":"#1298ff","brightRed":"#ff4242","brightYellow":"#8ed4fd","black":"#012026","white":"#405555","blue":"#449a86","cyan":"#5d7e19","green":"#00a941","purple":"#00599d","red":"#b2302d","yellow":"#5e8baa"}, {"name":"Slate", "foreground":"#35b1d2","background":"#222222","brightBlack":"#ffffff","brightWhite":"#e0e0e0","brightBlue":"#7ab0d2","brightCyan":"#8cdfe0","brightGreen":"#beffa8","brightPurple":"#c5a7d9","brightRed":"#ffcdd9","brightYellow":"#d0ccca","black":"#222222","white":"#02c5e0","blue":"#264b49","cyan":"#15ab9c","green":"#81d778","purple":"#a481d3","red":"#e2a8bf","yellow":"#c4c9c0"}, {"name":"SleepyHollow", "foreground":"#af9a91","background":"#121214","brightBlack":"#4e4b61","brightWhite":"#d2c7a9","brightBlue":"#8086ef","brightCyan":"#a4dce7","brightGreen":"#d6b04e","brightPurple":"#e2c2bb","brightRed":"#d9443f","brightYellow":"#f66813","black":"#572100","white":"#af9a91","blue":"#5f63b4","cyan":"#8faea9","green":"#91773f","purple":"#a17c7b","red":"#ba3934","yellow":"#b55600"}, {"name":"Smyck", "foreground":"#f7f7f7","background":"#1b1b1b","brightBlack":"#7a7a7a","brightWhite":"#f7f7f7","brightBlue":"#8dcff0","brightCyan":"#6ad9cf","brightGreen":"#c4f137","brightPurple":"#f79aff","brightRed":"#d6837c","brightYellow":"#fee14d","black":"#000000","white":"#a1a1a1","blue":"#62a3c4","cyan":"#207383","green":"#7da900","purple":"#ba8acc","red":"#b84131","yellow":"#c4a500"}, {"name":"Snazzy", "foreground":"#ebece6","background":"#1e1f29","brightBlack":"#555555","brightWhite":"#ededec","brightBlue":"#49baff","brightCyan":"#8be9fe","brightGreen":"#50fb7c","brightPurple":"#fc4cb4","brightRed":"#fc4346","brightYellow":"#f0fb8c","black":"#000000","white":"#ededec","blue":"#49baff","cyan":"#8be9fe","green":"#50fb7c","purple":"#fc4cb4","red":"#fc4346","yellow":"#f0fb8c"}, {"name":"SoftServer", "foreground":"#99a3a2","background":"#242626","brightBlack":"#666c6c","brightWhite":"#d2e0de","brightBlue":"#62b1df","brightCyan":"#64e39c","brightGreen":"#bfdf55","brightPurple":"#606edf","brightRed":"#dd5c60","brightYellow":"#deb360","black":"#000000","white":"#99a3a2","blue":"#6b8fa3","cyan":"#6ba58f","green":"#9aa56a","purple":"#6a71a3","red":"#a2686a","yellow":"#a3906a"}, {"name":"Solarized Darcula", "foreground":"#d2d8d9","background":"#3d3f41","brightBlack":"#25292a","brightWhite":"#d2d8d9","brightBlue":"#2075c7","brightCyan":"#15968d","brightGreen":"#629655","brightPurple":"#797fd4","brightRed":"#f24840","brightYellow":"#b68800","black":"#25292a","white":"#d2d8d9","blue":"#2075c7","cyan":"#15968d","green":"#629655","purple":"#797fd4","red":"#f24840","yellow":"#b68800"}, {"name":"Solarized Dark - Patched", "foreground":"#708284","background":"#001e27","brightBlack":"#475b62","brightWhite":"#fcf4dc","brightBlue":"#708284","brightCyan":"#819090","brightGreen":"#475b62","brightPurple":"#5956ba","brightRed":"#bd3613","brightYellow":"#536870","black":"#002831","white":"#eae3cb","blue":"#2176c7","cyan":"#259286","green":"#738a05","purple":"#c61c6f","red":"#d11c24","yellow":"#a57706"}, {"name":"Solarized Dark Higher Contrast", "foreground":"#9cc2c3","background":"#001e27","brightBlack":"#006488","brightWhite":"#fcf4dc","brightBlue":"#178ec8","brightCyan":"#00b39e","brightGreen":"#51ef84","brightPurple":"#e24d8e","brightRed":"#f5163b","brightYellow":"#b27e28","black":"#002831","white":"#eae3cb","blue":"#2176c7","cyan":"#259286","green":"#6cbe6c","purple":"#c61c6f","red":"#d11c24","yellow":"#a57706"}, {"name":"Spacedust", "foreground":"#ecf0c1","background":"#0a1e24","brightBlack":"#684c31","brightWhite":"#fefff1","brightBlue":"#67a0ce","brightCyan":"#83a7b4","brightGreen":"#aecab8","brightPurple":"#ff8a3a","brightRed":"#ff8a3a","brightYellow":"#ffc878","black":"#6e5346","white":"#f0f1ce","blue":"#0f548b","cyan":"#06afc7","green":"#5cab96","purple":"#e35b00","red":"#e35b00","yellow":"#e3cd7b"}, {"name":"SpaceGray Eighties Dull", "foreground":"#c9c6bc","background":"#222222","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5486c0","brightCyan":"#58c2c1","brightGreen":"#89e986","brightPurple":"#bf83c1","brightRed":"#ec5f67","brightYellow":"#fec254","black":"#15171c","white":"#b3b8c3","blue":"#7c8fa5","cyan":"#80cdcb","green":"#92b477","purple":"#a5789e","red":"#b24a56","yellow":"#c6735a"}, {"name":"SpaceGray Eighties", "foreground":"#bdbaae","background":"#222222","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#4d84d1","brightCyan":"#83e9e4","brightGreen":"#93d493","brightPurple":"#ff55ff","brightRed":"#ff6973","brightYellow":"#ffd256","black":"#15171c","white":"#efece7","blue":"#5486c0","cyan":"#57c2c1","green":"#81a764","purple":"#bf83c1","red":"#ec5f67","yellow":"#fec254"}, {"name":"SpaceGray", "foreground":"#b3b8c3","background":"#20242d","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#7d8fa4","brightCyan":"#85a7a5","brightGreen":"#87b379","brightPurple":"#a47996","brightRed":"#b04b57","brightYellow":"#e5c179","black":"#000000","white":"#b3b8c3","blue":"#7d8fa4","cyan":"#85a7a5","green":"#87b379","purple":"#a47996","red":"#b04b57","yellow":"#e5c179"}, {"name":"Spiderman", "foreground":"#e3e3e3","background":"#1b1d1e","brightBlack":"#505354","brightWhite":"#fffff9","brightBlue":"#1d50ff","brightCyan":"#6184ff","brightGreen":"#ff3338","brightPurple":"#747cff","brightRed":"#ff0325","brightYellow":"#fe3a35","black":"#1b1d1e","white":"#fffef6","blue":"#2c3fff","cyan":"#3256ff","green":"#e22928","purple":"#2435db","red":"#e60813","yellow":"#e24756"}, {"name":"Spring", "foreground":"#4d4d4c","background":"#ffffff","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#15a9fd","brightCyan":"#3e999f","brightGreen":"#1fc231","brightPurple":"#8959a8","brightRed":"#ff0021","brightYellow":"#d5b807","black":"#000000","white":"#ffffff","blue":"#1dd3ee","cyan":"#3e999f","green":"#1f8c3b","purple":"#8959a8","red":"#ff4d83","yellow":"#1fc95b"}, {"name":"Square", "foreground":"#acacab","background":"#1a1a1a","brightBlack":"#141414","brightWhite":"#e2e2e2","brightBlue":"#b6defb","brightCyan":"#d7d9fc","brightGreen":"#c3f786","brightPurple":"#ad7fa8","brightRed":"#f99286","brightYellow":"#fcfbcc","black":"#050505","white":"#f2f2f2","blue":"#a9cdeb","cyan":"#c9caec","green":"#b6377d","purple":"#75507b","red":"#e9897c","yellow":"#ecebbe"}, {"name":"Subliminal", "foreground":"#d4d4d4","background":"#282c35","brightBlack":"#7f7f7f","brightWhite":"#d4d4d4","brightBlue":"#6699cc","brightCyan":"#5fb3b3","brightGreen":"#a9cfa4","brightPurple":"#f1a5ab","brightRed":"#e15a60","brightYellow":"#ffe2a9","black":"#7f7f7f","white":"#d4d4d4","blue":"#6699cc","cyan":"#5fb3b3","green":"#a9cfa4","purple":"#f1a5ab","red":"#e15a60","yellow":"#ffe2a9"}, {"name":"Sundried", "foreground":"#c9c9c9","background":"#1a1818","brightBlack":"#4d4e48","brightWhite":"#ffffff","brightBlue":"#7999f7","brightCyan":"#fad484","brightGreen":"#128c21","brightPurple":"#fd8aa1","brightRed":"#aa000c","brightYellow":"#fc6a21","black":"#302b2a","white":"#c9c9c9","blue":"#485b98","cyan":"#9c814f","green":"#587744","purple":"#864651","red":"#a7463d","yellow":"#9d602a"}, {"name":"Symfonic", "foreground":"#ffffff","background":"#000000","brightBlack":"#1b1d21","brightWhite":"#ffffff","brightBlue":"#0084d4","brightCyan":"#ccccff","brightGreen":"#56db3a","brightPurple":"#b729d9","brightRed":"#dc322f","brightYellow":"#ff8400","black":"#000000","white":"#ffffff","blue":"#0084d4","cyan":"#ccccff","green":"#56db3a","purple":"#b729d9","red":"#dc322f","yellow":"#ff8400"}, {"name":"synthwave", "foreground":"#dad9c7","background":"#000000","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#2f9ded","brightCyan":"#19cde6","brightGreen":"#25c141","brightPurple":"#f97137","brightRed":"#f841a0","brightYellow":"#fdf454","black":"#000000","white":"#ffffff","blue":"#2186ec","cyan":"#12c3e2","green":"#1ebb2b","purple":"#f85a21","red":"#f6188f","yellow":"#fdf834"}, {"name":"Tango Adapted", "foreground":"#000000","background":"#ffffff","brightBlack":"#8f928b","brightWhite":"#f6f6f4","brightBlue":"#88c9ff","brightCyan":"#00feff","brightGreen":"#93ff00","brightPurple":"#e9a7e1","brightRed":"#ff0013","brightYellow":"#fff121","black":"#000000","white":"#e6ebe1","blue":"#00a2ff","cyan":"#00d0d6","green":"#59d600","purple":"#c17ecc","red":"#ff0000","yellow":"#f0cb00"}, {"name":"Tango Half Adapted", "foreground":"#000000","background":"#ffffff","brightBlack":"#797d76","brightWhite":"#f4f4f2","brightBlue":"#76bfff","brightCyan":"#00f6fa","brightGreen":"#8af600","brightPurple":"#d898d1","brightRed":"#ff0013","brightYellow":"#ffec00","black":"#000000","white":"#e0e5db","blue":"#008ef6","cyan":"#00bdc3","green":"#4cc300","purple":"#a96cb3","red":"#ff0000","yellow":"#e2c000"}, {"name":"Teerb", "foreground":"#d0d0d0","background":"#262626","brightBlack":"#1c1c1c","brightWhite":"#efefef","brightBlue":"#86aed6","brightCyan":"#b1e7dd","brightGreen":"#aed686","brightPurple":"#d6aed6","brightRed":"#d68686","brightYellow":"#e4c9af","black":"#1c1c1c","white":"#d0d0d0","blue":"#86aed6","cyan":"#8adbb4","green":"#aed686","purple":"#d6aed6","red":"#d68686","yellow":"#d7af87"}, {"name":"Terminal Basic", "foreground":"#000000","background":"#ffffff","brightBlack":"#666666","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00e5e5","brightGreen":"#00d900","brightPurple":"#e500e5","brightRed":"#e50000","brightYellow":"#e5e500","black":"#000000","white":"#bfbfbf","blue":"#0000b2","cyan":"#00a6b2","green":"#00a600","purple":"#b200b2","red":"#990000","yellow":"#999900"}, {"name":"Thayer Bright", "foreground":"#f8f8f8","background":"#1b1d1e","brightBlack":"#505354","brightWhite":"#f8f8f2","brightBlue":"#3f78ff","brightCyan":"#23cfd5","brightGreen":"#b6e354","brightPurple":"#9e6ffe","brightRed":"#ff5995","brightYellow":"#feed6c","black":"#1b1d1e","white":"#ccccc6","blue":"#2757d6","cyan":"#38c8b5","green":"#4df840","purple":"#8c54fe","red":"#f92672","yellow":"#f4fd22"}, {"name":"The Hulk", "foreground":"#b5b5b5","background":"#1b1d1e","brightBlack":"#505354","brightWhite":"#e5e6e1","brightBlue":"#506b95","brightCyan":"#4085a6","brightGreen":"#48ff77","brightPurple":"#72589d","brightRed":"#8dff2a","brightYellow":"#3afe16","black":"#1b1d1e","white":"#d9d8d1","blue":"#2525f5","cyan":"#378ca9","green":"#13ce30","purple":"#641f74","red":"#269d1b","yellow":"#63e457"}, {"name":"Tomorrow Night Blue", "foreground":"#ffffff","background":"#002451","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#bbdaff","brightCyan":"#99ffff","brightGreen":"#d1f1a9","brightPurple":"#ebbbff","brightRed":"#ff9da4","brightYellow":"#ffeead","black":"#000000","white":"#ffffff","blue":"#bbdaff","cyan":"#99ffff","green":"#d1f1a9","purple":"#ebbbff","red":"#ff9da4","yellow":"#ffeead"}, {"name":"Tomorrow Night Bright", "foreground":"#eaeaea","background":"#000000","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#7aa6da","brightCyan":"#70c0b1","brightGreen":"#b9ca4a","brightPurple":"#c397d8","brightRed":"#d54e53","brightYellow":"#e7c547","black":"#000000","white":"#ffffff","blue":"#7aa6da","cyan":"#70c0b1","green":"#b9ca4a","purple":"#c397d8","red":"#d54e53","yellow":"#e7c547"}, {"name":"Tomorrow Night Burns", "foreground":"#a1b0b8","background":"#151515","brightBlack":"#5d6f71","brightWhite":"#f5f5f5","brightBlue":"#fc595f","brightCyan":"#ba8586","brightGreen":"#a63c40","brightPurple":"#df9395","brightRed":"#832e31","brightYellow":"#d2494e","black":"#252525","white":"#f5f5f5","blue":"#fc595f","cyan":"#ba8586","green":"#a63c40","purple":"#df9395","red":"#832e31","yellow":"#d3494e"}, {"name":"Tomorrow Night Eighties", "foreground":"#cccccc","background":"#2d2d2d","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#6699cc","brightCyan":"#66cccc","brightGreen":"#99cc99","brightPurple":"#cc99cc","brightRed":"#f2777a","brightYellow":"#ffcc66","black":"#000000","white":"#ffffff","blue":"#6699cc","cyan":"#66cccc","green":"#99cc99","purple":"#cc99cc","red":"#f2777a","yellow":"#ffcc66"}, {"name":"Tomorrow Night", "foreground":"#c5c8c6","background":"#1d1f21","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#81a2be","brightCyan":"#8abeb7","brightGreen":"#b5bd68","brightPurple":"#b294bb","brightRed":"#cc6666","brightYellow":"#f0c674","black":"#000000","white":"#ffffff","blue":"#81a2be","cyan":"#8abeb7","green":"#b5bd68","purple":"#b294bb","red":"#cc6666","yellow":"#f0c674"}, {"name":"Tomorrow", "foreground":"#4d4d4c","background":"#ffffff","brightBlack":"#000000","brightWhite":"#ffffff","brightBlue":"#4271ae","brightCyan":"#3e999f","brightGreen":"#718c00","brightPurple":"#8959a8","brightRed":"#c82829","brightYellow":"#eab700","black":"#000000","white":"#ffffff","blue":"#4271ae","cyan":"#3e999f","green":"#718c00","purple":"#8959a8","red":"#c82829","yellow":"#eab700"}, {"name":"ToyChest", "foreground":"#31d07b","background":"#24364b","brightBlack":"#336889","brightWhite":"#d5d5d5","brightBlue":"#34a6da","brightCyan":"#42c3ae","brightGreen":"#31d07b","brightPurple":"#ae6bdc","brightRed":"#dd5944","brightYellow":"#e7d84b","black":"#2c3f58","white":"#23d183","blue":"#325d96","cyan":"#35a08f","green":"#1a9172","purple":"#8a5edc","red":"#be2d26","yellow":"#db8e27"}, {"name":"Treehouse", "foreground":"#786b53","background":"#191919","brightBlack":"#433626","brightWhite":"#ffc800","brightBlue":"#85cfed","brightCyan":"#f07d14","brightGreen":"#55f238","brightPurple":"#e14c5a","brightRed":"#ed5d20","brightYellow":"#f2b732","black":"#321300","white":"#786b53","blue":"#58859a","cyan":"#b25a1e","green":"#44a900","purple":"#97363d","red":"#b2270e","yellow":"#aa820c"}, {"name":"Twilight", "foreground":"#ffffd4","background":"#141414","brightBlack":"#262626","brightWhite":"#ffffd4","brightBlue":"#5a5e62","brightCyan":"#8a989b","brightGreen":"#ccd88c","brightPurple":"#d0dc8e","brightRed":"#de7c4c","brightYellow":"#e2c47e","black":"#141414","white":"#ffffd4","blue":"#44474a","cyan":"#778385","green":"#afb97a","purple":"#b4be7c","red":"#c06d44","yellow":"#c2a86c"}, {"name":"Ubuntu", "foreground":"#eeeeec","background":"#300a24","brightBlack":"#555753","brightWhite":"#eeeeec","brightBlue":"#729fcf","brightCyan":"#34e2e2","brightGreen":"#8ae234","brightPurple":"#ad7fa8","brightRed":"#ef2929","brightYellow":"#fce94f","black":"#2e3436","white":"#d3d7cf","blue":"#3465a4","cyan":"#06989a","green":"#4e9a06","purple":"#75507b","red":"#cc0000","yellow":"#c4a000"}, {"name":"UltraViolent", "foreground":"#c1c1c1","background":"#242728","brightBlack":"#636667","brightWhite":"#f9f9f5","brightBlue":"#7fecff","brightCyan":"#69fcd3","brightGreen":"#deff8c","brightPurple":"#e681ff","brightRed":"#fb58b4","brightYellow":"#ebe087","black":"#242728","white":"#e1e1e1","blue":"#47e0fb","cyan":"#0effbb","green":"#b6ff00","purple":"#d731ff","red":"#ff0090","yellow":"#fff727"}, {"name":"UnderTheSea", "foreground":"#ffffff","background":"#011116","brightBlack":"#384451","brightWhite":"#58fbd6","brightBlue":"#61d5ba","brightCyan":"#98d028","brightGreen":"#2aea5e","brightPurple":"#1298ff","brightRed":"#ff4242","brightYellow":"#8ed4fd","black":"#022026","white":"#405555","blue":"#459a86","cyan":"#5d7e19","green":"#00a941","purple":"#00599d","red":"#b2302d","yellow":"#59819c"}, {"name":"Unikitty", "foreground":"#0b0b0b","background":"#ff8cd9","brightBlack":"#434343","brightWhite":"#fff3fe","brightBlue":"#0075ea","brightCyan":"#79ecd5","brightGreen":"#d3ffaf","brightPurple":"#fdd5e5","brightRed":"#d91329","brightYellow":"#ffef50","black":"#0c0c0c","white":"#e2d7e1","blue":"#145fcd","cyan":"#6bd1bc","green":"#bafc8b","purple":"#ff36a2","red":"#a80f20","yellow":"#eedf4b"}, {"name":"Urple", "foreground":"#877a9b","background":"#1b1b23","brightBlack":"#5d3225","brightWhite":"#bfa3ff","brightBlue":"#867aed","brightCyan":"#eaeaea","brightGreen":"#29e620","brightPurple":"#a05eee","brightRed":"#ff6388","brightYellow":"#f08161","black":"#000000","white":"#87799c","blue":"#564d9b","cyan":"#808080","green":"#37a415","purple":"#6c3ca1","red":"#b0425b","yellow":"#ad5c42"}, {"name":"Vaughn", "foreground":"#dcdccc","background":"#25234f","brightBlack":"#709080","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#93e0e3","brightGreen":"#60b48a","brightPurple":"#ec93d3","brightRed":"#dca3a3","brightYellow":"#f0dfaf","black":"#25234f","white":"#709080","blue":"#5555ff","cyan":"#8cd0d3","green":"#60b48a","purple":"#f08cc3","red":"#705050","yellow":"#dfaf8f"}, {"name":"VibrantInk", "foreground":"#ffffff","background":"#000000","brightBlack":"#555555","brightWhite":"#e5e5e5","brightBlue":"#0000ff","brightCyan":"#00ffff","brightGreen":"#00ff00","brightPurple":"#ff00ff","brightRed":"#ff0000","brightYellow":"#ffff00","black":"#878787","white":"#f5f5f5","blue":"#44b4cc","cyan":"#44b4cc","green":"#ccff04","purple":"#9933cc","red":"#ff6600","yellow":"#ffcc00"}, {"name":"Violet Dark", "foreground":"#708284","background":"#1c1d1f","brightBlack":"#45484b","brightWhite":"#c9c6bd","brightBlue":"#2176c7","brightCyan":"#259286","brightGreen":"#738a04","brightPurple":"#c61c6f","brightRed":"#bd3613","brightYellow":"#a57705","black":"#56595c","white":"#c9c6bd","blue":"#2e8bce","cyan":"#32a198","green":"#85981c","purple":"#d13a82","red":"#c94c22","yellow":"#b4881d"}, {"name":"Violet Light", "foreground":"#536870","background":"#fcf4dc","brightBlack":"#45484b","brightWhite":"#c9c6bd","brightBlue":"#2176c7","brightCyan":"#259286","brightGreen":"#738a04","brightPurple":"#c61c6f","brightRed":"#bd3613","brightYellow":"#a57705","black":"#56595c","white":"#d3d0c9","blue":"#2e8bce","cyan":"#32a198","green":"#85981c","purple":"#d13a82","red":"#c94c22","yellow":"#b4881d"}, {"name":"WarmNeon", "foreground":"#afdab6","background":"#404040","brightBlack":"#fefcfc","brightWhite":"#d8c8bb","brightBlue":"#7b91d6","brightCyan":"#5ed1e5","brightGreen":"#9cc090","brightPurple":"#f674ba","brightRed":"#e97071","brightYellow":"#ddda7a","black":"#000000","white":"#d0b8a3","blue":"#4261c5","cyan":"#2abbd4","green":"#39b13a","purple":"#f920fb","red":"#e24346","yellow":"#dae145"}, {"name":"Wez", "foreground":"#b3b3b3","background":"#000000","brightBlack":"#555555","brightWhite":"#ffffff","brightBlue":"#5555ff","brightCyan":"#55ffff","brightGreen":"#55ff55","brightPurple":"#ff55ff","brightRed":"#ff5555","brightYellow":"#ffff55","black":"#000000","white":"#cccccc","blue":"#5555cc","cyan":"#7acaca","green":"#55cc55","purple":"#cc55cc","red":"#cc5555","yellow":"#cdcd55"}, {"name":"Whimsy", "foreground":"#b3b0d6","background":"#29283b","brightBlack":"#535178","brightWhite":"#ffffff","brightBlue":"#65aef7","brightCyan":"#43c1be","brightGreen":"#5eca89","brightPurple":"#aa7ff0","brightRed":"#ef6487","brightYellow":"#fdd877","black":"#535178","white":"#ffffff","blue":"#65aef7","cyan":"#43c1be","green":"#5eca89","purple":"#aa7ff0","red":"#ef6487","yellow":"#fdd877"}, {"name":"WildCherry", "foreground":"#dafaff","background":"#1f1726","brightBlack":"#009cc9","brightWhite":"#e4838d","brightBlue":"#308cba","brightCyan":"#ff919d","brightGreen":"#f4dca5","brightPurple":"#ae636b","brightRed":"#da6bac","brightYellow":"#eac066","black":"#000507","white":"#fff8de","blue":"#883cdc","cyan":"#c1b8b7","green":"#2ab250","purple":"#ececec","red":"#d94085","yellow":"#ffd16f"}, {"name":"Wombat", "foreground":"#dedacf","background":"#171717","brightBlack":"#313131","brightWhite":"#ffffff","brightBlue":"#a5c7ff","brightCyan":"#b7fff9","brightGreen":"#ddf88f","brightPurple":"#ddaaff","brightRed":"#f58c80","brightYellow":"#eee5b2","black":"#000000","white":"#dedacf","blue":"#5da9f6","cyan":"#82fff7","green":"#b1e969","purple":"#e86aff","red":"#ff615a","yellow":"#ebd99c"}, {"name":"Wryan", "foreground":"#999993","background":"#101010","brightBlack":"#3d3d3d","brightWhite":"#c0c0c0","brightBlue":"#477ab3","brightCyan":"#6096bf","brightGreen":"#53a6a6","brightPurple":"#7e62b3","brightRed":"#bf4d80","brightYellow":"#9e9ecb","black":"#333333","white":"#899ca1","blue":"#395573","cyan":"#31658c","green":"#287373","purple":"#5e468c","red":"#8c4665","yellow":"#7c7c99"}, {"name":"Zenburn", "foreground":"#dcdccc","background":"#3f3f3f","brightBlack":"#709080","brightWhite":"#ffffff","brightBlue":"#94bff3","brightCyan":"#93e0e3","brightGreen":"#c3bf9f","brightPurple":"#ec93d3","brightRed":"#dca3a3","brightYellow":"#e0cf9f","black":"#4d4d4d","white":"#dcdccc","blue":"#506070","cyan":"#8cd0d3","green":"#60b48a","purple":"#dc8cc3","red":"#705050","yellow":"#f0dfaf"} ] } ```` ## VS Code ### `keybindings.json` ```json // Place your key bindings in this file to override the defaults [ { "key": "f8", "command": "workbench.action.terminal.runSelectedText"} ] ``` ### `settings.json` ```json // Place your settings in this file to overwrite the default settings { "window.zoomLevel": 2, "workbench.colorTheme": "Visual Studio Dark", "workbench.startupEditor": "newUntitledFile", "window.restoreWindows": "all", "editor.fontFamily": "Delugia Nerd Font Book Regular", "editor.fontLigatures": true, "editor.insertSpaces": false, "editor.mouseWheelZoom": true, "editor.suggestSelection": "first", "editor.copyWithSyntaxHighlighting": false, "explorer.confirmDragAndDrop": false, // "http.proxy": "http://127.0.0.1:8888", "http.proxyStrictSSL": false, "rest-client.rememberCookiesForSubsequentRequests": false, "rest-client.defaultuseragent": "", "rest-client.followredirect": false, "rest-client.includeAdditionalInfoInResponse": true, "rest-client.enableTelemetry": false, "rest-client.excludeHostsForProxy": [ ], "terminal.integrated.fontFamily": "Delugia Nerd Font Book Regular", "terminal.integrated.shell.linux": "bash", "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\bash.exe", "terminal.integrated.rightClickBehavior": "paste", "elixir.autoSpawnElixirSenseServers": true, "elixir.useElixirSense": true, "elixirLS.dialyzerEnabled": true, "git.autofetch": true, "telemetry.enableTelemetry": false, "vsintellicode.modify.editor.suggestSelection": "automaticallyOverrodeDefaultValue", "workbench.colorCustomizations": { "window.activeBorder": "#ffffff", "window.inactiveBorder": "#000000" } } ``` # Time handling and Scheduling ## `time.is` Using [time.is](https://time.is/), it's relatively easy to craft links to points in time. For example, to understand which time *'09:00 AM'* in *Eastern Standard Time* would be relative to Germany, the UK, New York and Dallay, you can craft this link: ``` https://time.is/compare/0900_27_Jan_2020_in_EST/Germany/United_Kingdom/New_York/Dallas ``` As result, you get this view: ![time.is screenshot with multiple timezones](/files/7RUWtaeQCTNuYSFzTflw) # Elgato from the shell Once the lamp joined my local WLAN network (and I know the IP address), I use this script for controlling it. ## Requirements * You must have `cURL` and `jq` installed. (`sudo apt-get install jq curl`) * The script sets a local `ELGATO_KEY_LIGHT_IPADDRESS` variable. Tweak that part to your needs. ## Manual * `e` turns it on or off * Brightness * `e +` increases the brightness * `e -` decreases the brightness * Color tempreature * `e 1` selects the coldest color temperature * `e 2` selects a medium color temperature * `e 3` selects the warmest color temperature * Identify the light * `e i` makes the light blink three times ## Contents of `/usr/local/bin/e` ```bash #!/bin/bash function _elgatoGet { local ip="$1" local path="$2" curl --silent \ --url "http://${ip}:9123/elgato/${path}" } function _elgatoPut { local ip="$1" local path="$2" local msg="$3" echo "${msg}" | curl --silent --request PUT \ --header "Content-Type: application/json" \ --url "http://${ip}:9123/elgato/${path}" \ --data @- } function _elgatoIdentify { local ip="$1" local displayName="$( _elgatoGet "${ip}" "accessory-info" | jq -r ".displayName" )" echo "Light ${displayName} is blinking" # empty POST curl --silent \ --request POST\ --url "http://${ip}:9123/elgato/identify" } function _elgatoTurnOnOff { local ip="$1" local current="$( _elgatoGet "${ip}" "lights" | jq ".lights[0].on" )" local new="$(( 1-${current} ))" local state="$( _elgatoPut "${ip}" "lights" "{ \"lights\" : [ { \"on\" : ${new} } ] }" | jq ".lights[0].on" )" local displayName="$( _elgatoGet "${ip}" "accessory-info" | jq -r ".displayName" )" echo "$( if [[ "${state}" == "0" ]] ; then echo "Light \"${displayName}\" turned off" ; else echo "Light \"${displayName}\" turned on" ; fi ; )" } function _elgatoTemperature { local ip="$1" local temperature="$2" # temperature must be in range [143..344] if [[ ${temperature} -lt 143 ]] ; then temperature="143" ; elif [[ ${temperature} -gt 344 ]] ; then temperature="344" ; fi echo "Temperature: $( _elgatoPut "${ip}" lights \ "{\"lights\":[{ \"on\": 1 , \"temperature\": ${temperature} }]}" \ | jq '.lights[0].temperature' )" } function _elgatoBrightness { local ip="$1" local brightnessDelta="$2" local currentBrightness="$( _elgatoGet "${ip}" lights | jq ".lights[0].brightness" )" local newBrightness="$(( $currentBrightness + $brightnessDelta ))" # brightness must be in range [3..100] if [[ ${newBrightness} -lt 3 ]] ; then newBrightness="3" ; elif [[ ${newBrightness} -gt 100 ]] ; then newBrightness="100" ; fi echo "Brightness: $( _elgatoPut "${ip}" lights \ "{\"lights\":[{ \"on\": 1 , \"brightness\": ${newBrightness} }]}" \ | jq '.lights[0].brightness' )" } # ELGATO_KEY_LIGHT_IPADDRESS="192.168.0.106" ELGATO_KEY_LIGHT_IPADDRESS="elgatoip" if [[ $# -eq 0 ]] ; then echo "$( _elgatoTurnOnOff "${ELGATO_KEY_LIGHT_IPADDRESS}" )" elif [[ $# -eq 1 ]] ; then command=$1 if [[ "${command}" == "+" ]] ; then echo "$( _elgatoBrightness "${ELGATO_KEY_LIGHT_IPADDRESS}" 20 )" elif [[ "${command}" == "-" ]] ; then echo "$( _elgatoBrightness "${ELGATO_KEY_LIGHT_IPADDRESS}" -20 )" elif [[ "${command}" == "1" ]] ; then echo "$( _elgatoTemperature "${ELGATO_KEY_LIGHT_IPADDRESS}" 143 )" elif [[ "${command}" == "2" ]] ; then echo "$( _elgatoTemperature "${ELGATO_KEY_LIGHT_IPADDRESS}" 243 )" elif [[ "${command}" == "3" ]] ; then echo "$( _elgatoTemperature "${ELGATO_KEY_LIGHT_IPADDRESS}" 344 )" elif [[ "${command}" == "i" ]] ; then echo "$( _elgatoIdentify "${ELGATO_KEY_LIGHT_IPADDRESS}" )" fi fi ``` # Typora I'm a big fan of the [Typora](https://typora.io) markdown editor, as it has a nice WYSIWYG experience. They also support [custom image uploads](https://support.typora.io/Upload-Image/) to a storage backend of your choice. This sample here is a small uploader CLI for Azure bblob storage. Essentially, you need to set a `TYPORA_IMAGE_UPLOAD_AZURE_CONNECTION` environment variable, and ensure you have a container named `typoraimages`. ```csharp namespace TyporaUploaderAzure { using System; using System.IO; using System.Linq; using System.Security.Cryptography; using System.Threading.Tasks; using Azure.Storage.Blobs.Models; using SimpleBase; class Program { // https://support.typora.io/Upload-Image/ static async Task Main(string[] args) { var connectionString = Environment.GetEnvironmentVariable("TYPORA_IMAGE_UPLOAD_AZURE_CONNECTION"); var serviceClient = new Azure.Storage.Blobs.BlobServiceClient(connectionString: connectionString); var containerClient = serviceClient.GetBlobContainerClient(blobContainerName: "typoraimages"); // await containerClient.CreateIfNotExistsAsync(Azure.Storage.Blobs.Models.PublicAccessType.Blob); var prefix = DateTime.Now.ToString("yyyy/MM/dd/HH/mm"); var tasks = args.Select(async filename => { var fi = new FileInfo(filename); var bytes = await File.ReadAllBytesAsync(path: fi.FullName); using var hashAlgo = MD5.Create(); using var reader = fi.OpenRead(); var hash = hashAlgo.ComputeHash(bytes); var hashBase32 = Base32.Crockford.Encode(hash); var fileWithoutExtension = fi.Name.Substring(0, fi.Name.Length - fi.Extension.Length); var blobName = $"{prefix}/{fileWithoutExtension}----{hashBase32}{fi.Extension}"; var blobClient = containerClient.GetBlobClient(blobName); string mimeType(string extension) => extension switch { ".png" => "image/png", ".jpeg" => "image/jpeg", ".jpg" => "image/jpeg", _ => "application/octet-stream", }; if (!await blobClient.ExistsAsync()) { using var ms = new MemoryStream(bytes); await blobClient.UploadAsync(ms); var headers = new BlobHttpHeaders { ContentType = mimeType(fi.Extension), ContentHash = hash, CacheControl = "max-age=31536000", }; await blobClient.SetHttpHeadersAsync(headers); } return blobClient.Uri.AbsoluteUri; }); await Task.WhenAll(tasks); tasks .Select(t => t.Result) .ToList() .ForEach(Console.WriteLine); } } } ``` # BEAM Live Introspection for AI Coding Agents **TL;DR** — Give your AI coding agent (GitHub Copilot, Claude Code, OpenAI Codex, Gemini CLI) a reusable "skill" that lets it start, connect to, and introspect a running Elixir/BEAM node. Instead of guessing at runtime behavior or writing throwaway test scripts, the agent can query GenServer state, inspect supervision trees, poke ETS tables, and hot-reload code — all through a single shell script. This article is self-contained: point your coding agent at it and say *"Adopt this pattern for my project."* *** ## Table of Contents 1. [The Problem](#the-problem) 2. [The Pattern](#the-pattern) 3. [Step 1: Enable Your Project for Introspection](#step-1-enable-your-project-for-introspection) 4. [Step 2: Add the `dev_node.sh` Script](#step-2-add-the-dev_nodesh-script) 5. [Step 3: Create the Skill Definition](#step-3-create-the-skill-definition) 6. [Step 4: Register the Skill with Your Agent](#step-4-register-the-skill-with-your-agent) 7. [Usage Examples](#usage-examples) 8. [Reference: Agent Configuration Paths](#reference-agent-configuration-paths) *** ## The Problem When an AI coding agent works on an Elixir project, it typically has two options for validating its changes: run the tests (`mix test`) or reason about the code statically. Neither lets it *observe a live system* — check whether a GenServer has the right state, whether a supervision tree recovered from a crash, or whether a message actually arrived. The BEAM VM has world-class introspection built in. Every Erlang/Elixir node can be connected to from another node using distributed Erlang. The trick is teaching the coding agent how to use this. ### Why this matters: "The Soul of Erlang and Elixir" In his talk [*"The Soul of Erlang and Elixir"*](https://www.youtube.com/watch?v=JvBT4XBdoUE), Saša Jurić demonstrates exactly this capability against a live system. He SSHs into a running server, opens a remote console, and without restarting anything, drills into the problem: > *"BEAM is a runtime which is highly debuggable, introspectable, observable if you will. BEAM allows us to hook into the running system and peek and poke inside it and get a lot of useful information — and I don't need to set some special flags, restart the system and whatnot. I can do this by default."* — Saša Jurić, [20:43](https://www.youtube.com/watch?v=JvBT4XBdoUE\&t=1243) From the remote shell, he lists all processes, identifies the CPU-hogging one by its reduction count, gets its stack trace, traces its function calls, kills it with `Process.exit(pid, :kill)` — and the rest of the system keeps running at 10K requests/second, undisturbed. Then he hot-deploys a fix into the running production node without a restart. > *"I was able to approach the system and look from inside it to figure out what the problems are, quickly fix those problems, and deploy into production without disturbing anything in the system itself. This is what I want from my tool."* — Saša Jurić, [29:27](https://www.youtube.com/watch?v=JvBT4XBdoUE\&t=1767) This is exactly what we're giving to AI coding agents: the same remote-shell-into-a-live-system capability that Saša demonstrates manually, but wrapped in a scriptable interface (`dev_node.sh rpc`) that a coding agent can invoke without needing an interactive TTY. The agent becomes the operator, SSHing into the running BEAM. ## The Pattern Three pieces work together: 1. **The project launches with a known node name and cookie** — so the agent's helper script can connect. 2. **A `dev_node.sh` script** in the project provides `start`, `stop`, `status`, `await`, `rpc`, and `eval_file` commands. 3. **A skill definition** tells the coding agent *when* and *how* to use live introspection. ![BEAM Live Introspection Pattern](/files/c9HKUiN5JaE0wJv0HpLz) The RPC node is started as a **hidden node** (`--hidden` flag). In distributed Erlang, hidden nodes do not participate in the global cluster mesh — they don't trigger transitive connections, don't appear in `nodes()`, and are invisible to `:global` process registration. This is exactly what we want: the introspection node should observe the system without joining it as a peer or causing the cluster to attempt scheduling work on it. *** ## Step 1: Enable Your Project for Introspection Your application must start with a **short name** (`--sname`) and a **cookie** (`--cookie`). The simplest approach is a `run` script in the project root: ### Create `run` ```bash #!/bin/bash cd "$(dirname "$0")" || exit 1 SNAME="$(basename "$(pwd)")" export ELIXIR_ERL_OPTIONS="-sname $SNAME -setcookie devcookie" exec mix phx.server > run.log 2>&1 ``` ```bash chmod +x run ``` > **How `--sname` works**: `--sname my_app` registers the node as `my_app@`. The (secret) cookie must match on both sides for distributed Erlang to connect. Using the project directory name as the `sname` is a convention that the `dev_node.sh` script mirrors — so everything just works without configuration. ### For non-Phoenix projects If you don't use Phoenix, replace `mix phx.server` with `mix run --no-halt`: ```bash export ELIXIR_ERL_OPTIONS="-sname $SNAME -setcookie devcookie" exec mix run --no-halt > run.log 2>&1 ``` ### For production / releases When running a Mix release, use the `--sname` and `--cookie` flags in your release config or `rel/env.sh.eex`: ```bash export RELEASE_NODE="my_app" export RELEASE_COOKIE="devcookie" export RELEASE_DISTRIBUTION="sname" ``` > ⚠️ Use a stronger cookie in production. `devcookie` is for local development only. The Erlang cookie is security-critical. It is literally the only thing standing between you and an attacker getting full access onto your cluster. *** ## Step 2: Add the `dev_node.sh` Script Create `scripts/dev_node.sh` in your project. This is the single entry point for all BEAM introspection: ```bash #!/usr/bin/env bash set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" PROJECT_DIR="$(dirname "$SCRIPT_DIR")" APP_NAME="${DEV_NODE_NAME:-$(basename "$PROJECT_DIR")}" COOKIE="${DEV_NODE_COOKIE:-devcookie}" HOSTNAME="$(hostname -s)" FQDN="${APP_NAME}@${HOSTNAME}" PIDFILE=".dev_node.pid" case "${1:-help}" in start) if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then echo "Node already running (pid $(cat "$PIDFILE"))" exit 0 fi echo "Starting node ${FQDN} ..." export ELIXIR_ERL_OPTIONS="-sname $APP_NAME -setcookie $COOKIE" mix run --no-halt > .dev_node.log 2>&1 & echo $! > "$PIDFILE" for i in $(seq 1 30); do if elixir --sname "probe_$$" --cookie "$COOKIE" --hidden -e " if Node.connect(:\"${FQDN}\"), do: System.stop(0), else: System.stop(1) " 2>/dev/null; then echo "Node ${FQDN} is up (pid $(cat "$PIDFILE"))" exit 0 fi sleep 1 done echo "ERROR: Node did not become reachable within 30s. Check .dev_node.log" exit 1 ;; stop) if [ -f "$PIDFILE" ]; then kill "$(cat "$PIDFILE")" 2>/dev/null && echo "Node stopped" || echo "Node was not running" rm -f "$PIDFILE" else echo "No pidfile found" fi ;; status) if epmd -names 2>/dev/null | grep -q "name ${APP_NAME} "; then echo "Node ${FQDN} is running" exit 0 else echo "Node ${FQDN} is not running" exit 1 fi ;; await) TIMEOUT="${2:-30}" echo "Waiting for node ${FQDN} ..." for i in $(seq 1 "$TIMEOUT"); do if elixir --sname "probe_$$" --cookie "$COOKIE" --hidden -e " if Node.connect(:\"${FQDN}\"), do: System.stop(0), else: System.stop(1) " 2>/dev/null; then echo "Node ${FQDN} is reachable" exit 0 fi sleep 1 done echo "ERROR: Node ${FQDN} did not become reachable within ${TIMEOUT}s" exit 1 ;; rpc) shift EXPR="$*" elixir --sname "rpc_$$" --cookie "$COOKIE" --hidden --no-halt -e " target = :\"${FQDN}\" true = Node.connect(target) {result, _binding} = :rpc.call(target, Code, :eval_string, [\"\"\" ${EXPR} \"\"\"]) IO.inspect(result, pretty: true, limit: 200, printable_limit: 4096) System.stop(0) " ;; eval_file) shift FILE="$1" elixir --sname "rpc_$$" --cookie "$COOKIE" --hidden --no-halt -e " target = :\"${FQDN}\" true = Node.connect(target) code = File.read!(\"${FILE}\") {result, _binding} = :rpc.call(target, Code, :eval_string, [code]) IO.inspect(result, pretty: true, limit: 200, printable_limit: 4096) System.stop(0) " ;; help|*) echo "Usage: scripts/dev_node.sh {start|stop|status|await [timeout]|rpc |eval_file }" echo "" echo "Commands:" echo " start - Start a standalone BEAM node" echo " stop - Kill the node process" echo " status - Check if node is registered with epmd (exit 0/1)" echo " await [secs] - Wait for node to be connectable via distributed Erlang (default: 30s)" echo " rpc - Execute an Elixir expression on the remote node" echo " eval_file - Evaluate a file on the remote node" echo "" echo "Environment variables:" echo " DEV_NODE_NAME - sname for the node (default: project directory name)" echo " DEV_NODE_COOKIE - cluster cookie (default: devcookie)" ;; esac ``` ```bash mkdir -p scripts chmod +x scripts/dev_node.sh ``` ### How `dev_node.sh` works | Command | What it does | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `start` | Launches `mix run --no-halt` as a background BEAM node, waits until it's connectable | | `stop` | Kills the background node via its PID file | | `status` | Checks if the node is registered with `epmd` — works regardless of how the node was started. Exits 0 (running) or 1 (not running) | | `await [secs]` | Waits for the node to become connectable via distributed Erlang (RPC probe). Useful when the node was started externally (e.g. via `just start-bg`). Default timeout: 30s | | `rpc ` | Spawns a *short-lived hidden* BEAM node, connects to the app node, evaluates `` via `:rpc.call`, prints the result, and exits | | `eval_file ` | Same as `rpc`, but reads the expression from a `.exs` file — useful for complex multi-line introspection | > **Key design decision**: Each `rpc` call is stateless. A fresh hidden BEAM node connects, runs one expression, and exits. This avoids stale connections but means bindings don't carry across calls. The `--hidden` flag ensures the RPC node doesn't join the cluster as a peer — it won't appear in `nodes()`, won't trigger transitive connections to other cluster members, and the BEAM scheduler won't try to distribute work to it. ### Add a `shutdown.sh` for graceful stop ```bash #!/usr/bin/env bash "$(cd "$(dirname "$0")" && pwd)/dev_node.sh" rpc "System.stop()" ``` This tells the running node to shut down through the BEAM's own `System.stop()`, which triggers application shutdown callbacks. *** ## Step 3: Create the Skill Definition A "skill" is a markdown file (`SKILL.md`) with a YAML front-matter header and instructions for the coding agent. It lives alongside its helper scripts. Create a directory structure: ``` beam-introspection/ ├── SKILL.md └── scripts/ └── dev_node.sh (symlink or copy) ``` ### `SKILL.md` ````markdown --- name: beam-introspection description: > Start, connect to, and introspect a running BEAM/Elixir node. Use when asked to test, debug, validate, or observe runtime behavior of an Elixir application, inspect GenServer state, supervision trees, ETS tables, process mailboxes, or hot-reload code into a live system. Use instead of writing one-off test scripts. --- # BEAM Live Introspection Skill ## Purpose This skill enables you to start, connect to, introspect, and control a running BEAM/Elixir node instead of writing one-off scripts. Use it whenever you need to validate behavior, debug state, or test functionality against a live system. ## When to Use This Skill Use live introspection instead of writing standalone scripts when: - You need to observe GenServer state, supervision trees, or process behavior - You want to test a sequence of interactions against a running system - You need to debug why something isn't working by inspecting the live process tree - You want to validate that a code change works by hot-reloading into a running node Do NOT use this skill when: - You just need to run unit or integration tests (`mix test`) - You need to compile-check code (`mix compile --warnings-as-errors`) - The task is pure code generation with no runtime validation needed ## Setup Before first use, ensure the project has `scripts/dev_node.sh`. If it does not exist, create it from the template in this skill's `scripts/` directory, then `chmod +x scripts/dev_node.sh`. ## Configuration The script auto-detects the project name from the directory. Override with environment variables: ```bash export DEV_NODE_NAME=my_app # sname for the node export DEV_NODE_COOKIE=devcookie # cluster cookie ``` ## Workflow ### Step 1: Start the node ```bash scripts/dev_node.sh start ``` Wait for the "is up" confirmation before proceeding. ### Step 2: Introspect via RPC ```bash # Check supervision tree scripts/dev_node.sh rpc "Supervisor.which_children(MyApp.Supervisor)" # Get GenServer state scripts/dev_node.sh rpc ":sys.get_state(GenServer.whereis(MyApp.SomeServer))" # Count processes scripts/dev_node.sh rpc "length(Process.list())" # Inspect an ETS table scripts/dev_node.sh rpc ":ets.tab2list(:my_table) |> Enum.take(5)" # Call application functions directly scripts/dev_node.sh rpc "MyApp.some_function(\"arg\")" ``` ### Step 3: For complex introspection, use eval_file Write a `.exs` file and evaluate it on the live node: ```bash scripts/dev_node.sh eval_file scripts/check_state.exs ``` ### Step 4: Hot-reload code changes After modifying source code: ```bash mix compile scripts/dev_node.sh rpc "IEx.Helpers.recompile()" ``` ### Step 5: Stop the node ```bash scripts/dev_node.sh stop ``` ## Common Recipes ### Process tree overview ```bash scripts/dev_node.sh rpc " Process.list() |> Enum.map(fn pid -> info = Process.info(pid, [:registered_name, :message_queue_len, :memory]) {Keyword.get(info, :registered_name), Keyword.get(info, :message_queue_len), Keyword.get(info, :memory)} end) |> Enum.filter(fn {name, _, _} -> name != [] end) |> Enum.sort_by(fn {_, _, mem} -> mem end, :desc) |> Enum.take(15) " ``` ### Find processes with full mailboxes ```bash scripts/dev_node.sh rpc " Process.list() |> Enum.map(fn pid -> {pid, Process.info(pid, :message_queue_len)} end) |> Enum.filter(fn {_, {:message_queue_len, n}} -> n > 0 end) |> Enum.sort_by(fn {_, {:message_queue_len, n}} -> n end, :desc) |> Enum.take(10) |> Enum.map(fn {pid, {:message_queue_len, n}} -> info = Process.info(pid, [:registered_name, :current_function]) {pid, n, info} end) " ``` ### Memory overview ```bash scripts/dev_node.sh rpc ":erlang.memory() |> Enum.map(fn {k, v} -> {k, Float.round(v / 1_048_576, 2)} end)" ``` ### Application environment ```bash scripts/dev_node.sh rpc "Application.get_all_env(:my_app)" ``` ## Important Notes - Each `rpc` invocation is stateless. A fresh hidden BEAM node connects, runs, and exits. - The RPC node uses `--hidden` so it doesn't join the cluster mesh or appear in `nodes()`. - Expressions must be valid Elixir. Use `eval_file` for complex expressions with quotes or multi-line logic. - The `limit: 200` in `IO.inspect` truncates large data structures. - If a call hangs, the target node may be stuck. Check `.dev_node.log`. ```` *** ## Step 4: Register the Skill with Your Agent Each coding agent looks for skills in a different location. The skill directory structure is the same everywhere — only the parent path changes. ### GitHub Copilot (Copilot CLI / Copilot Coding Agent) **User-level skills** (available to all projects): ``` ~/.github/skills/beam-introspection/ ├── SKILL.md └── scripts/ └── dev_node.sh ``` **Project-level instructions** — add to `.github/copilot-instructions.md` or `AGENTS.md` at the project root: ```markdown ### BEAM introspection Use the `beam-introspection` skill to connect to the running BEAM node for runtime validation. The node runs with: - sname: `` (derived from directory name) - cookie: `devcookie` Use `scripts/dev_node.sh rpc ''` for one-shot introspection and `scripts/dev_node.sh eval_file ` for multi-line scripts. ``` ### Claude Code **User-level skills** (available to all projects): ``` ~/.claude-/skills/beam-introspection/ ├── SKILL.md └── scripts/ └── dev_node.sh ``` Where `` is your Claude profile identifier (e.g., `claude-user@example.com`). **Project-level instructions** — add a `CLAUDE.md` file at the project root referencing the skill, or add the instructions to your existing `CLAUDE.md`: ```markdown ## BEAM introspection When asked to test, debug, validate, or observe runtime behavior, use the `beam-introspection` skill to connect to the live BEAM node. Node configuration: - sname: derived from project directory name - cookie: devcookie Key commands: - `scripts/dev_node.sh start` — start a background node - `scripts/dev_node.sh rpc ''` — evaluate on the live node - `scripts/dev_node.sh eval_file ` — evaluate a script file on the live node - `scripts/dev_node.sh stop` — stop the background node ``` ### OpenAI Codex **User-level skills**: ``` ~/.codex/skills/beam-introspection/ ├── SKILL.md └── scripts/ └── dev_node.sh ``` **Project-level instructions** — Codex reads `AGENTS.md` at the project root. Add the same introspection section as shown for Copilot above. ### Gemini CLI **User-level skills**: ``` ~/.gemini/skills/beam-introspection/ ├── SKILL.md └── scripts/ └── dev_node.sh ``` **Project-level instructions** — Gemini reads `GEMINI.md` or `AGENTS.md` at the project root. Add the introspection instructions there. ### Generic / Multi-Agent (`~/.agents/`) Some agent frameworks check `~/.agents/` as a shared skills directory: ``` ~/.agents/skills/beam-introspection/ ├── SKILL.md └── scripts/ └── dev_node.sh ``` ### Quick setup script To install the skill for **all** agents at once, run this from your project root: ```bash #!/usr/bin/env bash set -euo pipefail SKILL_NAME="beam-introspection" SKILL_SOURCE="$(cd "$(dirname "$0")" && pwd)/scripts/dev_node.sh" # All known agent skill directories AGENT_DIRS=( "$HOME/.github/skills" "$HOME/.agents/skills" "$HOME/.codex/skills" "$HOME/.gemini/skills" ) # Claude uses a profile-specific directory — find it for d in "$HOME"/.claude-*/; do [ -d "$d" ] && AGENT_DIRS+=("${d}skills") done for dir in "${AGENT_DIRS[@]}"; do target="$dir/$SKILL_NAME" mkdir -p "$target/scripts" # Copy SKILL.md (from this article or from an existing install) if [ -f "$target/SKILL.md" ]; then echo "SKILL.md already exists at $target, skipping" else echo "Creating $target/SKILL.md — paste the SKILL.md content from this article" fi # Copy dev_node.sh if [ -f "$SKILL_SOURCE" ]; then cp "$SKILL_SOURCE" "$target/scripts/dev_node.sh" chmod +x "$target/scripts/dev_node.sh" echo "Installed dev_node.sh to $target/scripts/" fi done echo "Done. Skill '$SKILL_NAME' registered for all agents." ``` *** ## Usage Examples Once the skill is installed, here's what a typical agent interaction looks like: ### "Is my GenServer running?" You ask: *"Check if the OrderProcessor GenServer is alive and what its state looks like."* The agent runs: ```bash scripts/dev_node.sh rpc " case GenServer.whereis(MyApp.OrderProcessor) do nil -> :not_running pid -> {:running, pid, :sys.get_state(pid)} end " ``` ### "Why is the queue backed up?" You ask: *"Something's wrong with message processing — debug it."* The agent runs: ```bash scripts/dev_node.sh rpc " Process.list() |> Enum.map(fn pid -> {pid, Process.info(pid, [:registered_name, :message_queue_len])} end) |> Enum.reject(fn {_, info} -> Keyword.get(info, :message_queue_len) == 0 end) |> Enum.sort_by(fn {_, info} -> Keyword.get(info, :message_queue_len) end, :desc) |> Enum.take(5) " ``` ### "Hot-reload my fix and test it" You ask: *"I changed the retry logic — reload it into the running node and test."* The agent runs: ```bash mix compile scripts/dev_node.sh rpc "IEx.Helpers.recompile()" scripts/dev_node.sh rpc "MyApp.OrderProcessor.retry_pending()" ``` *** ## Reference: Agent Configuration Paths | Agent | User-level skill path | Project instructions file | | ------------------ | ------------------------------------ | ------------------------------------------------ | | **GitHub Copilot** | `~/.github/skills//` | `.github/copilot-instructions.md` or `AGENTS.md` | | **Claude Code** | `~/.claude-/skills//` | `CLAUDE.md` | | **OpenAI Codex** | `~/.codex/skills//` | `AGENTS.md` | | **Gemini CLI** | `~/.gemini/skills//` | `GEMINI.md` or `AGENTS.md` | | **Generic** | `~/.agents/skills//` | `AGENTS.md` | ### What goes where * **`SKILL.md`** — The skill definition with YAML front-matter (`name`, `description`) and instructions. This is what the agent reads to understand *when* and *how* to use the skill. * **`scripts/dev_node.sh`** — The helper script that handles node lifecycle and RPC. Can be a copy in each agent's skills dir, or a symlink to the project's `scripts/dev_node.sh`. * **Project instructions file** (`AGENTS.md`, `CLAUDE.md`, etc.) — Tells the agent that introspection is available for *this specific project*, including the node name and cookie. ### Minimum viable setup If you want the simplest possible setup for a single agent (e.g., GitHub Copilot): 1. Add `scripts/dev_node.sh` to your project (chmod +x) 2. Ensure your app starts with `--sname` and `--cookie devcookie` 3. Add this to `AGENTS.md`: ```markdown ## BEAM introspection Use `scripts/dev_node.sh` to introspect the running BEAM node: - `scripts/dev_node.sh start` — start the node - `scripts/dev_node.sh rpc ''` — evaluate on the live node - `scripts/dev_node.sh eval_file ` — evaluate a script file - `scripts/dev_node.sh stop` — stop the node Node name: derived from project directory name. Cookie: `devcookie`. When asked to debug, validate, or observe runtime behavior, prefer live introspection over writing throwaway scripts. ``` That's it. No skill registration needed — the agent reads `AGENTS.md` and knows how to use the script. *** ## How It Works Under the Hood When `dev_node.sh rpc` runs, it: 1. Starts a **new, short-lived hidden BEAM node** with a unique sname (`rpc_`), the same cookie, and the `--hidden` flag. 2. Calls `Node.connect/1` to connect to the target app node via distributed Erlang. Because the RPC node is hidden, this connection is **not transitive** — the app node won't try to mesh with it, and it won't appear in `nodes()` on the app side (only in `nodes(:hidden)`). 3. Uses `:rpc.call/4` to execute `Code.eval_string/1` on the target node — so the expression runs in the app's process context with access to all its modules and state. 4. Prints the result with `IO.inspect/2` and exits. This is the same mechanism that `iex --remsh` uses, but wrapped in a scriptable interface that coding agents can invoke without interactive TTY support. ### Security considerations * The cookie `devcookie` is well-known. Anyone on the same machine (or network, if using `--name` instead of `--sname`) can connect. Use only for local development. * `Code.eval_string/1` can execute arbitrary code. This is by design — the agent needs full access — but be aware of it in shared environments. * `--sname` restricts connections to the same hostname. `--name` would allow cross-host connections (not recommended without TLS distribution). # Stable Port Assignments for Any Dev Server (AI Agent-Friendly) **TL;DR** — Give your AI coding agent (GitHub Copilot, Claude Code, OpenAI Codex, Gemini CLI) a `justfile` that launches *any* local development server — Phoenix, Node.js, Python, Go, Ruby, or anything else that listens on a port — with stable, collision-free port assignments using [`phx-port`](https://github.com/chgeuer/phx-port). The agent can start, stop, and open your app without guessing ports or stepping on other running projects. > Despite the name, `phx-port` is **not** Phoenix-specific. It works with any project that needs a local port. This article is self-contained: point your coding agent at it and say *"Adopt the pattern in `https://cookbook.geuer-pollmann.de/elixir-beam/phx-port-and-justfile-for-ai-agents.md` for my project."* *** ## Table of Contents 1. [The Problem](#the-problem) 2. [The Pattern](#the-pattern) 3. [Step 1: Install `phx-port`](#step-1-install-phx-port) 4. [Step 2: Add the `justfile`](#step-2-add-the-justfile) 5. [Step 3: Update `.gitignore`](#step-3-update-gitignore) 6. [Step 4: Add Project Instructions for Your Agent](#step-4-add-project-instructions-for-your-agent) 7. [How It All Fits Together](#how-it-all-fits-together) 8. [Elixir / BEAM Bonus: Combining with BEAM Introspection](#elixir--beam-bonus-combining-with-beam-introspection) *** ## The Problem When you work on multiple web projects, they tend to default to the same port — 4000 for Phoenix, 3000 for Node/Rails, 8000 for Django, 8080 for Go. You end up either: * Killing the old server before starting the new one * Manually remembering which port you assigned to which project * Passing `PORT=4007` and hoping you haven't already used 4007 somewhere else AI coding agents make this worse. When an agent needs to start your dev server to validate a change, it doesn't know which port to use. If another project is already running on that port, the server fails to bind and the agent wastes time debugging a port conflict instead of doing real work. **What we want**: every project gets a stable, unique port — automatically — and the agent has a single command to start the server, open the browser, or stop the process. ## The Pattern Two pieces work together: 1. [**`phx-port`**](https://github.com/chgeuer/phx-port) — a small Rust CLI that maintains a TOML registry (`~/.config/phx-ports.toml`) mapping project directories to port numbers. Each project gets a unique port, allocated once and reused forever. Despite its name, it is **framework-agnostic** — it works with Phoenix, Express, Django, Rails, Go, or any server that reads a `PORT` environment variable. Port 4000 is kept free for ad-hoc use. 2. **A `justfile`** — a [`just`](https://github.com/casey/just) command runner file in the project root that wires together `phx-port` and your project's start command. For Elixir/BEAM projects, it can additionally integrate distributed Erlang (`--sname` / `--cookie`) and the [BEAM introspection script](/elixir-beam/beam-introspection-for-ai-agents) (`scripts/dev_node.sh`). The agent (or you) runs `just start` and gets a server on a known, stable port — every time, on every machine. ### How `phx-port` works `phx-port` auto-detects behavior based on context: | Context | Behavior | | ----------------------------------------------------- | ----------------------------------------------------------------------------------- | | **Piped** (e.g. `PORT=$(phx-port)`) | Prints just the port number. Auto-registers the current directory if not yet known. | | **Interactive** (run in a terminal with no arguments) | Shows help text. Never auto-registers accidentally. | | `phx-port list` | Shows all registered projects as a directory tree with clickable URLs. | | `phx-port open` | Opens the default browser at `http://localhost:` for the current project. | | `phx-port register` | Explicitly registers the current directory for a new port. | | `phx-port register debug` | Registers a named port role (e.g., for a debug port, metrics endpoint, etc.). | Projects can have multiple named port roles: ```bash # Main server port (works with any framework) PORT=$(phx-port) npm start # Node.js PORT=$(phx-port) mix phx.server # Phoenix PORT=$(phx-port) python manage.py runserver 0.0.0.0:$PORT # Django PORT=$(phx-port) go run . # Go (if your app reads $PORT) # Main + debug port PORT=$(phx-port) PORT_DEBUG=$(phx-port debug) mix phx.server ``` The registry looks like this: ```toml [ports."/home/user/projects/my_app"] main = 4001 [ports."/home/user/projects/api_gateway"] main = 4002 debug = 4003 ``` *** ## Step 1: Install `phx-port` ```bash cargo install --git https://github.com/chgeuer/phx-port ``` Or build from source: ```bash git clone https://github.com/chgeuer/phx-port cd phx-port cargo build --release cp target/release/phx-port ~/.local/bin/ # or anywhere on your PATH ``` Verify it works: ```bash phx-port --version ``` *** ## Step 2: Add the `justfile` Create a `justfile` in your project root. This is the single entry point for starting, stopping, and managing the server. Below are examples for different stacks. ### Generic `justfile` (Node.js, Python, Go, etc.) This works for any server that reads the `PORT` environment variable: ```just # Start the server (visible output, logs to run.log) start: #!/usr/bin/env bash export PORT="${PORT:-$(phx-port)}" echo "Starting on port $PORT..." exec your-start-command 2>&1 | tee run.log # Start the server in background (logs to run.log only) start-bg: #!/usr/bin/env bash export PORT="${PORT:-$(phx-port)}" your-start-command > run.log 2>&1 & echo "Started in background on port $PORT (PID $!). Logs in run.log" # Open the app in a browser open: phx-port open # Show the assigned port port: @phx-port ``` Replace `your-start-command` with what your project needs — `npm start`, `python manage.py runserver 0.0.0.0:$PORT`, `go run .`, `bundle exec rails server -p $PORT`, etc. ### Elixir / Phoenix `justfile` For Phoenix or other BEAM projects, the `justfile` can additionally wire in distributed Erlang for live introspection: ```just # Start the Phoenix server (visible output, logs to run.log) start: #!/usr/bin/env bash SNAME="$(basename "$(pwd)")" export PORT="${PORT:-$(phx-port)}" export ELIXIR_ERL_OPTIONS="-sname $SNAME -setcookie devcookie" exec mix phx.server 2>&1 | tee run.log # Start the Phoenix server in background (logs to run.log only) start-bg: #!/usr/bin/env bash SNAME="$(basename "$(pwd)")" export PORT="${PORT:-$(phx-port)}" export ELIXIR_ERL_OPTIONS="-sname $SNAME -setcookie devcookie" exec mix phx.server > run.log 2>&1 # Open the app in a browser (starts the server if not running) open: #!/usr/bin/env bash SNAME="$(basename "$(pwd)")" if ! scripts/dev_node.sh status > /dev/null 2>&1; then echo "Node $SNAME not running, starting in background..." export PORT="${PORT:-$(phx-port)}" export ELIXIR_ERL_OPTIONS="-sname $SNAME -setcookie devcookie" mix phx.server > run.log 2>&1 & scripts/dev_node.sh await fi phx-port open # Stop the running BEAM node gracefully stop: scripts/dev_node.sh rpc "System.stop()" # Check if the BEAM node is running status: scripts/dev_node.sh status # Execute an expression on the running BEAM node rpc EXPR: scripts/dev_node.sh rpc "{{EXPR}}" ``` ### What each recipe does | Recipe | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `just start` | Starts the server in the foreground. Output goes to the terminal **and** `run.log`. | | `just start-bg` | Same, but runs silently in the background. All output goes to `run.log`. | | `just open` | Opens the app in your browser. For the Elixir version, starts the server in the background first if needed. | | `just stop` | *(Elixir)* Gracefully shuts down the running BEAM node via `System.stop()`. | | `just status` | *(Elixir)* Checks whether the BEAM node is registered with `epmd`. | | `just rpc ''` | *(Elixir)* Evaluates an Elixir expression on the running node (requires `scripts/dev_node.sh` from the [BEAM introspection pattern](/elixir-beam/beam-introspection-for-ai-agents)). | ### Key design decisions * **`PORT` respects overrides** — `${PORT:-$(phx-port)}` means you can still do `PORT=9999 just start` if needed, but the default is always the stable `phx-port` assignment. * **`exec` replaces the shell** — the server process takes over the shell's PID, so signals (Ctrl+C) go directly to it. * *(Elixir-specific)* **`--sname` is derived from the directory name** — no configuration needed. The project directory `my_app` becomes node `my_app@hostname`. * *(Elixir-specific)* **`--cookie devcookie`** — a shared development cookie so `dev_node.sh` can connect for introspection. *** ## Step 3: Update `.gitignore` Add runtime artifacts that shouldn't be committed: ```gitignore # Runtime files run.log .dev_node.log .dev_node.pid ``` *** ## Step 4: Add Project Instructions for Your Agent Tell your AI coding agent about the `justfile` so it knows how to start and manage the server. Add this to your project's agent instructions file. ### GitHub Copilot — `AGENTS.md` or `.github/copilot-instructions.md` ```markdown ### Server management Use `just` recipes to manage the development server: - `just start` — start the server (foreground, visible output) - `just start-bg` — start the server in the background - `just open` — open the app in the browser - `just port` — show the assigned port number The server uses `phx-port` for stable port assignment. Never hardcode port numbers. To make HTTP requests against the running server, use `http://localhost:$(phx-port)`. ``` > For Elixir/BEAM projects, add `just stop`, `just status`, and `just rpc ''` to the list above — see the Elixir `justfile` variant. ### Claude Code — `CLAUDE.md` ```markdown ## Server management Use `just` recipes to manage the development server: - `just start` — start the server (foreground, visible output) - `just start-bg` — start the server in the background - `just open` — open the app in the browser - `just port` — show the assigned port number The server uses `phx-port` for stable port assignment. Never hardcode port numbers. To make HTTP requests against the running server, use `http://localhost:$(phx-port)`. ``` ### OpenAI Codex / Gemini CLI — `AGENTS.md` Same content as the Copilot section above. Both tools read `AGENTS.md` at the project root. *** ## How It All Fits Together Here's a typical workflow, whether it's you or an AI agent: ```bash # First time — phx-port auto-registers the project ~/projects/my_app $ just start Registered /home/user/projects/my_app → port 4001 # ← stderr, first time only Starting on port 4001... Listening on http://localhost:4001 # Second time — port is already known, instant ~/projects/my_app $ just start Starting on port 4001... Listening on http://localhost:4001 # Meanwhile, in another project (Node.js, Python, anything) — no conflict ~/projects/api $ just start Registered /home/user/projects/api → port 4002 Starting on port 4002... Server running at http://localhost:4002 # Check what's registered across all your projects $ phx-port list /home/user/projects ├── my_app ...... http://localhost:4001 └── api ......... http://localhost:4002 ``` An AI agent working on `my_app` simply runs `just start-bg`, waits for the server, and can then `curl http://localhost:$(phx-port)` to validate its changes — without worrying about port conflicts with the `api` project running in the background. *** ## Elixir / BEAM Bonus: Combining with BEAM Introspection This pattern is designed to work together with the [BEAM Live Introspection](/elixir-beam/beam-introspection-for-ai-agents) pattern. The `justfile` recipes (`stop`, `status`, `rpc`) delegate to `scripts/dev_node.sh`, which provides full runtime introspection capabilities. To set up both patterns together: 1. Follow this article to add `phx-port` and the `justfile` 2. Follow the [BEAM introspection article](/elixir-beam/beam-introspection-for-ai-agents) to add `scripts/dev_node.sh` and the introspection skill The `justfile` becomes the high-level interface ("start my server"), while `dev_node.sh` provides the low-level distributed Erlang plumbing ("connect to the running node and evaluate this expression"). ``` justfile ← Human / agent entry point ├── start / start-bg ← Uses phx-port for port, --sname for node naming ├── open ← Auto-starts + opens browser via phx-port ├── stop ← Delegates to dev_node.sh rpc "System.stop()" ├── status ← Delegates to dev_node.sh status └── rpc ← Delegates to dev_node.sh rpc scripts/dev_node.sh ← BEAM introspection engine ├── start / stop / status ← Node lifecycle ├── await ← Wait for node to be connectable ├── rpc ← Execute expression on live node └── eval_file ← Evaluate .exs file on live node ```