Gideon Wolfe

How to add a picom toggle to polybar

Sat, 07 Nov 2020 00:00:00 +0000

Intro

I think a tasteful blur and a bit of transparency can really spice up a good setup. However, there are times that I don’t want to have the blur on. Whether this be for a specific color theme, or just to make reading code easier, I like having easy access to my blur settings without memorizing huge commands.

Solution

Since I already have a similar module for dunst notifications, I decided to attempt to replicate the same thing for picom. We can add the following module to our polybar config.

[module/blur-toggle]
type = custom/ipc
hook-0 = echo 
hook-1 = echo 
click-left = /home/gideon/dotfiles/dotfiles/scripts/blurtoggle.sh
initial = 1
format-foreground = ${colors.color4}

Now you can change the filepath on the click-left variable to point at wherever you keep your scripts. But point it at a script that has the following contents:

#!/usr/bin/sh
if (($(ps -aux | grep [p]icom | wc -l) > 0))
then
  polybar-msg hook blur-toggle 1
  pkill -9 picom
else
  polybar-msg hook blur-toggle 2
  picom -b --config=/home/gideon/.config/picom/picom.conf --experimental-backends --backend glx --blur-method dual_kawase &
fi

Again, you can replace the path used in the --config flag with wherever you keep your picom config.

Conclusion

I have found this method to work rather well with a couple exceptions. I find that if I leave the blur on for extended periods, the “button” indicator in polybar will flip to inactive even though the blur remains.

When this “desyncing” occurs, I simply flip the toggle twice and it resets things to normal. If anyone knows a fix or cause for this, I’d be glad to hear it.

xmenu: Desktop Agnostic Menu Utility

Tue, 04 Aug 2020 00:00:00 +0000

Introduction

In the past, I have detailed some of the helper programs I use with the i3 window manager. I have a new one, something that is actually really helpful for my workflow.

So far, my workflow has been entirely keyboard driven. But that doesn’t mean I hate the mouse. I really hate switching between the keyboard and mouse. So if I am just using the mouse to click youtube videos or play a game, why would I want to reach for the keyboard to bring up rofi?

That’s where xmenu comes in. xmenu gives you the openbox style menu that can be navigated with either the keyboard or the mouse.

Installation

Since I run arch, I was able to grab the xmenu program off of the AUR with

yay -Syu xmenu

Otherwise you can clone the repository, enter it, and

make
sudo make install

I choose to build from source because it allows me to theme on the fly.

Usage

The beauty of xmenu is that it works like dmenu or rofi, and takes its options from STDIN.

This means that by default, xmenu does nothing. When we want to invoke it, we would pipe in the options we want, usually through a bash script for convenience.

For example, to create the example in the above gif, simply use this script:

#!/bin/sh

cat <<EOF | xmenu | sh &
Applications
	IMG:./icons/web.png	Web Browser	firefox
	IMG:./icons/gimp.png	Image editor	gimp
Terminal (xterm)	xterm
Terminal (urxvt)	urxvt
Terminal (st)		st
Shutdown		poweroff
Reboot			reboot
EOF

Each value is separated by the TAB character. On the left is the name to be displayed, and then one tab over is the command/script to be run if that option is selected.

You can make nested menus by indenting another tab, and line breaks with empty lines. Note that nested line breaks need the appropriate number of tabs at the beginning of the line followed by nothing.

Theming

If you know anything about me, it’s that I will not be happy with that lame white menu. Inside the xmenu directory, we can find a file called config.h.

This file defines all of the colors and styles used by xmenu. Changing this file requires a rebuild of xmenu.

So I created a pywal template for the config.h file.

static struct Config config = {{
	/* font */
  .font = "HackNerdFont:size=9,FontAwesome:size=9,FontAwesomeBrands:size=11",

	/* colors */
	.background_color = "{background}",
	.foreground_color = "{foreground}",
	.selbackground_color = "{background}",
	.selforeground_color = "{color2}",
	.separator_color = "{color4}",
	.border_color = "{color5}",

	/* sizes in pixels */
	.width_pixels = 130,        /* minimum width of a menu */
	.height_pixels = 25,        /* height of a single menu item */
	.border_pixels = 1,         /* menu border */
	.separator_pixels = 3,      /* space around separator */
	.gap_pixels = 0,            /* gap between menus */

	/*
	 * The variables below cannot be set by X resources.
	 * Their values must be less than .height_pixels.
	 */

	/* geometry of the right-pointing isoceles triangle for submenus */
	.triangle_width = 4,
	.triangle_height = 7,

	/* the icon size is equal to .height_pixels - .iconpadding * 2 */
	.iconpadding = 2,

	/* area around the icon, the triangle and the separator */
	.horzpadding = 10,
}};

Every time wal is run, a new file is created in ~/.cache/wal/. I symlink this file to the location xmenu, and rebuild it every time I change theme.

ln -s /home/gideon/.cache/wal/xmenu-config.h /home/gideon/Programs/xmenu/config.h

I use my own theming tool chameleon to handle themes. If chameleon detects xmenu on your system, it will attempt to rebuild it on the fly.

Look at these colors!!

polybar integration

I use i3, which doesn’t exactly have a built in mechanism for “right click desktop menu”. The author of xmenu has another tool, xclickroot which helps with this.

However, since my windows are tiled, I rarely see the desktop anyways. That is why I integrated xmenu into my polybar like a traditional start menu!

In my polybar config, I have the following module:

[module/xmenu-left]
type = custom/script
click-left = /home/gideon/dotfiles/dotfiles/scripts/xmenu-left-monitor.sh
click-right = /home/gideon/dotfiles/dotfiles/scripts/xmenu-left-monitor.sh
exec = /home/gideon/.config/polybar/scripts/xmenu-button.sh
format-foreground = ${colors.color4}
tail = true
interval = 90

I have three almost identical modules, one for each of my monitors. The reason for this is that each monitor calls a different xmenu script, which tells xmenu to position itself below the bar of that display.

The script xmenu-button.sh is dead simple, and can remain consistent across all modules if you like. It simply echoes out the character you want to click on in polybar.

#!/usr/bin/sh

while true; do
  echo 
  sleep 30 &
  wait
done

You can slap pretty much anything in there, but I like to use FontAwesome icons because that’s what the rest of my bar uses.

My config

Here is the ENTIRE xmenu script I have at the moment for my bars. Note the invocation of xmenu.

xmenu -i -p 0x25:1

-i tells xmenu to remove the space for icons
-p tells xmenu to use a specific position
0x25 the coordinates that place xmenu below my bar (25px down)
:1 on screen one

#!/bin/sh

COLORSCRIPTSDIR=$HOME/Programs/color-scripts/color-scripts
CONFIGDIR=$HOME/dotfiles/dotfiles/config/


cat <<EOF | xmenu -i -p 0x25:1 | sh &
 Applications
	 Firefox	firefox
	 Terminal	urxvt
	 Files	thunar
	
	 School
		 canvas-tui 		urxvt -e canvas-tui
		 Zotero 		zotero
		 CS
			145
				Notes		urxvt -e nvim $HOME/School/CS/145/notes.java
			241
				Notes		urxvt -e nvim $HOME/School/CS/241/notes
			247
				Notes		urxvt -e nvim $HOME/School/CS/247/notes
				Books
					Susheel's notes		zathura $HOME/School/CS/247/CSCI247_CourseNotes.pdf
					Intel manual		zathura $HOME/School/CS/247/intel.pdf
					C Programming Language		zathura $HOME/School/CS/247/the_c_programming_language_2.pdf
			301
				Notes		urxvt -e nvim $HOME/School/CS/301/notes
				Books
					Theory of Computation		zathura $HOME/School/CS/301/TheoryOfComputation.pdf
					Book of Proof		zathura $HOME/School/CS/301/Main.pdf
			305
				Notes		urxvt -e nvim $HOME/School/CS/305/notes.md
				Textbook		zathura $HOME/School/CS/305/textbook.pdf
			330
				Notes		urxvt -e nvim $HOME/School/CS/330/notes.md
			347
				Notes		urxvt -e nvim $HOME/School/CS/347/notes.md
				Textbook		zathura $HOME/School/CS/347/textbook.pdf
			367
				Notes		urxvt -e nvim $HOME/School/CS/367/notes.md
				Textbook		zathura $HOME/School/CS/367/textbook.pdf
			461
				Notes		urxvt -e nvim $HOME/School/CS/461/notes.md
				Textbook		zathura $HOME/School/CS/461/textbook.pdf
			474
				Notes		urxvt -e nvim $HOME/School/CS/474/notes.md
	 Comms
		Discord (GUI)		discord
		Discord (cordless)		urxvt -e cordless
		 Email (Neomutt)		urxvt -e neomutt
		 IRC (weechat)		urxvt -e weechat
		 SMS (kde-connect)		kdeconnect-sms --style gtk2
		 Slack 	slack
		 Signal (GUI)		signal-desktop
		 Telegram (GUI)		telegram-desktop
		 Telegram (nctelegram)		nctelegram
	 Finance
		 Crypto (TUI)		urxvt -e cointop
		 Stocks (TUI)		urxvt -e mop
		 Stonks (CLI)		urxvt -e mop
	 Utilities
		 Calculator (TUI)		urxvt -e qalq
		 Calculator (GUI)		qalculate-gtk
		 Calendar (khal)		urxvt -e khal interactive
		 Color Picker		gpick
		 Map (TUI)		urxvt -e mapscii
		
		 System
			 Keyboard (Corsair)		ckb-next
			 Fonts 	gucharmap
			 Smartphone (kdeconnect)		kdeconnect-app --style gtk2
			 Docker (lazydocker)		urxvt -e lazydocker
			 Kill Window	xkill
			 Screen Recording (OBS)		obs
			 Screenshot
				GUI	flameshot gui
				All Displays	flameshot screen -d 5000 -n 3 -p $HOME/Photos/screenshots
				Middle Display	flameshot screen -d 5000 -n 0 -p $HOME/Photos/screenshots
				Right Display	flameshot screen -d 5000 -n 1 -p $HOME/Photos/screenshots
				Left Display	flameshot screen -d 5000 -n 2 -p $HOME/Photos/screenshots
		 Personal
			 Nextcloud		nextcloud --style gtk2
			 Passwords (keepassxc)		keepassxc --style gtk2
			 Keys (Seahorse)		seahorse
		 Theming
			lxappearance 	lxappearance
			GTK (oomox) 	oomox-gui
			Qt (qt5ct) 	qt5ct --style gtk2
			WPGTK 	wpg
		 Monitors
			System (ytop)		urxvt -e ytop
			System (bashtop)		urxvt -e bashtop
			System (glances)		urxvt -e glances
			
			Disk Usage (GUI)		baobab
			Disk Usage (TUI)		urxvt -e ncdu
			IO (iotop)		urxvt -e iotop
			
			Kernel (kmon)		urxvt -e kmon
			Nvidia GPU (nvtop)		urxvt -e nvtop
			Power (powertop)		urxvt -e powertop
			
			DNS (dnstop)		urxvt -e powertop
			Network Usage (jnettop)		urxvt -e jnettop
			Network Load (nload)		urxvt -e nload
			Bandwidth (bmon)		urxvt -e bmon
			Media Server		urxvt -e jellyfinips.sh
		 Media
			 EasyTag		easytag
	 Entertainment
		 Media
			 Podcasts (castero)	castero
			 RSS (newsboat)	newsboat
			 Reddit (tuir)	tuir
			 Music (cmus)	cmus
			 Spotify (GUI)	spotify
			 Spotify (spotifytui)	spt
			Soulseek (Nicotine+)	nicotine
		 Games
			 Steam	steam
			Itch	itch
			Lutris	lutris
			Tetris	tetris
			Solitaire	ttysolitaire
			Battleship	bs
			Minecraft	minecraft-launcher
			Dopewars	dopewars
		 Misc
			Color Scripts
				alpha	urxvt -e sh -c '$COLORSCRIPTSDIR/alpha; read'
				arch	urxvt -e sh -c '$COLORSCRIPTSDIR/arch; read'
				bars	urxvt -e sh -c '$COLORSCRIPTSDIR/bars; read'
				blocks2	urxvt -e sh -c '$COLORSCRIPTSDIR/blocks2; read'
				bloks	urxvt -e sh -c '$COLORSCRIPTSDIR/bloks; read'
				colorbars	urxvt -e sh -c '$COLORSCRIPTSDIR/colorbars; read'
				colortest	urxvt -e sh -c '$COLORSCRIPTSDIR/colortest; read'
				colortest-slim	urxvt -e sh -c '$COLORSCRIPTSDIR/colortest-slim; colortest'
				colorview	urxvt -e sh -c '$COLORSCRIPTSDIR/colorview; read'
				crunch	urxvt -e sh -c '$COLORSCRIPTSDIR/crunch; read'
				crunchbang	urxvt -e sh -c '$COLORSCRIPTSDIR/crunchbang; read'
				crunchbang-mini	urxvt -e sh -c '$COLORSCRIPTSDIR/crunchbang-mini; read'
				darthvader	urxvt -e sh -c '$COLORSCRIPTSDIR/darthvader; read'
				dna	urxvt -e sh -c '$COLORSCRIPTSDIR/dna; read'
				dotx	urxvt -e sh -c '$COLORSCRIPTSDIR/dna; read'
				elfman	urxvt -e sh -c '$COLORSCRIPTSDIR/elfman; read'
				faces	urxvt -e sh -c '$COLORSCRIPTSDIR/faces; read'
				fade	urxvt -e sh -c '$COLORSCRIPTSDIR/fade; read'
				ghosts	urxvt -e sh -c '$COLORSCRIPTSDIR/ghosts; read'
				guns	urxvt -e sh -c '$COLORSCRIPTSDIR/guns; read'
				hex	urxvt -e sh -c '$COLORSCRIPTSDIR/hex; read'
				hex-block	urxvt -e sh -c '$COLORSCRIPTSDIR/hex-block; read'
				illumina	urxvt -e sh -c '$COLORSCRIPTSDIR/illumina; read'
				jangofett	urxvt -e sh -c '$COLORSCRIPTSDIR/jangofett; read'
				monster	urxvt -e sh -c '$COLORSCRIPTSDIR/monster; read'
				mouseface	urxvt -e sh -c '$COLORSCRIPTSDIR/mouseface; read'
				pacman	urxvt -e sh -c '$COLORSCRIPTSDIR/pacman; read'
				panes	urxvt -e sh -c '$COLORSCRIPTSDIR/panes; read'
				pinguco	urxvt -e sh -c '$COLORSCRIPTSDIR/pinguco; read'
				pipes1	urxvt -e sh -c '$COLORSCRIPTSDIR/pipes1; read'
				pipes2	urxvt -e sh -c '$COLORSCRIPTSDIR/pipes2; read'
				pipes2-slim	urxvt -e sh -c '$COLORSCRIPTSDIR/pipes2-slim; read'
				pukeskull	urxvt -e sh -c '$COLORSCRIPTSDIR/pukeskull; read'
				rails	urxvt -e sh -c '$COLORSCRIPTSDIR/rails; read'
				rally-x	urxvt -e sh -c '$COLORSCRIPTSDIR/rally-x; read'
				rupees	urxvt -e sh -c '$COLORSCRIPTSDIR/rupees; read'
				space-invaders	urxvt -e sh -c '$COLORSCRIPTSDIR/space-invaders; read'
				spectrum	urxvt -e sh -c '$COLORSCRIPTSDIR/spectrum; read'
				square	urxvt -e sh -c '$COLORSCRIPTSDIR/square; read'
				tanks	urxvt -e sh -c '$COLORSCRIPTSDIR/tanks; read'
				thebat	urxvt -e sh -c '$COLORSCRIPTSDIR/thebat; read'
				thebat2	urxvt -e sh -c '$COLORSCRIPTSDIR/thebat2; read'
				tiefighter1-no-invo	urxvt -e sh -c '$COLORSCRIPTSDIR/tiefighter1-no-invo; read'
				tiefighter2	urxvt -e sh -c '$COLORSCRIPTSDIR/tiefighter2; read'
				zwaves	urxvt -e sh -c '$COLORSCRIPTSDIR/zwaves; read'
			cava	urxvt -e cava
			pipes.sh	urxvt -e pipes.sh
			rain.sh	urxvt -e rain.sh
			unimatrix	urxvt -e unimatrix -l Gg
			asciiquarium	urxvt -e asciiquarium
			bonsai.sh	urxvt -e bonsai -l -i -T
			eDEX-UI		sh -c (cd $HOME/Programs/edex-ui/ ; npm start)
	 Science
		 Astronomy
			Celestia	celestia
		 Biology
			Pymol		pymol
		 Chemistry
			ptable	urxvt -e ptable
			chemtool	chemtool
		 Math
			Desmos	desmos
			Geogebra	geogebra
		Anki		anki
	 Development
		 IDEs
			Neovim	urxvt -e nvim
			VS Code	code
			Dr. Racket	drracket
		 Github (TUI)	urxvt -e github
		bitwise	bitwise
		Github Activity (TUI)	urxvt -e sh -c 'ghcal --no-ansi ; read'
		QDbusViewer	qdbusviewer --style gtk2	

 Configs
	 System
		 i3	urxvt -e nvim $CONFIGDIR/i3/config
		 Start Menu	urxvt -e nvim $HOME/dotfiles/dotfiles/scripts/xmenu.sh
		 Notifications	urxvt -e nvim $CONFIGDIR/wal/templates/dunstrc
		 Smartphone	kdeconnect-settings --style gtk2
		 Sound	pavucontrol
		 Shell
			fish
				config.fish	urxvt -e nvim $CONFIGDIR/fish/config.fish
				Web Config	fish -c fish_config
			bash	urxvt -e nvim $HOME/.bashrc
		 polybar
			config	urxvt -e nvim $CONFIGDIR/polybar/config
			launch.sh	urxvt -e nvim $CONFIGDIR/polybar/scripts/launch.sh
		 rofi
			config	urxvt -e nvim $CONFIGDIR/rofi/config
			template	urxvt -e nvim $CONFIGDIR/wal/templates/custom-rofi.rasi
		 Utilities
			 khard	urxvt -e nvim $CONFIGDIR/khard/khard.conf
			 khal	urxvt -e nvim $CONFIGDIR/khal/config
			 vdirsyncer	urxvt -e nvim $CONFIGDIR/vdirsyncer/config
		 .Xresources	urxvt -e nvim $HOME/.Xresources
	 User
		ranger	urxvt -e nvim $CONFIGDIR/ranger/rc.conf
		newsboat	urxvt -e nvim $CONFIGDIR/newsboat/config
		neomutt
			neomuttrc	urxvt -e nvim $CONFIGDIR/neomutt/neomuttrc
			colors	urxvt -e nvim $CONFIGDIR/neomutt/colors
			settings	urxvt -e nvim $CONFIGDIR/neomutt/settings
			mappings	urxvt -e nvim $CONFIGDIR/neomutt/mappings
			mailcap	urxvt -e nvim $CONFIGDIR/neomutt/mailcap
		neovim
			coc-settings	urxvt -e nvim $HOME/.config/nvim/coc-settings.json
			functions	urxvt -e nvim $HOME/.config/nvim/configs/functions.vim
			main	urxvt -e nvim $HOME/.config/nvim/configs/main.vim
			mappings	urxvt -e nvim $HOME/.config/nvim/configs/mappings.vim
			plugin-settings	urxvt -e nvim $HOME/.config/nvim/configs/plugin-settings.vim
			plugins	urxvt -e nvim $HOME/.config/nvim/configs/plugin.vim
		neofetch	urxvt -e nvim $CONFIGDIR/neofetch/config.conf
		htop	urxvt -e nvim $CONFIGDIR/htop/htoprc
		s-tui	urxvt -e nvim $CONFIGDIR/s-tui/s-tui.conf
		spicetify	urxvt -e nvim $CONFIGDIR/spicetify/config.ini
		stonks	urxvt -e nvim $CONFIGDIR/stonks.yml

 System
	 Refresh i3		i3-msg restart
	 Refresh polybar		pkill -9 polybar ; $CONFIGDIR/polybar/scripts/launch.sh
	
	 Logout	i3-nagbar -t warning -m 'Do you really want to exit i3? This will end your X session.' -b 'Yes, exit i3' 'i3-msg exit'
	 Shutdown		poweroff
	 Reboot			reboot
EOF

Feel free to take inspiration!

Conclusion

Good job for making it this far. If you followed along, you should have a desktop environment agnostic menu utility with infinite potential!

Single and Pairwise Mutations and Their Impact on SARS-CoV-2 Proteins

Mon, 22 Jun 2020 00:00:00 +0000

Intro

This past quarter (Spring 2020) has been an interesting one for sure. With the constant news of COVID and worldwide protests coming through every media outlet, focusing on schoolwork seemed like futile pursuit at times.

Thankfully, I was fortunate enough to have (mostly) fantastic professors that managed to provide informative and engaging lectures and assignments for the entirety of this challenging quarter.

A part of me is really enjoying working from home this quarter, and that part is mostly my GPA, which ended up being over an entire point higher than my unimpressive cumulative GPA. Not quite sure why or how that happened, but I guess I’ll just continue as I am…

The Class

One of the things that made this quarter special is the fact that I was able to take my schools Bioinformatics class, which I have been trying to get into for a couple years.

It was taught by one of my favorite professors (and now my advisor), is only taught once a year, and requires a manual override to register for. I was thrilled when I found out I scraped one of the last slots!

This class was a unique opportunity to collaborate with other STEM students (bioilogy/chemistry) as well as grad students and tackle a real world research problem with techniques and tools we learn in the first part of the quarter.

The first 4 weeks brought all the CS people up to speed on the basics of DNA transcription/translation, protein structure, and alignment concepts.

The Project

The latter half of the quarter was devoted to a collaborative research project, the groups for which we were allowed to self select out of a pool of pre approved project ideas.

The project I chose was this: Target various amino acids on the spike protein of the SARS-Cov-2, mutate them, and analyze the effectiveness of said mutations.

I chose this project because I like the idea of pinpointing vulnerable areas on the protein and mutating them to cause instability in the protein, possibly breaking functionality in the process.

My role in the project mainly consisted of managing communications between us and our professor, as well as creating the pipeline for our data. In short, the pipeline worked like this:

Use proMuteBatch to generate batch files for proMute, in order to exhaustively mutate multiple locations on the protein
Extract the output of proMute (modified PDB files) as well as energy minimization data for each individual mutation (19 per target site)
Generate a script for the SDM tool, which allows us to predict $$\Delta\Delta G$$ values for each mutation.

I created a short bash script to automate this task. Although it could certainly be better optimized, I sacrificed some efficiency for readability by non linux nerds.

#!/bin/bash

# After making promute, copy this script into the build directory
# Change the "FILES" variable to the script (or scripts) you would like to execute
# to profile different mutation scripts, you can run this script with the "time" command 

# Mutations on second PDB 6Y2E
./proMuteBatch 6Y2E A:A 25:25 X T25 # exhaustively mutate T at residue 25, chain A
./proMuteBatch 6Y2E A:A 26:26 X T26 # exhaustively mutate T at residue 26, chain A
./proMuteBatch 6Y2E A:A 27:27 X L27 # exhaustively mutate L at residue 27, chain A
./proMuteBatch 6Y2E A:A 28:28 X N28 # exhaustively mutate N at residue 28, chain A
./proMuteBatch 6Y2E A:A 29:29 X G29 # exhaustively mutate G at residue 29, chain A
./proMuteBatch 6Y2E A:A 30:30 X L30 # exhaustively mutate L at residue 30, chain A
./proMuteBatch 6Y2E A:A 31:31 X W31 # exhaustively mutate W at residue 31, chain A
./proMuteBatch 6Y2E A:A 32:32 X L32 # exhaustively mutate L at residue 32, chain A


declare -A residues
residues+=( 
  ["PHE"]="F" 
  ["LEU"]="L" 
  ["ILE"]="I"
  ["MET"]="M"
  ["VAL"]="V"
  ["SER"]="S"
  ["PRO"]="P"
  ["THR"]="T"
  ["ALA"]="A"
  ["TYR"]="Y"
  ["HIS"]="H"
  ["GLN"]="Q"
  ["ASN"]="N"
  ["LYS"]="K"
  ["ASP"]="D"
  ["GLU"]="E"
  ["CYS"]="C"
  ["TRP"]="W"
  ["ARG"]="R"
  ["GLY"]="G"
)

sed -i 's/$/ em/' T25
sed -i 's/$/ em/' T26
sed -i 's/$/ em/' L27
sed -i 's/$/ em/' N28
sed -i 's/$/ em/' G29
sed -i 's/$/ em/' L30
sed -i 's/$/ em/' W31
sed -i 's/$/ em/' L32


# Customize with your scripts
FILES=("T25" "T26" "L27" "N28" "G29" "L30" "W31" "L32")

# Execute the lines in the script
for FILE in ${FILES[@]}; do


  # Create a file to store SDM script
  SDMFILE="${FILE}_sdm"
  touch $SDMFILE

  # Read each mutation into an array
  readarray -t LINES < "$FILE"

  # Routine to execute for every mutation
  for LINE in "${LINES[@]}"; do

    # Make a directory to store outputs
    # Structure is PDBID.Chain.Location.Res2.output
    OUTPUTDIRNAME=$(echo $LINE | awk '{ print $2"."$3"."$4"."$5".output" }')
    echo [INFO] Creating Directory $OUTPUTDIRNAME
    mkdir -p $OUTPUTDIRNAME

    # Directory to look for EM files that were generated
    # Structure is PDBID.ChainLocationRes2
    EMDIRNAME=$(echo $LINE | awk '{ print $2"."$3$4$5 }')

    # Actually call proMute
    $LINE

    # translate the promute command to an SDM script line
    PDBID=$(echo $LINE | awk '{print $2}')
    RESNUM=$(echo $LINE | awk '{print $4}')
    CHAIN=$(echo $LINE | awk '{print $3}')
    FINALRES=$(echo $LINE | awk '{print $5}')
    ORIGRESCODE=$(cat ${PDBID}.pdb | grep ATOM | awk -v rn="$RESNUM" -v chn=$CHAIN '$6 == rn && $5 == chn {print $4}' | sed 's/.*\(...\)/\1/' | head -n 1)
    ORIGRES=${residues[$ORIGRESCODE]}
    if [ "$ORIGRES" != "$FINALRES" ]
    then
      # put this line in the script
      echo $(echo $CHAIN $ORIGRES$RESNUM$FINALRES) >> $SDMFILE
    fi

    # Move all output files: PDB, Fasta, and EM data
    mv *.txt $OUTPUTDIRNAME
    mv *.pdb $OUTPUTDIRNAME
    mv external/em/$EMDIRNAME $OUTPUTDIRNAME
    mv $FILE $OUTPUTDIRNAME
    echo [INFO] Moving files to $OUTPUTDIRNAME

  done

  # cleanup output
  rm -rf "${FILE}_output"
  mkdir -p "${FILE}_output"
  echo [INFO] Cleaning outputs from last run
  echo [INFO] Moving new files to "${FILE}_output"
  mv $SDMFILE "${FILE}_output/"
  mv *.output "${FILE}_output/"

done

Most of this script is pretty self explanatory. I simply generate a list of proMute commands using proMuteBatch, and executed all of them.

I used clues from filenames and commands (such as protein name and residue number) to generate unique directories of every mutation ran by proMute.

The only “confusing” part would be the generation of the SDM script. In short, the script format for SDM requires the original residue type for the location that is being mutated.

proMute does not care about or output this information, so I had to do some fancy pipelining to get it to work.

I look into the PDB file for the target protein, and find isolate all the ATOM lines, that correspond to individual atoms in the protein.

I then isolate the residue number of interest, making sure we are looking at the correct chain in the PDB file.

The first atom of the residue is kept, and then we simply look at the residue it belongs to. I wasn’t sure about the PDB notation, because sometimes one amino acid would be represented by multiple similar 3-4 letter codes.

So I just chopped off anything after three characters, and then run the three letter name through my dictionary, which spits out the single letter amino acid code required by SDM.

That’s a lot of work to go through for one letter…

The final step in the pipeline was a script created by an adjacent group studying double mutations in the same protein. This script analyzed the energy minimization data and used it to predict an effect on protein function.

Results

Here are my favorite figures from our final paper. We used pymol to color and label the locations of interest, as well as illustrate their bonds and connections.

The colored bar charts on the right represent one mutation for each of the 19 bars. The data plotted is the output $$\Delta\Delta G$$ produced by SDM, where a negative value indicates reduced stability in the protein.

A heatmap visualization of the same data:

As you can see by the heatmaps, we were able to generate a significant amount of destabilizing mutations, especially on the 6Y2Estructure.

Conclusion

If anyone wants to read our final project, I’ll leave it here to download. At the last minute, we decided to combine the reports of group 4 and 5, who investigated single and double mutations respectively. This provides a good way to compare a variety of possible mutations of the same structure.

I had a really great time in this class, and could not have dreamed of creating a project of this caliber, and I could not have done it without my awesome group members and my professor who had the perfect “hands off” approach where he would happily provide guidance, but left large decisions up to the groups themselves.

Canvas in the terminal with canvas-tui

Thu, 30 Apr 2020 00:00:00 +0000

Intro

What better way to spend time during quarantine than to program an application nobody will ever use? Well obviously I couldn’t come up with anything, so I did just that.

canvas-tui is an application I wrote in order to view my grades, assignments, and other school information from the comfort of my terminal.

This is one of those occasions I have mentioned in previous articles, where no premade solution was readily available, so I was forced to give it a shot myself. The fact that I was able to produce a perfectly functional application in less than a week is a testament to the flexibility and efficacy of a linux terminal based workflow.

History

About two weeks before I developed canvas-tui, me and a friend teamed up on a small, python based utility called canvas-cli. This program can take simple command line arguments and output tables containing your grades, assignments, and more.

Here, I used a great program called wtfutil to throw together a static dashboard that showed the result of running various canvas-cli commands.

Other than that, I still use it to quickly download a desired file from my course into the working directory.

canvas-tui

My original plan was to use canvas-cli and plug its output into a terminal interface. I decided on go as my language for this project, as I wanted to take advantage of the termui library.

Before this project, I had not written a single line of go in my life. I looked into piping output from python into go, but it seemed like far more trouble than it was worth in the end.

I decided on rewriting most all of canvas-clis features in go, which makes it much easier to integrate new features not possible in canvas-cli.

Dashboard

I want this page to be a quick glance at your school status.

So far I have a bar chart that displays your current score in each course, as well as a “To do list”, which is comprised of any outstanding assignments in any class.

Some professors have yet to grade any assignments, causing my “Current Score” to be zero. Many of the features of canvas-tui are impacted by how professors choose to utilize the Canvas features.

Course Overview

Each course has their own “dashboard” of sorts. My goal here was to display some of the most commonly sought information such as grades and assignments, but also visualize other data about the course, such as the weighting of different assignment groups.

I use this page daily to view the most recent announcement, click on the zoom link for class, or quickly view assignments to complete.

Other Pages

On the lefthand side of the course overview, you will see the navigation menu that can be scrolled using j and k (like civilized people).

Pressing Space on any of these items will either bring up a placeholder page, or open the dashboard you choose. While I am always thinking of ways to visualize the course tabs in the terminal, some are not feasible or useful to see there.

That is why after selecting the page you want to view, pressing o will open it in your browser of choice.

The main goal of canvas-tui is to provide statistics and metrics that are usually not provided within the default canvas experience.

Visualizing scores over time, by assignment group, by course, and more are planned additions to the program, as well as whatever else I can wrangle out of the data provided by the Canvas API.

Conclusion

I saw a program that didn’t exist, and wondered why. I realized that was probably because the market for an application like this is practically nonexistent.

I promptly ignored this and set out to create it anyways. At its current form, it does far more than I had initially planned. Nonetheless, I am continually thinking of new features to add, so make sure to stay up to date on the github!

Agents Standing By: Notifications through Telegram

Tue, 31 Mar 2020 00:00:00 +0000

I got a request to go over the setup of a Telegram agent for Huginn. I personally don’t use Telegram at the moment, but I managed to get a basic bot set up.

Create your Telegram bot

First you will need to create a bot by messaging the BotFather and requesting a new bot.

This can be done by typing /newbot and then filling out the prompts for a username and bot name.

Once this is completed, BotFather will give you an HTTP API token. We will need this value.

Click the link to start messaging your bot. Send a message, this will come in handy later.

Setting up the Telegram Agent

In Huginn, we will create a new Telegram Agent.

The fields are pretty self explanatory.

Paste your auth token from BotFather into the Auth token field
To get your chat_id, visit https://api.telegram.org/bot<API TOKEN>/getUpdates, replacing <API TOKEN> with your actual token. Then the id field under chat will have the chat_id.
Change other options to preference

I think parsing into markdown is great, since I am familiar with it already.

Sending Data

The Telegram agent expects events with one of the following tags: text, photo, audio, document or video.

Most likely you will need to use an EventFormattingAgent to put your desired messages into this format.

To test it out, I created a simple ManualEventAgent and pointed it at the Telegram agent.

I fired off an event that was simply

{
  "text": "Hello World"
}

and promptly received the text “Hello World” from my bot.

With the use of the EventFormattingAgent and a bit of imagination, you can format pretty much any incoming event to a telegram message

Email in the Terminal: Configuring Neomutt

Tue, 31 Mar 2020 00:00:00 +0000

Intro

In one of my first posts about building an optimized workflow, I briefly mentioned neomutt was a program who’s configuration deserved a post all its own. This is that post.

Much like weechat, Neomutt is an extremely powerful tool but it needs a bit of customization to make it more “user friendly”.

On Arch we can install Neomutt with pacman -Syu neomutt

You have probably seen tons of tutorials about setting up offlineIMAP or a similar service. I personally opt to connect directly to the IMAP server as its less of a hassle. Maybe there is a good reason to do the whole offline sync thing but I haven’t found it.

Files

You may have heard of a muttrc or neomuttrc. Like a vimrc, these files define the settings for neomutt. I place my neomuttrc in ~/.config/neomutt/ because its where most programs store their config files.

However much like my custom neovim configuration, the main configuration file will simply be used to source the discrete configuration files containing specific configurations.

# vim: filetype=muttrc

source ~/.config/neomutt/settings
source ~/.config/neomutt/colors
source ~/.config/neomutt/mappings

# Accounts

source ~/.config/neomutt/accounts/gmail
folder-hook $folder 'source ~/.config/neomutt/accounts/gmail'

source ~/.config/neomutt/accounts/school
folder-hook $folder 'source ~/.config/neomutt/accounts/school'

source ~/.config/neomutt/accounts/main
folder-hook $folder 'source ~/.config/neomutt/accounts/main'

In the above neomuttrc file, I source three files: settings for general program configuration, colors to adjust colors, and mappings to store custom keybinds.

Additionally, in the accounts folder, I have a configuration file for each of the email accounts I want to use with neomutt. We will get to that configuration later.

General Settings

Here is where we can declare some universal settings for neomutt that change core program behavior.

# vim: filetype=muttrc

# set editor to neovim
set editor = "nvim"

set my_name = "Gideon Wolfe"
set imap_check_subscribed

# Set preffered view modes
auto_view text/html text/calendar application/ics # view html automatically
alternative_order text/html text/plain text/enriched text/*


# main options
set envelope_from
set edit_headers                     # show headers when composing
set fast_reply                       # skip to compose when replying
set askcc                            # ask for CC:
set fcc_attach                       # save attachments with the body
set forward_format = "Fwd: %s"       # format of subject when forwarding
set forward_decode                   # decode when forwarding
set attribution = "On %d, %n wrote:" # format of quoting header
set reply_to                         # reply to Reply to: field
set reverse_name                     # reply as whomever it was to
set include                          # include message in replies
set forward_quote                    # include message in forwards
set text_flowed
unset sig_dashes                     # no dashes before sig
unset mime_forward                   # forward attachments as part of body
unset help                           # No help bar at the top of index
# set status_on_top                    # Status bar on top of index
set tmpdir = ~/Programs/neomutt/temp # where to keep temp files

unset confirmappend      # don't ask, just do!
set quit                 # don't ask, just do!!
unset mark_old           # read/new is good enough for me
set beep_new             # bell on new mails
set pipe_decode          # strip headers and eval mimes when piping
set thorough_search      # strip headers and eval mimes before searching
set timeout = 0

# status bar, date format, finding stuff etc.
set status_chars = " *%A"
set status_format = "[ Folder: %f ] [%r%m messages%?n? (%n new)?%?d? (%d to delete)?%?t? (%t tagged)? ]%>─%?p?( %p postponed )?"
set date_format = "%d.%m.%Y %H:%M"
set sort = threads
set sort_aux = reverse-last-date-received
set uncollapse_jump
set sort_re
set reply_regexp = "^(([Rr][Ee]?(\[[0-9]+\])?: *)?(\[[^]]+\] *)?)*"
set quote_regexp = "^( {0,4}[>|:#%]| {0,4}[a-z0-9]+[>|]+)+"
set send_charset = "utf-8:iso-8859-1:us-ascii"
set charset = "utf-8"
set arrow_cursor = "no" # Change `color indicator` depending 

# Pager View Options
set pager_index_lines = 10  # Shows 10 lines of index when pager is active
set pager_context = 3
set pager_stop
set menu_scroll
set tilde
unset markers

set mailcap_path = ~/.config/neomutt/mailcap
set header_cache = "~/.cache/mutt"
set message_cachedir = "~/.cache/mutt"

set query_command = "khard email --parsable --search-in-source-files '%s'"

I’ll go over some of the important parts, and the things people might often want to change:

editor is set to neovim, so I can write all my emails with vim.reaper!
query_command hooks in with the contact manager khard, which I wrote about here. Tab complete your contacts!
help is unset so the top bar disappears. Just press ? if you need to see keybindings.
Optionally set status_on_top to see the status bar on top. Personally I like it on the bottom, as that’s where weechat puts it.
arrow_cursor can be enabled to have an arrow point to the currently selected message. I prefer the line as it’s more space efficient.

If you’re curious about any configuration variable, search this page and you can find a description.

Colors

This file contains the color settings for various elements within neomutt.

# vim: filetype=muttrc

# Header colors:
color header blue default ".*"
color header brightmagenta default "^(From)"
color header brightcyan default "^(Subject)"
color header brightwhite default "^(CC|BCC)"

mono bold bold
mono underline underline
mono indicator reverse
mono error bold
color normal default default
color indicator brightyellow default # currently selected message. default makes bar clear, disabled arrow to save space.
color sidebar_highlight red default
color sidebar_divider brightblack black
color sidebar_flagged red black
color sidebar_new green black
color normal brightyellow default
color error red default
color tilde black default
color message cyan default
color markers red white
color attachment white default
color search brightmagenta default
color status brightyellow black
color hdrdefault brightgreen default
color quoted green default
color quoted1 blue default
color quoted2 cyan default
color quoted3 yellow default
color quoted4 red default
color quoted5 brightred default
color signature brightgreen default
color bold black default
color underline black default
color normal default default

color body brightred default "[\-\.+_a-zA-Z0-9]+@[\-\.a-zA-Z0-9]+" # Email addresses
color body brightblue default "(https?|ftp)://[\-\.,/%~_:?&=\#a-zA-Z0-9]+" # URL
color body green default "\`[^\`]*\`" # Green text between ` and `
color body brightblue default "^# \.*" # Headings as bold blue
color body brightcyan default "^## \.*" # Subheadings as bold cyan
color body brightgreen default "^### \.*" # Subsubheadings as bold green
color body yellow default "^(\t| )*(-|\\*) \.*" # List items as yellow
color body brightcyan default "[;:][-o][)/(|]" # emoticons
color body brightcyan default "[;:][)(|]" # emoticons
color body brightcyan default "[ ][*][^*]*[*][ ]?" # more emoticon?
color body brightcyan default "[ ]?[*][^*]*[*][ ]" # more emoticon?
color body red default "(BAD signature)"
color body cyan default "(Good signature)"
color body brightblack default "^gpg: Good signature .*"
color body brightyellow default "^gpg: "
color body brightyellow red "^gpg: BAD signature from.*"
mono body bold "^gpg: Good signature"
mohttps://neomutt.org/code/config_vars.htmlno body bold "^gpg: BAD signature from.*"
color body red default "([a-z][a-z0-9+-]*://(((([a-z0-9_.!~*'();:&=+$,-]|%[0-9a-f][0-9a-f])*@)?((([a-z0-9]([a-z0-9-]*[a-z0-9])?)\\.)*([a-z]([a-z0-9-]*[a-z0-9])?)\\.?|[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)(:[0-9]+)?)|([a-z0-9_.!~*'()$,;:@&=+-]|%[0-9a-f][0-9a-f])+)(/([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*(;([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*)*(/([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*(;([a-z0-9_.!~*'():@&=+$,-]|%[0-9a-f][0-9a-f])*)*)*)?(\\?([a-z0-9_.!~*'();/?:@&=+$,-]|%[0-9a-f][0-9a-f])*)?(#([a-z0-9_.!~*'();/?:@&=+$,-]|%[0-9a-f][0-9a-f])*)?|(www|ftp)\\.(([a-z0-9]([a-z0-9-]*[a-z0-9])?)\\.)*([a-z]([a-z0-9-]*[a-z0-9])?)\\.?(:[0-9]+)?(/([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*(;([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*)*(/([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*(;([-a-z0-9_.!~*'():@&=+$,]|%[0-9a-f][0-9a-f])*)*)*)?(\\?([-a-z0-9_.!~*'();/?:@&=+$,]|%[0-9a-f][0-9a-f])*)?(#([-a-z0-9_.!~*'();/?:@&=+$,]|%[0-9a-f][0-9a-f])*)?)[^].,:;!)? \t\r\n<>\"]"

# Default index colors:
color index yellow default '.*'
color index_author red default '.*'
color index_number blue default
color index_subject cyan default '.*'

# For new mail:
color index brightyellow black "~N"
color index_author brightred black "~N"
color index_subject brightcyan black "~N"

color progress black cyan

Here are some of the highlights (pun very intended):

color indicator changes the color of the highlighted message line, or the arrow if arrow_cursor is enabled.
neomutt does colors by layering. For example you might assign one color to the whole index, and then manually add different colors for new mail, from address, etc.
The various color body settings assign specific colors to anything matching a given regular expression, from list items to emoticons.

These colors look good with my pywal generated themes, and are easy to read without being too hard on the eyes. Here is a reference for coloring the index page if you want to make any tweaks.

Mappings

On any page in neomutt, press ? to get a full list of defined mappings for that page.

neomutt assigns mappings on a view basis. For example you could have a specific set of keybinds for when you are viewing an inbox (the index view) and another set for actually reading a message (pager view).

Many keybinds can be assigned to multiple views by listing the views we want to pick.

Here is the contents of my mappings file:

# vim: filetype=muttrc

# General rebindings
bind attach <return> view-mailcap
bind attach l view-mailcap
bind editor <space> noop
bind pager c imap-fetch-mail
bind index G last-entry
bind index g noop
bind index gg first-entry
bind pager,attach h exit
bind pager j next-line
bind pager k previous-line
bind pager l view-attachments
bind index D delete-message
bind index U undelete-message
bind index L limit
bind index h noop
bind index l display-message
bind browser h goto-parent
bind browser l select-entry
bind pager,browser gg top-page
bind pager,browser G bottom-page
bind index,pager,browser d half-down
bind index,pager,browser u half-up
bind index,pager R group-reply
bind index \031 previous-undeleted	# Mouse wheel
bind index \005 next-undeleted		# Mouse wheel
bind pager \031 previous-line		# Mouse wheel
bind pager \005 next-line		# Mouse wheel
bind editor <Tab> complete-query


# sidebar mappings
bind index,pager \Ck sidebar-prev
bind index,pager \Cj sidebar-next
bind index,pager \Co sidebar-open
bind index,pager \Cp sidebar-prev-new
bind index,pager \Cn sidebar-next-new
bind index,pager B sidebar-toggle-visible

# global index and pager shortcuts
bind index,pager @ compose-to-sender
bind index,pager D purge-message
bind index <tab> sync-mailbox
bind index <space> collapse-thread

# Email completion bindings
bind editor <Tab> complete-query
bind editor ^T complete

# Press A to add contact to Khard address book
macro index,pager A \
    "<pipe-message>khard add-email<return>" \
    "add the sender email address to khard"

## Shortcuts
macro index,pager <f2> '<sync-mailbox><enter-command>source ~/.config/neomutt/accounts/gmail<enter><change-folder>!<enter>'
macro index,pager <f3> '<sync-mailbox><enter-command>source ~/.config/neomutt/accounts/main<enter><change-folder>!<enter>'
macro index,pager <f4> '<sync-mailbox><enter-command>source ~/.config/neomutt/accounts/school<enter><change-folder>!<enter>'

The main differences between my config and the default:

vim keybinds! k and j to go up and down, and l and h to move forwards and back pages.
Open the sidebar with B. Ctrl+j and Ctrl+k to navigate, Ctrl+o to open selection
A will attempt to create a new contact with khard.
F2, F3, and F4 are used to switch between my three mailboxes

I put the important binds in a little cheatsheet table for myself!

Function	Keybind
Go to last entry	`G`
Go to first entry	`gg`
Page down	`d`
Page up	`u`
Delete message	`D`
Delete thread	`Ctrl+d`
Mark thread read	`Ctrl+r`
Undelete message	`U`
Limit/Filter	`L`
Open message	`l`, `Enter`
Group reply	`R`
Reply	`r`
Forward message	`f`
New message	`m`
View attachments	`v`
Compose to Sender	`@`
Resend message	`Esc+e`
Save message (to file)	`s`
Toggle collapse thread	`Space`, `Esc+v`
Toggle collapse all threads	`Esc+V`
Add contact to Khard	`A`
Jump to parent message	`P`
Email PGP public key	`Esc+k`

Function	Keybind
Bottom of page	`G`
Top of page	`gg`
Next line	`j`
Previous line	`k`
View attachments	`l`
Page down	`d`
Page up	`u`
Group reply	`R`,`g`
Reply	`r`
Compose to Sender	`@`
Delete thread	`Ctrl+d`
Mark thread read	`Ctrl+r`
Mark thread new	`N`

Function	Keybind
Show sidebar	`B`
Up	`Ctrl+k`
Down	`Ctrl+j`
Open	`Ctrl+o`
Open next new	`Ctrl+n`
Open previous new	`Ctrl+p`

Mailcap

This file defines which programs should handle various filetypes included in email attachments.

# MS Word documents
application/msword; ~/dotfiles/office/view-attachment.sh %s "-" '/Applications/LibreOffice.app'
application/vnd.ms-excel; ~/dotfiles/office/view-attachment.sh %s "-" '/Applications/LibreOffice.app'
application/vnd.openxmlformats-officedocument.presentationml.presentation; ~/dotfiles/office/view-attachment.sh %s "-" '/Applications/LibreOffice.app'
application/vnd.oasis.opendocument.text; ~/dotfiles/office/view-attachment.sh %s "-" '/Applications/LibreOffice.app'

# HTML
text/html; w3m -I %{charset} -T text/html; copiousoutput;
text/plain; nvim %s

#PDFs
application/pdf; /usr/bin/zathura %s pdf

#Images
image/png; /usr/bin/feh %s
image/jpeg; /usr/bin/feh %s

I use feh as a lightweight image viewer
All MS office docs are handled by LibreOffice
zathura reads my PDFs (with style)
HTML is rendered as text with w3m

Accounts

I won’t go over the specifics of each account here, because getting neomutt to actually talk to your mailserver is another matter entirely, for which there are dozens of tutorials.

Here is a generic template for hooking up an IMAP account.

# vim: filetype=muttrc
unmailboxes *

# in/out mail servers
set realname = "Firstname Lastname"
set imap_pass = "SOME_PASSWORD"
set smtp_url = "smtps://you@somedomain.com@mailserver.com:465"
set smtp_pass = "SOME_PASSWORD"
set folder = "imaps://you@somedomain.com@mailserver.com:993"
set spoolfile = "+INBOX"

set from = "you@somedomain.com"
set envelope_from
set use_from = "yes"
set record = "+Sent"
set trash = "+Trash"
set postponed = "+Drafts"
set mail_check = 6

# Allow Mutt to open a new IMAP connection automatically.
unset imap_passive

# Keep the IMAP connection alive by polling intermittently (time in seconds).
set imap_keepalive = 300

## Hook -- IMPORTANT!
account-hook $folder "set imap_pass=SOME_PASSWORD"

unmailboxes * hides the inboxes of accounts we are not currently viewing.
account-hook is the line that allows us to switch to this account with a keybind.

Conclusion

And that’s basically it. If you have done everything correctly, you should be able to use neomutt to interact with multiple email accounts, switching with just a keypress.

The colors should look good, and the keybinds should feel natural to a vim user. I am constantly improving this setup so don’t hesitate to throw me some suggestions.

Agents Standing By: Monitoring National Emergencies

Fri, 27 Mar 2020 00:00:00 +0000

This scenario we will go over today will allow you to set up real time alerts as FEMA updates its online database of natural disasters and emergencies.

Firstly, we are going to need to set up a Website Agent. This will be similar to our previous agent, as it will extract the target data from a json data stream.

Let’s start by messing around with the API a little bit. Take a look at the full documentation if you have questions, but this should introduce you to the basics.

We can take a look at all the APIs offered by FEMA. Let’s select the Disaster Declarations Summaries - V2 dataset.

Under the Data Fields dropdown, we can see all of the various metrics returned for a disaster declaration. To view the data, we can visit the link under the API Endpoint dropdown.

This will display ALL disaster declarations. For our case, we may want to provide some filters to this data. Let’s narrow this down to our state shall we? For my cases, I will change the URL to the following

https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries?$filter=state eq ‘WA’

And then we can use a second URL to provide a list of all terrorist attacks nationwide;

https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries?$filter=incidentType eq ‘Terrorist’

We can create website agents to monitor any number of specific queries. We could narrow down the list arbitrarily, say to Floods in Oregon from 1980 onwards. Take full advantage of the URI Commands section on the documentation, which details these filtering options.

The json for one of these agents is shown below

{
  "expected_update_period_in_days": "2",
  "url": "https://www.fema.gov/api/open/v2/DisasterDeclarationsSummaries?$filter=incidentType%20eq%20%27Terrorist%27",
  "type": "json",
  "mode": "on_change",
  "extract": {
    "declarationTitle": {
      "path": "DisasterDeclarationsSummaries.[*].declarationTitle"
    },
    "state": {
      "path": "DisasterDeclarationsSummaries.[*].state"
    },
    "declarationDate": {
      "path": "DisasterDeclarationsSummaries.[*].declarationDate"
    },
    "incidentType": {
      "path": "DisasterDeclarationsSummaries.[*].incidentType"
    },
    "designatedArea": {
      "path": "DisasterDeclarationsSummaries.[*].designatedArea"
    },
    "placeCode": {
      "path": "DisasterDeclarationsSummaries.[*].placeCode"
    }
  }
}

This will extract values from the data like placeCode, incidentType, and more which can all be used as variables to format your notifications.

In this scenario I have the notifications sent via email and push notification. I will show how you can format the email to correctly display information passed by the website agent.

{
  "subject": "{{incidentType}} Alert!",
  "expected_receive_period_in_days": "2",
  "from": "Huginn@somedomain.com",
  "body": "<h1>{{declarationTitle}}<\/h1><br>Alert! <br><ul><li>State: {{state}}<\/li><li>Date Declared: {{declarationDate}}<\/li><li>Area: {{designatedArea}}<\/li><li>ZIP: {{placeCode}}<\/li><\/ul>"
}

Now I certainly could have passed the original event through a Event Formatting agent to format the message before it hit the email agent. But I decided it would be easier to simply write the formatting inline in this case.

Notice that I use HTML tags like <h1> and <br>. This is extremely useful for formatting emails (especially digests) where you might want a visual indicator of how information is separated.

We also use liquid formatting to include variables passed by the incoming event, like {{declarationTitle}}. The end result of this scenario is an email with all the information, for example;

COVID-19 PANDEMIC

Alert!

    State: WA
    Date Declared: 2020-03-22T16:21:00.000Z
    Area: Chelan (County)
    ZIP: 99007

NOTE!

These listeners might appear to be Not Working according to Huginn. I believe this happens if the json source goes a long time without updating, or more days than allowed in expected_receive_period_in_days. Be assured the agent will function when the actual data source changes.

Conclusion

Just by poking around the API documentation from FEMA, we are able to scrape tons of valuable data relating to events happening in the country. No longer be reliant on news reports, be the first to know.

This technique can be applied to countless open data sources and incorporated into your informational web!

Agents Standing By: Monitoring Website Status

Wed, 25 Mar 2020 00:00:00 +0000

Intro

One thing you might want to get a notification for immediately is when one of your websites or services is malfunctioning. Luckily, huginn has agents built to handle usecases like this.

Creating our Status Agent

For each service you want to monitor, you are going to want to create a new HttpStatusAgent with the url field pointing towards the site you are monitoring. I decided to enable redirect follows, so I can see the true HTTP code my site is returning.

Additionally, I set changes_only to true. That means that an event is only created by this agent when the HTTP status code changes from its previous value (a 200 success code).

The changes_only field is very common with agents, and prevents your agent from re-emitting tons of duplicate events.

The JSON for such an agent looks like this:

{
  "url": "https://www.somedomain.com",
  "disable_redirect_follow_radio": "false",
  "disable_redirect_follow": "false",
  "changes_only_radio": "true",
  "changes_only": "true",
  "headers_to_save": ""
}

When creating a huginn agent, it is extremely helpful to look at the agent documentation to the right of the page. Any confusing fields should be explained there.

Creating our Alert Agent

Now that we have an agent that creates events when the status of your site changes, we now want an agent to act upon those events and notify you in your preferred manner.

Here we will set up a simple Email Agent to notify us of a service disruption. Take a look at the complete list of agents for more ideas about notifications.

We want to make sure the Event Sources field is populated with the status agent we made in a previous step. This means the status agent sends its events to the alert agent.

Set the subject and headline field to whatever you want. In the below json code you will notice that I used variables like {{url}}. If we look in the bottom right of the documentation for our agent, we can see a field that shows what data fields we are expecting to see from our source status agent.

{
  "subject": "Service Interruption!",
  "headline": "Your notification:",
  "expected_receive_period_in_days": "2",
  "body": "{{url}} is returning status code {{status}}.",
  "from": "Huginn@somedomain.com"
}

We can reference these fields in our receiving agent, using what is called liquid formatting. This allows us to create reusable, modular agents.

We can create many status agents, and because they will all output an event of the same format, we can have them all pointing to the same alert agent.

Here I have a push notification set up in tandem with email, because I want to know about these alerts ASAP.

Conclusion

With just two agent types we were able to create a robust HTTP monitor that will format and send notifications whenever a service is experiencing disruption.

In later posts we will build up to more complex scenarios with more moving parts.

Agents Standing By: Setting Up Huginn

Wed, 25 Mar 2020 00:00:00 +0000

Intro

In my last post, I went over what Huginn is used for, and how I implement it in my workflow. Today we are going to go over the initial set up of a basic Huginn container as well as an SMTP container to send our emails.

Docker setup

Here’s the docker compose file I am currently using for my Huginn instance:

version: '3.5'
services:
  huginn:
    image: huginn/huginn
    restart: always
    ports:
      - 4750:3000
    volumes:
      - $HOME/data/programs/docker/huginn/config:/var/lib/mysql
    environment:
      # Email Configuration
      EMAIL_FROM_ADDRESS: "someemail@somedomain.com"
      SMTP_USER_NAME: "none"
      SMTP_PASSWORD: "none"
      SMTP_SERVER: "mail"
      SMTP_PORT: "25"
      SMTP_AUTHENTICATION: "none"
      SMTP_ENABLE_STARTTLS_AUTO: "true"

  mail:
    image: bytemark/smtp
    restart: always

The important parts here are the config volume mounted to the huginn container which will store our persistent data, and the SMTP server that simply listens for and sends emails sent by huginn.

This setup should run fine after a docker-compose up, -d optional. Now navigate to your huginn instance on the machine you are running it on at port 4750.

Navigate to the Agents tab and start looking around at some of the Default Scenario agents. Pay attention to the layout and documentation of each agent, because in my next post er will be doing some agent configuration.

Agents Standing By: Introduction to Huginn

Mon, 23 Mar 2020 00:00:00 +0000

What is Huginn?

Have you ever felt out of the loop on current events? Do you tire of manually checking dozens of sites every day for the information you need? Do you wish you could quickly whip up a bot without learning a new API and programming language?

Before last week, I had heard of Huginn, but not seen how it could fit into my workflow. Now that I have a better understanding of its features, allow me to explain the ways Huginn can improve the way you monitor important data.

At its core, Huginn is an event monitoring framework. Huginn operates with something called agents. These agents are analogous to a function in programming.

An agent will listen for a specific type of data (known as an event), preform some logic based operations with that data, and then emit another event with the processed data.

On their own, the agents have limited capability. The true power of Huginn is leveraged when we combine agents, into something called a scenario. A scenario is a tagged collection of agents, which usually communicate with each other to perform more complex tasks.

Let me go over an example scenario so you can get an idea of how agents interact

The top level shows three Website Agents. These scrape the sites of your choice and extract data from HTML, JSON, or XML. I have these website agents monitoring a JSON feed of cryptocurrency, market indexes, and stocks.

Here is the JSON for the stock monitoring agent.

{
  "expected_update_period_in_days": "2",
  "url": "https://financialmodelingprep.com/api/v3/quote/AAPL,FB",
  "type": "json",
  "mode": "on_change",
  "extract": {
    "price": {
      "path": "[*].price"
    },
    "symbol": {
      "path": "[*].symbol"
    },
    "name": {
      "path": "[*].name"
    },
    "changesPercentage": {
      "path": "[*].changesPercentage"
    }
  }
}

These JSON feeds get sent to the middle layer of Trigger Agents. These agents take the incoming event (in this case JSON stock data) and evaluate it based on a value comparison or regex.

{
  "expected_receive_period_in_days": "2",
  "keep_event": "false",
  "rules": [
    {
      "type": "field>=value",
      "value": "10.00",
      "path": "changesPercentage"
    }
  ],
  "message": "{{name}} changed {{changesPercentage}}%. Current rate is {{price}}$.",
  "must_match": "1"
}

I have them set to extract the changesPercentage field from the incoming event and re-emit an event if the symbol fluctuated +/- 10%.

When these trigger agents emit an event, it is a stock fluctuation I want to know about immediately. That is why these events are sent to two notification agents. One is an email, which I will see whenever I check it. I also send a push notification to myself so I can see the events as they happen.

What about Grafana?

The way I see it, Huginn and Grafana fill very separate rolls in my workflow. Grafana (and the whole TIG stack) excel at monitoring lots of different data points over time and visualizing them beautifully. For example I could view how my hard drives are filling up over time, or which docker container is causing my CPU to throttle.

Huginn is great for on the fly creation of “bots” that will monitor very specific data points and perform actions based on them. Maybe I want an text whenever a specific model of CPU goes on sale. Or maybe I want an XKCD comic and a weather report delivered to me in a daily morning email. This is where Huginn shines.

However, I plan on experimenting with combining the two. What if I used Huginn to monitor and format a web data source, and send that nice formatted data to Grafana to display? I am still trying to figure out how this would work. I have attempted to use the JSON Datasource on Grafana, but I have not been able to get it working even once.

Does Grafana require you to store the data in some external DB? I could make the Huginn data go to InfluxDB, but it seems like it would be a lot of extra work.

Conclusion

In my next post, I will go over installing Huginn using docker-compose and setting up the basic SMTP server which allows for email notifications. Then we will get into setting up some useful agents and getting used to the Huginn workflow.

Keeping your docker containers updated with Watchtower

Thu, 19 Mar 2020 00:00:00 +0000

Who needs this?

I for one have found myself guilty of neglecting my containers once they’re up and running. Not only am I missing out on critical security patches, but many quality of life improvements made to my programs since I initially set them up.

However I didn’t want to cause downtime for my users by updating during peak usage hours, or even break something by doing something sloppily.

That’s where Watchtower comes in. Watchtower constantly checks docker containers for updates to their image. What it does next is up to you. By default, it will stop, update, and restart the containers. It can also be set to simply download the image and notify you.

Setting it up

This is one of the most painless setups I’ve experienced. This docker-compose file contains everything I used to get it working.,

version: '3'
services:
   watchtower:
      image: containrrr/watchtower
      container_name: watchtower
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
      command: --schedule "0 0 3 * * ?"

You have to mount the docker socket in order for Watchtower to be able see the other containers. I added an optional --schedule command, which takes a cron expression as its input. Here I have Watchtower scheduled to do its business at 3am, a time when I expect little to no traffic.

Learn more about the configuration options here.

One great use case is if you’re working with your own docker images/registries, you can integrate watchtower into your CI/CD pipeline to have the newest image pulled into production as soon as it is deployed.

Using preset layouts in i3

Tue, 17 Mar 2020 00:00:00 +0000

While I already knew how to launch programs with my i3 config file, I was lacking a way to predefine specific workspace layouts that I would want to launch on startup.

That’s where i3 layouts come in handy.

How to create a layout

The first step in creating a layout is to spawn the windows you want to be launched when the layout is deployed.

In this example I created a monitor layout, which I use to monitor system stats, crypto prices, the news, and more.

We can save the layout of the target workspace into a json file with the following command:

i3-save-tree --workspace 8 > ./monitor.json

In this command 8 is the number of the workspace I wanted to capture.

Opening this file should give you a json layout that includes the programs and position of your windows.

{
    // tabbed split container with 4 children
    "border": "pixel",
    "floating": "auto_off",
    "layout": "tabbed",
    "percent": null,
    "type": "con",
    "nodes": [
        {
            "border": "pixel",
            "current_border_width": 7,
            "floating": "auto_off",
            "geometry": {
               "height": 614,
               "width": 914,
               "x": 0,
               "y": 0
            },
            "name": "gotop",
            "percent": 0.25,
            "swallows": [
               {
                "class": "^URxvt$",
                "instance": "^gotop-startup$"
               }
            ],
            "type": "con"
        },
        {
            "border": "pixel",
            "current_border_width": 7,
            "floating": "auto_off",
            "geometry": {
               "height": 614,
               "width": 914,
               "x": 0,
               "y": 0
            },
            "name": "newsboat",
            "percent": 0.25,
            "swallows": [
               {
                "class": "^URxvt$",
                "instance": "^newsboat-startup$"
               }
            ],
            "type": "con"
        },
        {
            "border": "pixel",
            "current_border_width": 7,
            "floating": "auto_off",
            "geometry": {
               "height": 614,
               "width": 914,
               "x": 0,
               "y": 0
            },
            "name": "cointop",
            "percent": 0.25,
            "swallows": [
               {
                "class": "^URxvt$",
                "instance": "^cointop-startup$"
               }
            ],
            "type": "con"
        },
        {
            "border": "pixel",
            "current_border_width": 7,
            "floating": "auto_off",
            "geometry": {
               "height": 614,
               "width": 914,
               "x": 0,
               "y": 0
            },
            "name": "mop",
            "percent": 0.25,
            "swallows": [
               {
                "class": "^URxvt$",
                "instance": "^mop-startup$"
               }
            ],
            "type": "con"
        }
    ]
}

We can save this json file in a safe place for later use.

Deploying a layout

Let’s say we want this layout to launch when we log into i3. First, we need to add the following command to your i3 config file.

exec --no-startup-id "i3-msg 'workspace 8: Monitor ; append_layout /home/gideon/.config/i3/layouts/monitor.json'"

Here we would replace workspace 8: Monitor  with the number and name of the workspace where you want this layout to be deployed.

We would also change the filepath to reflect the actual location of the file where you stored your layout.

What this will do is open up a series of windows corresponding to the position defined in the json file. However, these windows will not be populated with any programs.

These windows are placeholders, and as soon as they see a specific program has been launched, it swallows it into that window.

So to get the programs to launch into our layout, we can launch them with our i3 config. Here is an example for urxvt

What this will do is open up a series of windows corresponding to the position defined in the json file. However, these windows will not be populated with any programs.

These windows are placeholders, and as soon as they see a specific program has been launched, it swallows it into that window.

So to get the programs to launch into our layout, we can launch them with our i3 config. Here is an example for urxvt.

exec i3-msg 'exec urxvt -name cointop-startup -e fish -c cointop'
exec i3-msg 'exec urxvt -name gotop-startup -e fish -c gotop'
exec i3-msg 'exec urxvt -name mop-startup -e fish -c mop'
exec i3-msg 'exec urxvt -name newsboat-startup -e fish -c newsboat'

Conclusion

If you know you want a certain series of windows opened up in a certain way, i3 layouts are for you.

I only covered launching a layout as i3 starts, but you could just as easily bind a hotkey to start a layout for a specific task.

Adding RSS to your Hugo Site

Tue, 11 Feb 2020 00:00:00 +0000

I recently received a request from a reader to make a working RSS feed for my website.

Even though I have posted about RSS and newsboat before, I had no idea that Hugo autogenerates an RSS feed when your website is built.

The only issue is that I had a couple things misconfigured, so not everything was working as desired. In this post I will describe how I went about fixing my RSS feed.

Editing config.toml

The biggest misconfiguration I had was setting baseurl="/" in my config.toml. This caused all links in my generated website to be relative paths.

For example, a link to an image on the site would be /img/category/image.jpg instead of https://gideonwolfe.com/img/category/image.jpg.

This made browsers fail to open the page when opening from an RSS reader, because the post URL was bad. Let’s change this by pointing baseurl to the actual address of your site. This should make the RSS feed at index.xml include valid links.

Next you can the following to the file so you show up as the “author” for all your posts.

[author]
    name = "My Name Here"

Showing all your content

I really like reading the entirety of an article in my RSS reader if I can. It removes all the bloat of the original website, and just gives me the text I care about.

By default, Hugo only includes a brief summary of your post in every RSS item. It took me a second to figure this out, but it’s quite a simple fix.

Much like how we overrode our themes header.html by copying it to <HUGO ROOT>/layouts/partials/header.html, we can override the default rss.xml by copying it to <HUGO ROOT>/layouts/_default/rss.xml and editing it.

{{- $pctx := . -}}
{{- if .IsHome -}}{{ $pctx = .Site }}{{- end -}}
{{- $pages := $pctx.RegularPages -}}
{{- $limit := .Site.Config.Services.RSS.Limit -}}
{{- if ge $limit 1 -}}
{{- $pages = $pages | first $limit -}}
{{- end -}}
{{- printf "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"yes\"?>" | safeHTML }}
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>{{ if eq  .Title  .Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} on {{ end }}{{ .Site.Title }}{{ end }}</title>
    <link>{{ .Permalink }}</link>
    <description>Recent content {{ if ne  .Title  .Site.Title }}{{ with .Title }}in {{.}} {{ end }}{{ end }}on {{ .Site.Title }}</description>
    <generator>Hugo -- gohugo.io</generator>{{ with .Site.LanguageCode }}
    <language>{{.}}</language>{{end}}{{ with .Site.Author.email }}
    <managingEditor>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</managingEditor>{{end}}{{ with .Site.Author.email }}
    <webMaster>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</webMaster>{{end}}{{ with .Site.Copyright }}
    <copyright>{{.}}</copyright>{{end}}{{ if not .Date.IsZero }}
    <lastBuildDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</lastBuildDate>{{ end }}
    {{ with .OutputFormats.Get "RSS" }}
	{{ printf "<atom:link href=%q rel=\"self\" type=%q />" .Permalink .MediaType | safeHTML }}
    {{ end }}
    {{ range $pages }}
    <item>
      <title>{{ .Title }}</title>
      <link>{{ .Permalink }}</link>
      <pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
      {{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
      <guid>{{ .Permalink }}</guid>
      <description>{{ .Content | html }}</description> 
    </item>
    {{ end }}
  </channel>
</rss>

The only thing that I did was change the word .Summary to .Content near the bottom of the file. Now my posts display the full text when opened in an RSS reader.

Conclusion

Now you should be able to view your site in an RSS reader, just like your other favorite blogs and websites.

Adding Google Analytics to your Hugo Site

Mon, 10 Feb 2020 00:00:00 +0000

When I first created this site, I really wasn’t expecting much traffic. But since that time, I have become aware of many backlinks pointing this way, and I wanted to find out some statistics about the visitors to my site.

The solution to this came when a friend of mine told me about Google Analytics. Being a Google product, the primary purpose of this tool is to collect metrics on your users so you can more effectively cram ads down their throats.

Since I have no interest in advertising on this site, I will only be making use of the basic analytics features.

Set up your Google Analytics account

This part is pretty straightforward, and took me about 5 minutes to do.

Head over to https://analytics.google.com/ and make an account (or sign in with your Google account).

Set up your “Property”, give it a name, and point it to the URL of the site you plan on tracking.

Click through the basic options until you land on a page with a Tracking Code. This is the value we need to be keeping track of.

If you’re having trouble with this step then I recommend checking out this tutorial, which has very detailed step by step instructions for configuring your GA account.

Configuring Hugo

Thankfully, Hugo has a built in template for Google Analytics. All we need to do is make sure the template gets included in all our pages, so we can have more detailed tracking.

header.html

The easiest way to do this is to insert the template into the header.html file used by your Hugo theme of choice.

Since it is common to use a git submodule as a theme directory, it could be undesirable and messy to write changes directly into the submodule.

The solution to this is to create another directory layouts in the root of your hugo directory. Hugo looks at the contents of this folder and uses it to overwrites the styles defined by your theme.

For example I use the Terminal theme. The header file is located at <HUGO ROOT>/themes/terminal/layouts/partials/header.html

We can copy this file to our new directory, so it sits at <HUGO ROOT>/layouts/partials/header.html.

Next we can edit our header.html file to include the template for google analytics. Your header.html will look different depending on the theme you’re using, but there should be a <head> section at the top. If not we can simply create it.

<head>
  {{ template "_internal/google_analytics.html" . }}
  {{ template "_internal/google_analytics_async.html" . }}
</head>

Adding these two lines inside <head> will make the google tracking code embed in the HTML of all of your pages.

config.toml

Finally we need to use that tracking code from earlier. At the top level of your config.toml, add the line

googleAnalytics = "UA-302012394-1"

Replacing the string with your Google Analytics tracking code. After you rebuild the site, everything should work as expected! You should also be able to detect the traffic by running the server locally as well, before you push to production.

Note!

I had quite the time figuring out why I couldn’t see any activity on my site. Being the paranoid privacy nut that I am, I have a multitude of tracker/script blocker plugins on my browsers, to block nefarious ads/scripts.

This also blocks the tracking I do want, wouldn’t you know. I switched over to a private window without plugins and the traffic was instantly visible!

Conclusion

Google Analytics is probably the most painless way to get in depth tracking for your Hugo site. In a future post I want to investigate ways to get all that beautiful data into Grafana, the tool built for scenarios like these.

vim.reaper: 💀 What's new?

Sat, 01 Feb 2020 00:00:00 +0000

What is vim.reaper?

vim.reaper is simply my publicly available neovim configuration that includes advanced functionality out of the box, is easy to modify to your usecase, and looks great with your system theme.

If you are a fan of customizing your dotfiles like me, you may find it helpful to have a solid jumping off point for an ambitious project as a fully customized vim setup.

I spent more time than I would like to admit trawling through the documentation for various plugins, finding the best ones, and configuring them to my preferences.

If your preferences don’t happen to match mine, vim.reaper allows you to easily find the configuration of choice and modify it to your hearts content.

Lazygit

Now I know some people swear by the command line git (which is what I use 99% of the time), but every once in a while it can be nice to have a visual overview of your current git environment, which can get quite complicated at times.

Even the built in git keybinds vim.reaper already has can only get you so far.

This is where lazygit comes in. This neat TUI provides you with a visual interface to manage your current git environment. Commits, merges and more can be managed with common sense keybinds.

I don’t need fancy VCS integration into my IDE, but a quick interface like lazygit really comes in handy!

Lazydocker

From the developer of lazygit comes lazydocker, a TUI for managing your docker environment.

Like lazygit, lazydocker opens in a centered floating window and uses common sense keybinds to allow you to quickly peruse your images, containers, volumes, configurations and more.

This is an invaluable tool for my media server, allowing me to check the logs and status of every container in the stack, with informative graphs and stats.

lazygit and lazydocker can be opened with <Leader>lg and <Leader>ld respectively.

vimtex

If you take any upper level Math/CS class, or research in pretty much any STEM field, you are bound to encounter or be required to produce LaTeX documents at some point.

LaTeX is an extremely powerful typesetting engine, but it can be quite a pain to learn at first.

Most people rely on online tools like overleaf which can be helpful, but I prefer a workflow where I can actually work on my projects if I have no internet, or if overleaf is down for whatever reason.

The plugin suite I have implemented in vim.reaper provides an overleaf like workflow, complete with automatic document compilation and linting.

Simply open a LaTeX document, press <Leader>lc to start compilation, and see the pdf open in your configured reader, I use zathura.

As you can see on the left side of the image, neovim is more than happy to scold me for my sloppy syntax, which is helpful for a learner like me.

Markdown preview

For years I struggled to find a way to effectively take notes on my computer, because every program seemed far too clunky to accommodate the variety of structures and equations I would need to record.

Some true gods take notes in pure LaTeX, but that seems a bit excessive, and I would waste most my time debugging errors instead of listening to the lecture.

The workflow I settled on was the same one I use to write this very blog. I use neovim to write markdown, and have it previewed to me live on a local webserver.

This setup is essentially the best of both worlds. These markdown files can be opened and read by any markdown viewer, or just read as plain text files.

What’s more, I can use any of the extremely flexible markdown syntax that suits my needs, including support for LaTeX rendering, diagrams, code blocks, and more.

To make the whole setup perfect, I manually modified the look of the markdown preview to take colors from wal, meaning it will look just like any other program in your cohesively themed system.

Finally, a setup that allows me to take the notes I need without being slowed down by cumbersome software.

Translation

Now there is a configurable translation plugin! Run your text of choice through your favorite translation engine.

You can set the output to display in a window, or simply replace the target text in your document.

Misc

Taking advantage of this fancy new centered floating window patch, vista fzf now Opens in a centered window instead of at the bottom of the screen.

Conclusion

I am always looking to improve this setup, so don’t hesitate to send a PR if you have a good idea. I’m looking forward to continued development of this robust configuration.

Chat like it's 2000 2020: Using IRC in the modern world

Fri, 17 Jan 2020 00:00:00 +0000

Have you ever been frustrated with the bloated nature of todays favorite chat applications? I for one prefer the simplicity of a text based chat to all the bells and whistles provided by competitors.

On IRC you can browse thousands of publicly listed chatrooms, many tailored to the exact software or system you happen to be using. While IRC might not be a drop in substitute to more fleshed out teamchat programs, nothing beats hopping into a chat room full of strangers to troubleshoot and discuss whatever the topic of the room may be. Think subreddits, but in real time.

Now currently I’m not in a position where I’m forced to use a proprietary chat software at work. Thusly I have not yet been forced to configure my IRC clients to work with other protocols like Slack or Steam.

If and when I am put into that unfortunate position, you can expect a tutorial on configuring bitlbee!

The IRC Client

To keep consistency in my terminal workflow, I opt to use Weechat, a fast and configurable terminal IRC client.

The configuration for weechat is crazy complex, so I will leave the custom settings for another post. Let’s address the basics. On my left side there is a list of active buffers. A buffer can be a public channel, a PM with another user, or even a window to issue settings and commands.

The networks I am connected to are the farthest to the left, and indented under them are the channels to which I am connected.

If a channel is highlighted pink, it means there has been activity since I last saw it. The buffer I am currently attached to is denoted with the >> before it.

Adding networks

In today’s climate, the go to IRC network is freenode, having by far the most active channels.

I also opt to join the snoonet network, which hosts channels for many popular subreddits.

You can browse thousands of channels here, with topics spanning all hobbies and professions (with a heavy tech slant, seeing as this is IRC after all).

I will quickly break down how to get up and running with freenode and weechat.

/server add freenode chat.freenode.net

/connect freenode

You should be connected to freenode. Now we need to register a nickname.

Change the your nickname to your desired one with this command

/nick <nickname>

Assuming the nick is available, you can register it with

/msg NickServ REGISTER password youremail@example.com

Assuming you are only using IRC on one machine at a time, and only plan on using weechat, you can go ahead and start joining channels with /join #channel.

ZNC

This wasn’t enough for me. I wanted to be able to access IRC on my phone, and see the history of conversations I had missed, or carry on a conversation if I had to leave the house. These are default features for any modern chat client.

As it stands now, you are only allowed to log in to your nickname from one client at a time, throwing a wrench in the above requirements. Even if I diligently logged in and out of each client anytime I went anywhere, I would be joining the channels without any context of the conversation currently in place.

That’s where an IRC bouncer comes in. A bouncer keeps a constant connection to your favorite networks and channels, allowing you to connect to your bouncer with as many clients as you please.

Not only that, there are a host of additional features such as logging and support for multiple users! The bouncer I am using is ZNC.

Normally, I would reverse proxy my ZNC traffic with Traefik. Unfortunately, Traefik seems to have a real tough time with TCP, and the documentation is just not there yet for me to get this working. Many others I talked to shared my frustrations.

Because of this, I am simply exposing the port on the container and then forwarding the port on my router to the IRC entrypoint.

The docker compose section for ZNC is as follows:

   znc:
      image: linuxserver/znc
      container_name: znc
      environment:
         - PUID=1000
         - PGID=1000
         - TZ=Europe/Los_Angeles
      volumes:
         - $HOME/data/programs/docker/znc/config/:/config
      ports:
         - 4798:4798 # Port bind for IRC only traffic
         - 6501:6501 # Only exposed locally for web admin

I opted to create a second port bind in ZNC, which I expose to the web. This only allows IRC traffic, nobody will mess with the config page.

Here you can see the ZNC settings page, where you can configure users, the networks they belong to, and the channels they should connect to. Scrolling down even further we can see a list of modules, their arguments and their status.

Now we want to connect to our ZNC instance from Weechat! That can be done with the following command:

/server add ZNC-Network <znc address>/<port> -username=user/Network -password=<znc pass> -autoconnect

If you are connected to multiple IRC networks through ZNC, you need to add a separate ZNC connection for every network. For example for freenode, I would do the following:

/server add ZNC-Freenode irc.mydomain.tld/4798 -username=gideon/Freenode -password=<znc pass> -autoconnect

Note the name of the network (defined in ZNC settings) becomes part of the username argument. Different clients will have different ways of Handling this.

If everything works correctly, your client should rapidly connect to all the channels you configured. Grab your favorite mobile IRC client (there is a mobile version of Weechat) and connect to the ZNC! Now you can take all your conversations on the go.

Conclusion

IRC is a super valuable resource for discussing and troubleshooting various topics with knowledgeable people from around the world!

This guide is meant to outline the steps to creating a seamless IRC experience in the 21st century, while keeping what makes IRC special in the first place. In another tutorial I will go into some advanced Weechat configuration to bring out the best of the client!

Quantifying Your life: Advanced TIG Techniques

Wed, 01 Jan 2020 00:00:00 +0000

Introduction

Happy 2020! I can’t think of a better way to welcome in the new decade than taking control of your technical and personal life through hard data. In my last post I went over how to setup the TIG stack, a software ecosystem that collects, stores, and displays data collected from a variety of sources.

In this post, I will go over some more advanced techniques for Grafana in particular, and how to get custom charts and graphics specifically tailored to your needs. Let’s get right into it!

Monitoring Docker

Since all of my mission critical software runs in docker containers, I thought my first priority should be to make sure I can see how all the containers are doing individually. I started searching the Grafana Dashboard website and started messing with some of the docker ones users had made. I’ll walk you through one I took and modified to suit my needs.

Defining variables

Grafana has a great feature where you can query your database and save the result to a dashboard variable. In this example, we are going to get the name of all the containers so we can make one row and duplicate it for every docker container.

The top query will return a list of the current docker containers.

Crafting Queries

Now we can start creating panels using this variable

If we start creating more panels to show more stats about a container, we can put it in a row and duplicate it for every instance of the Container variable.

Jellyfin isn’t logging anything interesting at the moment, but all these metrics are helpful to have on hand if you are experiencing issues with any of your services.

System Metrics

Much like monitoring docker containers, monitoring your actual hardware mostly involves clicking around the query constructor to see what fields are available in the database. I organized my system metrics dashboard in a sane way, with a variable for the host so I could reuse the same dashboard to monitor different hosts.

Make sure to check out the full list of Grafana plugins to see if there are any visualizations that make sense for your usecase. I love the singlestats and guages, which allow you to see the current status of a metric at a quick glance without worrying about temporal changes.

I also prefer a black background for most of my dashboards, with transparent panels on top. I guess that’s because I’m 1337. Or I don’t have better design ideas. Let’s just go with the first one.

Flowcharts

Speaking of third party plugins, allow me to introduce the flowcharting plugin. This plugin allows you to create custom xml diagrams using draw.io, and link the xml elements to database queries.

If all that sounded like nonsense, let me give you an example. Remember the cover of this article? That’s a neat little diagram of my current docker stack.

What if I told you this was more than a static diagram? By simply hovering my mouse over a container, I am presented with at-a-glance statistics about that container.

How does this work? Well it took me a bit of trial and error to get this working as it is a bit unintuitive at first. First, we have to create database queries for everything we want to measure. This is done from within the “Queries” menu of your new flowcharting panel.

There could be a lot of queries. I was monitoring the CPU and memory usage of 12+ containers, so plugging in all the queries can get tedious.

Now we go back to the “Visualization” section of the panel. Here we can click the link to open draw.io and edit our XML diagram to your hearts content. Once you’ve done this, scroll down to the “rules” section.

Here we will need to create a new rule for every metric we want to monitor. In the example below, I am setting up a rule to monitor the CPU usage of Lidarr, like the query from the previous image.

In the “Apply to metrics” field, we use the alias names of one of the queries we created in the previous menu. We set Aggregation to “Last” to collect the last stored value in our database.

Next we can set some value thresholds, which define acceptable levels for any particular metric. In this example, any CPU usage above 50% would be a warning level, but that would change to critical if the CPU usage exceeded 80%.

On the lower half of the rule menu, we can enable the on-hover tooltip metrics, along with an optional graph. I like to have the graph colored to whatever state the rule is in. That way, I can have a visual representation of any issues in my system.

Not only can you hover over objects to get information, you can color the objects too!

Here I changed the warning threshold to 1% to show you what it looks like if I was at a high level of memory usage. The box turned the warning color (whatever you define it to be) and flashes an icon (if enabled).

I set the Traefik container and the surrounding box to always display the state color, so I could always be sure my reverse proxy was functioning intentionally.

Other data sources

Getting data from Telegraf is great, but InfluxDB and Grafana can do so much more. In the future when I can afford such luxuries, I intend to hook up all manors of metrics into Influx, so they can all be viewed within Grafana. The possibilities are truly endless. I could monitor everything from my stock portfolio to my smarthome (when I get around to setting that up).

One interesting source is the “Sun and Moon” datasource. Here I’ve created a fun dashboard that covers most of the metrics available from that datasource.

Here’s a snippet from an economics dashboard I threw together with the “Finance” datasource:

I really would like more documentation on the “Simple JSON” datasource, so we could use regular JSON endpoints as data and log it from there.

Conclusion

In this post I showed you how to take your Grafana game to the next level and utilize some amazing plugins to make your TIG stack work for you.

With enough experimentation, you can get your TIG stack to monitor everything from your beer brewing operation to your home security system. I hope to explore some of these options in the near future!

Happy New Years! 😄

Quantifying Your life: Introduction to the TIG Stack

Wed, 11 Dec 2019 00:00:00 +0000

Introduction

Ever since I was little, I admired the flashy neon-filled screens of fictional tech-wizards, with colorful graphs and charts flying all over place. Since settling into my Linux lifestyle, I had come to terms with the fact that I will be interacting much more with plaintext output logs and debug files than I would be hypnotizing myself with data straight out of NGE.

That was before I found out about the TIG/TICK stack. These software stacks can be used to monitor and visualize pretty much any data input you can imagine. Smart home appliances, sensors, ping stats, CPU usage, stock prices, you name it. There are a few differences between the two stacks, which I will briefly outline:

TICK Stack:

Telegraf: Scoops up metrics from a huge variety of inputs
InfluxDB: Stores metrics in a time-series database
Chronograf: Visualize and interact with the metrics and database
Kapacitor: Streams data from database and acts on it

These are all programs made by the same company, InfluxData. This stack would probably be my first choice for a production level enterprise environment. This is because the ecosystem is all tied together, and Kapacitor allows you to designate actions to be executed in the case of defined metrics being hit.

However for the home lab usecase, I think the TIG stack is a better fit.

Telegraf: Scoops up metrics from a huge variety of inputs
InfluxDB: Stores metrics in a time-series database
Grafana: Query database and display desired data

With this stack you are still able to see all of your collected data in whatever format you like, but there’s no need to set up an actions/trigger system if you don’t need one. Additionally, I prefer the interface and options provided by Grafana. Let’s start setting stuff up!

A Note About InfluxDB

When I began this project, I opted for the newest version of InfluxDB, V 2.0. Although I knew it was in alpha, I wanted to reduce my technical debt up the road as much as possible. Getting it up and running was fairly easy, and it has a beautiful UI that has much of the dashboarding ability of Grafana.

The only issue is that querying the V2.0 database is quite different than V1, and requires use of the proprietary Flux language. Not that I have anything against this, but V2 and Flux are not widely used/documented, so setting things up would have been frustrating to say the least.

For example, none of the premade dashboards on the community forum (which I relied on to get a feel for how things worked) work with V2.0, and require rewriting the queries from scrath.

For this reason I decided to spin up a V1 database instead, and it works great! The configuration for V2 is already in the file (commented out), so I can hook it up whenever it is in a usable state. The only downside to V1 is that it doesn’t have a web UI to configure things (usually Chronograf is used). The good news is there is minimal configuration required, if any.

Docker

If you’ve read my other posts, you know the drill by now. All these services are going into the Docker stack, so I can easily configure and deploy them anywhere. Here is a docker-compose.yaml for the TIG stack. A Traefik container is included here because I use it to reverse proxy my services.


version: '3'
services:

#      ____                                   ____
#     / __ \___ _   _____  _____________     / __ \_________  _  ____  __
#    / /_/ / _ \ | / / _ \/ ___/ ___/ _ \   / /_/ / ___/ __ \| |/_/ / / /
#   / _, _/  __/ |/ /  __/ /  (__  )  __/  / ____/ /  / /_/ />  </ /_/ /
#  /_/ |_|\___/|___/\___/_/  /____/\___/  /_/   /_/   \____/_/|_|\__, /
#                                                               /____/

   traefik:
      image: "traefik:v2.0.2"
      container_name: "traefik"
      command:
         - "--log.level=DEBUG"
         - "--api.insecure=true"
         - "--providers.docker=true"
         - "--providers.docker.exposedbydefault=false"
         - "--entrypoints.web.address=:80"
         - "--entrypoints.websecure.address=:443"
         - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
         #- "--certificatesresolvers.mytlschallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
         - "--certificatesresolvers.mytlschallenge.acme.email=you@yourdomain.com"
         - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
      ports:
         - "80:80"
         - "8081:8081"
         - "443:443"
      volumes:
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
         - "$HOME/Data/Programs/Docker/TraefikV2/letsencrypt:/letsencrypt"
      networks:
         - traefik
      restart: unless-stopped
      labels:
         - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
         - "traefik.http.routers.http-catchall.entrypoints=web"
         - "traefik.http.routers.http-catchall.middlewares=redirect-to-https@docker"
         - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"

#    ________________   _____ __             __
#   /_  __/  _/ ____/  / ___// /_____ ______/ /__
#    / /  / // / __    \__ \/ __/ __ `/ ___/ //_/
#   / / _/ // /_/ /   ___/ / /_/ /_/ / /__/ ,<
#  /_/ /___/\____/   /____/\__/\__,_/\___/_/|_|
#
   grafana:
      image: grafana/grafana
      container_name: grafana
      ports:
         - 3000:3000
      volumes:
         - /home/gideon/Data/Programs/Docker/TIG/Grafana/Data:/var/lib/grafana
      restart: always
      user: "0"
      networks:
         - traefik
      labels:
         - "traefik.enable=true"
         - "traefik.http.routers.grafana.rule=Host(`your.grafana.url`)"
         - "traefik.http.routers.grafana.entrypoints=websecure"
         - "traefik.http.routers.grafana.tls.certresolver=mytlschallenge"
         - "traefik.http.middlewares.grafana.redirectscheme.scheme=https"
      environment:
         - "GF_SERVER_ROOT_URL=your.grafana.url"

   influxdb_V1:
      image: influxdb
      container_name: influxdb_v1
      restart: always
      ports:
         - 8086:8086
      networks:
         - traefik
      volumes:
         - $HOME/Data/Programs/Docker/TIG/InfluxDBV1/:/var/lib/influxdb


   telegraf:
      image: telegraf
      hostname: nas
      volumes:
         - "/:/hostfs:ro"
         - "/etc:/hostfs/etc:ro"
         - "/proc:/hostfs/proc:ro"
         - "/sys:/hostfs/sys:ro"
         - "/var:/hostfs/var:ro"
         - "/run:/hostfs/run:ro"
         #- "/etc/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf"
         - "$HOME/Data/Programs/Docker/TIG/telegraf.conf:/etc/telegraf/telegraf.conf"
         - "/var/run/docker.sock:/var/run/docker.sock:ro"
      environment:
         - "HOST_ETC=/hostfs/etc"
         - "HOST_PROC=/hostfs/proc"
         - "HOST_SYS=/hostfs/sys"
         - "HOST_VAR=/hostfs/var"
         - "HOST_RUN=/hostfs/run"
         - "HOST_MOUNT_PREFIX=/hostfs"
      ports:
         - 8888:8888
      networks: traefik

networks:
  traefik:
    external: true

Telegraf

Telegraf took me a bit to get used to when I first learned it, but now it all makes sense. To get anything useful out of telegraf, you have to use plugins. Using a plugin is as simple as adding a line to your telegraf.conf file, that is located by default at etc/telegraf. This is where we mounted in the docker container, so you can keep it anywhere on your local filesystem.

The two types of plugin we need today are input and output. output is pretty simple. We simply need to point our telegraf instance at out database and say “store data there!”. That can be done with the following code:

# OUTPUTS
[[outputs.influxdb]]
  url = "http://IP.OF.INFLUXDB:8086" # required.
  database = "telegraf" # required.
  precision = "s"

Note: I had to explicitly type my local IP into the url field. localhost did not work for whatever reason…

And that’s it! You could have telegraf broadcast data to multiple locations if you wanted. Next is to configure the input plugins. I will post my config here, but I recommend you take a look at the list of plugins for some that may suit a usecase you have and I don’t.

The lines below will set up monitoring for the host’s docker environment, as well as system hardware/software statistics. Additionally there are some “fun” plugins to gather data about github repositories, the weather, and more.

# INPUTS
[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  container_names = []
  timeout = "5s"
  perdevice = true
  ## Whether to report for each container total blkio and network stats or not
  total = false
  ## docker labels to include and exclude as tags.  Globs accepted.
  ## Note that an empty array for both will include all labels as tags
  docker_label_include = []
  docker_label_exclude = []

[[inputs.docker_log]]
  ## Docker Endpoint
  ##   To use TCP, set endpoint = "tcp://[ip]:[port]"
  ##   To use environment variables (ie, docker-machine), set endpoint = "ENV"
  endpoint = "unix:///var/run/docker.sock"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  # filter all fields beginning with 'time_'
  fielddrop = ["time_*"]

[[inputs.disk]]
  [inputs.disk.tagpass]
    # tagpass conditions are OR, not AND.
    # If the (filesystem is ext4 or xfs) OR (the path is /opt or /home)
    # then the metric passes
    fstype = [ "ext4", "xfs" ]
    # Globs can also be used on the tag values
    path = [ "/opt", "/home*" ]

[[inputs.diskio]]

[[inputs.kernel]]

[[inputs.net]]

[[inputs.netstat]]

[[inputs.nstat]]

# Collect response time of a TCP or UDP connection
[[inputs.net_response]]
  ## Protocol, must be "tcp" or "udp"
  ## NOTE: because the "udp" protocol does not respond to requests, it requires
  ## a send/expect string pair (see below).
  protocol = "tcp"
  ## Server address (default localhost)
  address = "localhost:80"

[[inputs.processes]]

[[inputs.sensors]]

[[inputs.swap]]

[[inputs.mem]]

[[inputs.internal]]

[[inputs.interrupts]]

[[inputs.temp]]

[[inputs.wireless]]

[[inputs.bond]]

# This plugin gets dns information/response time from configured domains.
# Super helpful for checking status of service or machine
[[inputs.dns_query]]
  ## servers to query
  servers = ["8.8.8.8"]
  domains = ["exampledomain.tld"]

[[inputs.system]]

[[inputs.github]]
  ## List of repositories to monitor
  repositories = [
	  "GideonWolfe/vim.reaper",
	  "GideonWolfe/gideonwolfe.com"
  ]

  ## Github API access token.  Unauthenticated requests are limited to 60 per hour.
  access_token = "YOUR_TOKEN_HERE"

[[inputs.openweathermap]]
  ## OpenWeatherMap API key.
  app_id = "YOUR_KEY_HERE"

  ## City ID's to collect weather data from.
  city_id = ["098008"]

  ## APIs to fetch; can contain "weather" or "forecast".
  fetch = ["weather", "forecast"]

  ## Preferred unit system for temperature and wind speed. Can be one of
  ## "metric", "imperial", or "standard".
  # units = "metric"

  ## Query interval; OpenWeatherMap weather data is updated every 10
  ## minutes.
  interval = "10m"

And that’s it! I’m always on the lookout for cool plugins and ways to integrate my data into telegraf.

InfluxDB

This is the quickest set up by far. Simply get a shell inside the influx docker container with docker exec -it <container-name> bash. Run influx to start the interactive database prompt.

Create a user with the following command:

CREATE USER <username> WITH PASSWORD <password> WITH ALL PRIVILEGES

Then add a database with this one:

CREATE DATABASE telegraf

If you want to add a data retention policy, do it like so:

CREATE RETENTION POLICY “keep_x_days” ON “telegraf” DURATION xd REPLICATION 1

where x is the amount of days you want InfluxDB to keep your data.

Bring it all up

The only thing to do now is bring the whole thing up with docker-compose.

To start out, bring the stack up with docker-compose up so you can read the logs and identify possible configuration errors. If everything is going as planned, you should be able to access your Grafana instance at http://localhost:3000. Additionally, telegraf should be feeding into the InfluxDB container without error.

Grafana

The first step for setting up Grafana is adding InfluxDB as a data source, so you can actually start getting some data. Go to Configuration > Data Sources > Add data source and fill in the details for your InfluxDB instance. Since Grafana is querying a database that isn’t exposed to the internet and instead running on the local system, we can skip the security settings for now and simply enter in the IP of our database, the database name (which should be telegraf), and the credentials for the user we just created.

If this was set up correctly, the Save & Test button will let you know. Next, we should download some panels. Browse the Grafana panel plugin list to find ones that you want. The last part of the url of the page of each panel is it’s id.

grafana-cli plugins install <plugin_id>

Now these added panels will show up in the menus. You can add datasources the same way, but they don’t seem to show up unless you manually add and enable them through the GUI as well.

The next step is where the real fun begins; adding dashboards. I would also recommend checking out the other available datasources, because there are some neat ones that don’t require any installation.

If you’re a complete novice at this like I was, browse some community dashboards to get inspired. I will briefly show you how to set up a simple CPU monitor dashboard, but the more advanced stuff will be for a later tutorial.

Click the “New Panel” icon at the top right of your dashboard. Then click “add query”. Here is the basic idea of how your queries should be structured. You can click on each field to see the possible values, and explore the metrics collected by your various input plugins.

Play around with the visualization settings, and the various configuration options each one has. Community dashboards are a great way to get inspired for what you should include on yours.

Conclusion

In this post I showed you how to get the TIG stack up and running with docker. This software stack allows you to collect metrics from one or more machines, store this metrics in a purpose built database, and display these metrics in a beautiful fashion with customizable dashboards and panels.

In my next post I will give some examples of advanced database queries, live diagrams, and customized dashboards.

Unifying the News: Nextcloud, Newsboat, and Linux.

Tue, 03 Dec 2019 00:00:00 +0000

In a previous post, I explained how to set up a Nextcloud server and integrate some crucial applications into your workflow. I briefly touched on the news application, which is a nice way of seeing some RSS feeds in your nextcloud. But much like email, I prefer using the terminal as my primary viewing method. I have found a great way to achieve this: an application called newsboat.

newsboat allows you to create a list of feed URLs, but I will be configuring it to pull from the feeds defined in my Nextcloud News app.

Install newsboat on Arch linux by running

sudo pacman -Syu newsboat

By default, newsboat is configured with files in the ~/.newsboat directory. However, I prefer the files to be in ~/.config/newsboat/, which will work just fine. Create the file config in the directory you choose. Here is my configuration file:


prepopulate-query-feeds yes

urls-source "ocnews"
ocnews-url "https://nextcloud.yourdomain.com" # Domain of your Nextcloud
ocnews-login "username" # Desired nextcloud user to log in as
ocnews-password "password" # Nextcloud password (app password if using 2Fa)

# -- display -------------------------------------------------------------------

show-read-feeds no
feed-sort-order unreadarticlecount-asc

color info default default reverse
color listnormal_unread cyan default
color listfocus blue default reverse bold
color listfocus_unread blue default reverse bold
color article cyan default
color listnormal yellow default

text-width 80

articlelist-format "%4i %f %D  %?T?|%-17T| ?%t"

highlight feedlist    "^  *[0-9]+  *N  "                    magenta  magenta
highlight articlelist "^  *[0-9]+  *N  "                    magenta  magenta

highlight article     "(^Feed:.*|^Title:.*|^Author:.*)"     red      default
highlight article     "(^Link:.*|^Date:.*)"                 white    default
highlight article     "^Podcast Download URL:.*"            cyan     default
highlight article     "^Links:"                             magenta  black    underline
highlight article     "https?://[^ ]+"                      green    default
highlight article     "^(Title):.*$"                        blue     default
highlight article     "\\[[0-9][0-9]*\\]"                   magenta  default  bold
highlight article     "\\[image\\ [0-9]+\\]"                green    default  bold
highlight article     "\\[embedded flash: [0-9][0-9]*\\]"   green    default  bold
highlight article     ":.*\\(link\\)$"                      cyan     default
highlight article     ":.*\\(image\\)$"                     blue     default
highlight article     ":.*\\(embedded flash\\)$"            magenta  default


# -- navigation ----------------------------------------------------------------

goto-next-feed no

browser "xdg-open" # This will open the browser defined by xdg-settings

# Below bindings apply vim like keybindings in most places

bind-key h quit articlelist
bind-key h quit article
bind-key h quit tagselection
bind-key h quit feedlist
bind-key j down feedlist
bind-key j down tagselection
bind-key j next articlelist
bind-key j down article
bind-key J next-feed articlelist
bind-key k up feedlist
bind-key k prev articlelist
bind-key k up tagselection
bind-key K prev-feed articlelist
bind-key k up article
bind-key l open articlelist
bind-key l open feedlist
bind-key l open tagselection

bind-key G end
bind-key g home

bind-key d pagedown
bind-key u pageup

If you filled out this information correctly, you should be able to start newsboat and see your saved RSS feeds populate! note: you may need to manually create a file called urls in order for newsboat to start, even if it’s empty.

Press t to see tags. These are the equivalent of categories in the news app on Nextcloud, and are nice if you prefer browsing by category.

Some articles will let you read the entire thing in newsboat, but usually only the description is visible. Pressing O will open the story in your preferred browser if you want to read a full article.

Let me know if you have any configuration tips, I just recently found about this killer app combo!

Chemistry Practice: Buffer Systems

Wed, 27 Nov 2019 00:00:00 +0000

$\require{\mhchem}$

Buffer systems are important part of of organic and inorganic chemistry. In short, a buffer is a solution that is resistant to pH changes, such as the addition of an acid or a base.

Let's go over an example problem you might encounter on a homework assignment or exam.

What is the pH of an 80 mL .30M NH₃/0.36M NH₄⁺ buffer system after 20.0 mL of 0.050M NaOH has been added? The Kₐ value for $\ce{NH4+}$ is $5.8\text{x}10^{-10}$

Lets break this problem down. We have 80 mL of a solution made up of $\ce{NH3}$ (ammonia) and $\ce{NH4+}$ (ammonium).

We know the concentrations of each of these species, denoted by brackets.

$[\ce{NH3}] = \frac{\text{0.30 moles } \ce{NH3}}{\text{1 Liter}}$ $[\ce{NH4+}] = \frac{\text{0.36 moles } \ce{NH4+}}{\text{1 Liter}}$

Then we add 20.0 mL of $\ce{NaOH}$, which has the following concentration.

\[[\ce{Na+}] = \frac{\text{0.050 moles } \ce{Na+}}{\text{1 Liter}}\]

To proceed with this problem, we need to figure out what will happen to the $\ce{NaOH}$ (sodium hydroxide) when it is added to the solution.

Sodium hydroxide is a strong base, and will completely disassociate into the Sodium and Hydroxide ions like so.

\[ \ce{NaOH ->[H₂O] Na+ + OH- } \]

Theoretically $\ce{Na+}$ would react with $\ce{H2O}$ to form more $\ce{NaOH}$, but this will not happen, because $\ce{NaOH}$ is a strong base and will not be formed. For this reason, we can ignore it in our further calculations. This leaves us only the $\ce{OH-}$ to worry about.

Since $\ce{OH-}$ is a base, it will react with the acid in our buffer solution. Buffer solutions are always made up of conjugate acid/base pairs. In this case, $\ce{NH4+}$ is the acid, because it is the conjugate acid of $\ce{NH3}$.

\[\ce{NH4+(aq) + OH- (aq) <-> NH3(aq) + H2O(l)}\]

In this reaction, hydroxide acts as a base and accepts the proton from ammonium.

To further solve this problem, we are going to need to use a RICE table.

$$ \begin{array}{c|c@{}c@{}c@{}c@{}c} \hline R & \ce{NH4+(aq) & $${}+{}$$ & OH- (aq) & <-> & NH3(aq) & $${}+{}$$ & H2O(l)} \\ \hline I & ?.?? && ?.?? && ?.?? && \text{---} \\ C & ?.?? && ?.?? && ?.?? && \text{---} \\ E & ?.?? && ?.?? && ?.?? && \text{---} \\ \hline \end{array} $$

Let's start by figuring out $I$, the initial amount of moles for each species. Since water is in such high abundance, we can leave it out of the equation.

\[ \text{moles } \ce{NH4+} = \frac{0.36 \text{ moles } \ce{NH4+}}{1 \text{ L}} * 0.0800 \text{ L} = 0.0288 \text{ moles}\]

\[ \text{moles } \ce{OH-} = \frac{0.050 \text{ moles } \ce{OH-}}{1 \text{ L}} * 0.0200 \text{ L} = 0.0010 \text{ moles}\]

\[ \text{moles } \ce{NH3} = \frac{0.30 \text{ moles } \ce{NH3}}{1 \text{ L}} * 0.0800 \text{ L} = 0.0240 \text{ moles}\]

Here is our updated rice table with the initial amounts of each species.

$$ \begin{array}{c|c@{}c@{}c@{}c@{}c} \hline R & \ce{NH4+(aq) & $${}+{}$$ & OH- (aq) & <-> & NH3(aq) & $${}+{}$$ & H2O(l)} \\ \hline I & 0.0288 && 0.0010 && 0.0240 && \text{---} \\ C & ?.?? && ?.?? && ?.?? && \text{---} \\ E & ?.?? && ?.?? && ?.?? && \text{---} \\ \hline \end{array} $$

We can see that all species in this reaction are one to one ratios. This means that $\ce{OH-}$ is the limiting reactant, as it has the least amount of moles present.

Additionally, this means the initial number of moles of $\ce{OH-}$ is the maximum number of moles that can be reacted of either of the other chemical species.

Now we can fill in $C $ and $E $, or the change in moles during the reaction and the final amount at equilibrium.

$$ \begin{array}{c|c@{}c@{}c@{}c@{}c} \hline R & \ce{NH4+(aq) & $${}+{}$$ & OH- (aq) & <-> & NH3(aq) & $${}+{}$$ & H2O(l)} \\ \hline I & 0.0288 && 0.0010 && 0.0240 && \text{---} \\ C & -0.0010 && -0.0010 && -0.0010 && \text{---} \\ E & 0.0278 && 0 && 0.0250 && \text{---} \\ \hline \end{array} $$

We can see that all of the hydroxide was depleted in this reaction. Now that we have the number of moles of each species at equilibrium, we can preform the following calculations to find the concentration at equilibrium.

$[\ce{NH3}]_{eq} = \frac{\text{0.0250 moles } \ce{NH3}}{\text{0.1000 Liter}}$ = 0.250 M

$[\ce{NH4+}]_{eq} = \frac{\text{0.0278 moles } \ce{NH4+}}{\text{0.1000 Liter}}$ = 0.278 M

We use the volume of .1 L because we had an initial volume of 80 mL and then we added an additional 20 mL of $\ce{NaOH}$.

We can use these equilibrium molarities as inputs to the Henderson–Hasselbalch equation, which takes the following form:

\[ pH = pK_a + \text{log}\left( \frac{[A^-]}{[HA]} \right) \]

In this example, $\ce{HA}$ and $\ce{A-}$ represent the acid and it's conjugate base respectively.

Let's start substituting in values. We know that the Kₐ of $\ce{NH4+}$ is $5.8\text{x}10^{-10}$, and from that we can derive the pKₐ value, which is equal to $-\text{log}\left( K_a \right)$.

We also have the equilibrium concentrations for the conjugate acid/base pair. After filling in our values, the HH equation should look like:

\[ pH = 9.24 + \text{log}\left( \frac{0.250}{0.278} \right) \]

Plug this into your calculator, and you should get a resulting pH of $9.19$.

Conclusion

Please let me know if you have any questions or corrections for me! Writing these exercises helps me practice for my exams 😄

Controlling Your Cloud: Integrating Nextcloud Into Your Workflow

Sun, 24 Nov 2019 00:00:00 +0000

Intro

In my last tutorial, I demonstrated how to set up a barebones Nextcloud instance using Docker. The following guide assumes you have completed this step, and will focus on tying it all together into a seamless system.

Files

The most essential part of your workflow is going to be your file management. Figure out which devices you want to include in this system and install the appropriate Nextcloud client on all of them.

Contacts
Calendars
Location
Tasks
Photos
Texts
Notes

For my photos, I simply add the camera libraries I care about to the “Auto Upload” section in the Android Nextcloud client. These are auto-organized into folders by month, and put into a central “Photos” location on my server. Any screenshot, meme, or photo will be auto uploaded to the appropriate folder.

Contacts and calendars are all handled with DAVx⁵ android app. Link this with the Nextcloud client and choose which address books and calendars you want to synchronize with your server. These should show up in the native Contacts and Calendar apps.

Texts can be synchronized with the Nextcloud SMS app. This does have some limitations, as you cannot write/send SMS messages in the mobile or desktop app. It is useful for backing up and viewing conversations though.

Optionally, a location client like Traccar can be set up using a link from the Phonetrack app on Nextcloud. This is kind of like a personal “Find my iPhone”, with the added benefit of logging and backing up gpx files which can be analyzed for whatever data you please. There are numerous gpx editor apps on the store, but PhoneTrack has a great built in filtering system.

I use the OpenTasks appkhard to synchronize tasks from my Nextcloud server. The Tasks app on Nextcloud is my primary way of keeping track of assignments and todos, and having it on mobile is great. My only wish was to be able to see due dates for tasks in the calendar application.

Although I usually prefer neovim as my text editor of choice, I am currently trying the Carnet app (also available on Nextcloud store) because it has some nifty notetaking features that I have yet to replicate in neovim.

Finally, I like to have some extra security for my server. I downloaded and enabled the Two-factor authentication application, and linked it up with Google Authenticator to easily verify my 2FA logins. Most of the applications going forward will be using “App Passwords” that can be generated in the Security section of the Settings. This way you can let certain applications bypass 2FA.

Both my laptop and desktop are running Arch Linux, so the setup for both of them is essentially the same. Not only this, but the directory structure of each machine is essentially 1:1, so the experience is like browsing one large synchronized filesystem.

Like with the mobile client, set up the source -> destination connections for the desired folders on your machines. I recommend taking some time to think through a directory structure you will be happy with in the long term.

Files
Contacts
Calendars
Email

To manage Contacts and Calendars, I use a program called vdirsyncer to synchronize the necessary contact/calendar files with the server. The configuration for this tool is quite simple. I will post an example below.


[general]
status_path = "~/.config/vdirsyncer/status/"

[pair my_contacts]
a = "my_contacts_local"
b = "my_contacts_remote"
collections = ["from a", "from b"]
conflict_resolution = "b wins"

[storage my_contacts_local]
type = "filesystem"
path = "~/Documents/Contacts/"
fileext = ".vcf"

[storage my_contacts_remote]
type = "carddav"

# We can simplify this URL here as well. In theory it shouldn't matter.
url = "URL_TO_NEXTCLOUD"
username = "USERNAME"
password = "GENERATED_APP_PASSWORD"

[pair my_calendar]
a = "my_calendar_local"
b = "my_calendar_remote"
collections = ["from a", "from b"]
conflict_resolution = "b wins"

[storage my_calendar_local]
type = "filesystem"
path = "~/Documents/Calendar/"
fileext = ".ics"

[storage my_calendar_remote]
type = "caldav"

# We can simplify this URL here as well. In theory it shouldn't matter.
url = "URL_TO_NEXTCLOUD"
username = "USERNAME"
password = "GENERATED_APP_PASSWORD"

Run the commands vdirsyncer discover my_contacts and vdirsyncer discover my_calendar to populate the directories. If everything is set up correctly, vdirsyncer sync should synchronize any changes made to the local or remote versions of the files.

Conflict resolution for these files is dependant on the setting defined by conflict_resolution.

But how do we interact with these files?

Currently, I use the application khal. Run khal --config to go through the interactive prompt and point it at your calendar files. This will generate a configuration file .config/khal/config. To view and edit your calendar, you can use the built in commands or khal interactive for a more traditional TUI experience.

In each calendar folder, you can create a file called color, and fill it with a single color that you would like to highlight that calendar with. For example, one color file could have the line light cyan.

Much like the calendar, contacts are handled khard. run through the interactive configuration with khard --config.

Now I really don’t care about creating contacts on my desktop, because creating them with my phone works well and will propagate to all my devices. You certainly could create contacts with khard. As usual, run vdirsyncer sync to synchronize your files.

My favorite use for khard is to provide a contact database for my email client neomutt. The mail application in Nextcloud is pointed at my email server, so this should point there too (or whatever email server you are using). Configuring neomutt is kind of complex, and will be the topic of another tutorial.

However if you already have a working neomutt, you can add the line

set query_command = "khard email --parsable --search-in-source-files '%s'"

to your ~/.config/neomutt/settings file. This will make neomutt parse your contacts list when composing an email!

Other cool features

While these may not be absolutely essential, they really help bring Nextcloud to the next level; a life dashboard as opposed to simply a file sync service.

News: The Nextcloud news app and companion android app allow you to subscribe to a number of RSS/news feeds, viewing them all in one comfy application. Has some limitations, but works great for the most part.
Cookbook: The Cookbook app is one of my favorites, you can create your own recipes or import one from a website (assuming they are using a specific JSON format required by the app). I definitely killed a few hours by building up a hefty recipe bank. App is still in alpha, but suits my needs so far! I really want a treelike directory structure to sort recipes though.
Maps: I used a free API key for MapBox (configured in settings) and now I have a nifty map inside Nextcloud! It would be even more powerful with a dedicated routing server to provide Google Maps like directions. This map will display detected data found on the server, such as GPX tracks, photo location metadata, and contact addresses.
Keepweb: This application allows you to open and edit Keepass files on your server. Putting these files on your server can be a potential secrity risk, but there are plenty of encryption options to suit your needs. This synced kdbx file can be edited on the desktop (I prefer keepassxc), and viewed on the server if you need a password in a pinch.
Talk: Host a chatroom right out of your Nextcloud instance! I haven’t used this much, but in theory you can make voice and video calls, send text, and even polls and other chat features.

Conclusion

If you set it up right, Nextcloud can be far more versatile than a simple file synchronization service. Through Nextcloud, you can back up and manage contacts, calendars, emails, and much more. In combination with the ease of deployment docker provides, this setup is a formidable match for the out of the box services provided by larger companies. Let me know how you use Nextclud, I’m always on the lookout for great ways to integrate it into my workflow.

Controlling Your Cloud: Deploying a Nextcloud Server With Docker

Fri, 22 Nov 2019 00:00:00 +0000

I have always loved the convenience and versatility of cloud based services, but never really liked the idea of not “owning” the files on them. I also found myself becoming increasingly uneasy with Google over the years, and wanted to migrate away from their platform as soon as possible. In general, I don’t really like the idea of getting locked into a single ecosystem, whether it be Apple/iCloud, Microsoft/O365, or Google/GSuite.

That’s why I was so excited when I heard about Nextcloud, a self hosted cloud platform that is practically a drop in replacement for the services mentioned above. In fact in a lot of ways, Nextcloud offers more functionality. Nextcloud is FOSS, anyone can write a useful application and share it with the world. I am able to do things with Nextcloud that I was never able to achieve through an off the shelf cloud suite.

There are many ways to use Nextcloud. You could set it up as an organization, and have a self hosted array of cloud servers for your company. Personally, I treat Nextcloud as a “life dashboard” of sorts. If done right, you can make Nextcloud a one stop shop for productivity, organization, and personal management. Here are some of the things one can use Nextcloud for:

Calendar
Contacts
Email
Task Manager
File Storage
Real Time Phone Tracking
SMS Backups
Recipe Book
Notes
Bookmarks

And much more!

I have been using Nextcloud for a couple years now, and since then, I have become more familiar with Docker. The first time I set up Nextcloud, I used a pretty sketchy install script. This worked, but I was never really sure exatcly how things were set up. If you know me, that’s not how I roll. I realized the time had come to redo things with Docker.

Getting a Server

While you could totally host this out of a spare PC in your house, I chose to go with a VPS for a couple reasons:

Reliable uptime
Better up/down speeds
Automated backups
Easily transfer to another VPS

I chose to go with an Ubuntu 18.04 VPS from DigitalOcean, because they have been fantastic in the past. Theoretically this should work anywhere.

One might also ask why I chose to use docker in this situation. I had multiple servers at DO all running separate services. With docker, I would be able to downsize to a single server, and run multiple containerized services side by side, much like with my Home Media Server build. This would allow me to easily deploy this configuration on any machine, and save me the hassle of setting up and reverse-proxying multiple services running on bare metal.

Setting up Docker

Run the following command to get the docker tools installed on your system. I would recommend making a new user so you don’t do the entire tutorial as root.

sudo apt install docker.io docker-compose

And give your user permission to execute docker commands without a password (requires you to sign in again)

sudo usermod -aG docker ${USER}

Deploying the Services

First, Let’s get our compose file out of the way. Make a new directory that you want to use for your docker services and create a file called docker-compose.yaml.

Here is my docker-compose.yaml, where I have declared the traefik, ncdatabase, and nextcloud services. ncdatabase is a MariaDB that will act as the database used by Nextcloud. Then, the nextcloud will host our main Nextcloud instance. Finally, I am using the traefik provide SSL certificates and reverse proxy incoming connections to the correct service.

version: '3'

services:

  traefik:
    image: "traefik:v2.0.5"
    container_name: "traefik"
    command:
      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.myhttpchallenge.acme.httpchallenge=true"
      - "--certificatesresolvers.myhttpchallenge.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.myhttpchallenge.acme.email=yourname@yourdomain"
      - "--certificatesresolvers.myhttpchallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "$HOME/data/letsencrypt:/letsencrypt"
    networks:
      - traefik

  ncdatabase:
    image: mariadb
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    restart: always
    volumes:
      - /home/gideon/data/nextcloud/db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=DECLARE_THIS_VALUE
      - MYSQL_PASSWORD=DECLARE_THIS_VALUE
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    labels:
      - "traefik.enable=false"
    networks:
      - traefik

  nextcloud:
    image: nextcloud
    ports:
      - 8080:80
    volumes:
      - NEXTCLOUD_DIRECTORY:/var/www/html
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud.rule=Host(`nextcloud.mydomain.com`)" # This is the domain you want nextcloud to be reached on
      - "traefik.http.routers.nextcloud.entrypoints=websecure"
      - "traefik.http.routers.nextcloud.tls.certresolver=myhttpchallenge"
      - "traefik.frontend.redirect.permanent=true"
      - "traefik.frontend.redirect.regex=https://(.*)/.well-known/(card|cal)dav" # These last few labels were added to silence some warnings
      - "traefik.frontend.redirect.replacement=https://$$1/remote.php/dav/"      # They may or may not be necessary
      - "traefik.frontend.headers.customResponseHeaders=Strict-Transport-Security:15552000"
    networks:
      - traefik

networks:
  traefik:
    external: true

Note: You must make sure you have A records pointing your chosen domain to the IP address of your server. With DigitalOcean, this can be set up in the Networking dashboard.

Nextcloud should have generated some files. In the main Nextcloud directory, cd into config/ and edit the config.php file. Edit or add the values shown below to reflect your configuration.

'trusted_domains' =>
array (
  0 => 'nextcloud.mydomain.com',
  1 => 'localhost',
),
'trusted_proxies' =>
array (
  0 => '<IP ADDRESS OF DOCKER>',
),
'forwarded_for_headers' =>
array (
  0 => 'HTTP_X_FORWARDED_FOR',
),

'overwritehost' => 'nextcloud.mydomain.com',
'overwrite.cli.url' => 'http://nextcloud.mydomain.com',
'overwriteprotocol' => 'https',

trusted_domains allows Nextcloud to be reached at your chosen domain. trusted_proxies allows Nextcloud to properly detect connecting IP addresses that are being reverse-proxied by traefik. forwarded_for_headers was added to get rid of an error about invalid HSTS headers. Finally, the last three lines enable HTTPS redirects. Without these lines, sync clients will be unable to correctly authenticate with the server.

To bring up the stack, run docker-compose up -d, or docker-compose up if you want to see the logs in your terminal. If all goes correctly, you should be able to access your cloud instance at the domain you selected.

Setting up Nextcloud

At this point, you should be looking at a stock Nextcloud welcome screen. First things firs, create yourself a non-admin user we can use after we get everything set up. This way we can delegate mission critical tasks to the admin account with a very strong password, and use our own account without access to admin settings.

One of the main features of Nextcloud I use is the sync clients. I have one on every device. On Arch linux, you can download the nextcloud-client package. Open it up and log into the server using your newly created user account. From here you can select a local folder on your machine and map it to a remote folder on your Nextcloud.

I tend to keep my Nextcloud folder structure almost a 1 to 1 match with the folders in /home of my computers. That makes syncing the folders absurdly easy, ~/Documents gets mapped to Documents, etc.

This instant synchronization allows me to edit a file on my laptop in one room, and resume editing the same file moments later on my desktop or phone. It’s almost like I have one /home directory synced across all my devices. Of course you might not want to sync everything. I found that nextcloud does not play nice with git directories, and will mess with the local branch. I exclude mission critical local repositories from syncing so I can manage them through github.

Conclusion

This guide showed you how to spin up the bare minimum Nextcloud instance using Docker. In my next guide, I’ll show you how to flesh out this instance, and what tools to use to make your digital life completely seamless.

Creating a pocket pentest platform with P4wnP1: Part 1

Sat, 09 Nov 2019 00:00:00 +0000

Intro

Those in the security community are most likely familiar with the USB rubber ducky. On the off chance that you haven’t heard of it, the ducky is a device with the form factor of a standard USB drive. However instead of simply storing files, the rubber ducky can emulate a Human Interface Device (HID) such as a keyboard and mouse. It can preform any task you would be able to using these devices, except it can type at the speed of a computer.

This allows an attacker to plug in the ducky, deliver their payload at lightning speed, and get the hell out.

That being said, the ducky has some limitations. First of all, it is $50 for a single unit, which seems like a lot for a tiny flash drive with a single capability. Secondly, the default language “Duckyscript” does not allow for particularly advanced payloads. Payloads can be selected with physical dip switches on the device.

Then I heard about P4wnP1, which solves many of the issues mentioned above. P4wnP1 runs on a raspberry pi, a tiny but fully featured linux computer. This means that on top of the HID injection features of the rubber ducky, there is a lightweight installation of Kali Linux under the hood, ready for you to use as a malicious AP, pivot box, network sniffer, or whatever else you would do with Kali.

Required hardware:

Here is a list of the hardware I used for this project.

Item	Price
Raspberry pi 0W	$10
GPIO pins	$7.97
OLED display hat	$16.95
Micro SD card(s)	$16.98
USB A addon board	$8.99
3D printed case	Free from work
Total	$60.89

For only $10 more than the ducky, we can expand our feature set by a great amount. Not only this, but many of these items contain duplicates, (multiple SD cards and GPIO pin sets) so you could make many of these devices, or save money by buying individually packaged.

Installing P4wnP1

Looking at the P4wnP1 readme, we can see there’s options for installing the P4wnP1 service on top of an existing linux distrobution, or installing the entire image to a fresh SD card.

Since we have fresh SD cards, this is by far the simplest method. Fire up your favorite image flasher (or use the dd command) and write the latest P4wnP1 image to your SD card. Most documentation recommends the etcher program to do this, so I figured I would give it a shot.

Open etcher as root, select your image, select your SD card, and flash away.

Once this step is complete, you have an SD card containing the P4wnP1 image, ready to roll in your raspberry pi.

Set up P4wnP1

To set up your P4wnP1 for the first time, we can plug in a Micro USB cable from a PC into the pi.

Make sure that the correct “Data” port is being used instead of the power port. Do not interrupt the first boot of the P4wnP1, which can take a few minutes.

There are a few ways to connect to your pi. My preferred method is to use the PC the pi is connected to and navigate to http://172.16.0.1:8000 in a browser. This should open up the P4wnP1 web configuration page.

I will delve further into the settings and workflow in another post, as I am mainly focusing on getting the physical device set up here.

Additionally, you can connect to P4wnP1 by using the web interface to connect it to your WiFi network, and ssh to it with ssh root@172.16.0.1. Or connect your device to the AP broadcasted by the P4wnP1, and ssh from that network. There is also a bluetooth configuration option, yet I have never tried it.

See the readme section for more info on how to connect to your P4wnP1.

Setting up the hardware

Now that we confirmed we have a working raspberry pi, and a valid version of P4wnP1 flashed to our SD card, we can start working on the hardware.

The first step is to solder the GPIO pins to the raspberry pi. If you don’t have a soldering gun or are just a bit lazy, you can opt for a raspbery pi 0W with pre soldered pins.

Holding the raspberry pi upright, insert the pins short side down into the pinholes. Moving to the underside of the pi, apply a small drop of solder to each of the pins, so it makes a connection with the pad at the base of the pin.

Notice the plastic screws poking through the corners of the board. Those are the standoffs for the USB A addon board, which you can proceed to secure with the provided set of plastic nuts.

The USB A addon board should make a solid connection with the pads on the bottom of the pi. Test to see if it works by powering the pi over the USB A plug. This addon board allows us to preform ducky like attacks without carrying an additional micro USB -> USB A converter.

Finally, we can install the OLED screen hat using the GPIO pins we just soldered to the pi. This may take a bit of force, but the screen should press down over the IO pins.

I used a bit of the foam that the OLED hat was packaged with to brace the PCB against the 3D printed case.

For the OLED menu, we will be using the amazing menu by Beboxos. The full tutorial can be found in the readme, but I will outline the steps here, as it’s quite simple.

To enable the screen to for P4wnP1, we first have to edit the config.txt located in the boot partition of the SD card.

In the dtparam section, make sure the following lines are set:

dtparam=i2c_arm=on dtparam=i2c1=on
dtparam=spi=on

There is a section in the readme that mentions only one of these lines is necessary depending on your display, but I had the display he recommended on the Thingiverse page, so I left all the options on to be safe. Haven’t seen any issues so far with that.

Next, ssh into the pi with ssh root@17.16.0.1. Connect the pi to the internet through the web interface, and clone the repository for this project. Inside it, you will find a script install.sh At the time of writing, this script was quite buggy, and I had to fix it up a bit to get it working. Mostly just messed up filepaths and such.

#!/bin/sh
echo "Install Luma.core drivers"
apt-get install python-dev python-pip libfreetype6-dev libjpeg-dev
pip install --upgrade pip
apt-get purge python-pip
pip install --upgrade luma.oled
echo "Create directory"
mkdir /root/BeBoXGui/
echo "Copying files"
cp *.py /root/BeBoXGui/
mkdir /root/BeBoXGui/images
cd images
cp * /root/BeBoXGui/images/
echo "Copying run script in local P4wnP1 script"
cp scripts/runmenu.sh /usr/local/P4wnP1/scripts/
echo "All files are ready"
echo "to run with P4wnP1 boot"
echo "Go thru web interface"
echo "Go in trigger section"
echo "Create new trigger"
echo "on service start :"
echo "run script sh and choose "
echo "runmenu.sh"
echo "Enjoy"
echo "by default gui.py use SPI interface"
echo "if you use I2C oled edit gui.py"
echo "and set I2C_USER = 1"

I would have created a PR for these fixes but there was an outstanding PR addressing the issue which has been ignored for some months now.

As you can see in the script, the script runmenu.sh was copied to /usr/local/P4wnP1/scripts/. As you might have guessed, running this script will in fact run the on screen menu. The final step is to create a trigger action that runs that script when the P4wnP1 service starts.

We can then store this trigger action set, and create a new “master template” in the Generic Settings menu that uses this stored trigger actions set. This master template should inherit all other settings modules from startup as to not interfere with the process.

We can then make our newly created master template the default startup template using the field on the righthand side of the screen.

Now when you reboot the device, you should be greeted by the OLED menu!

I replaced the filename at this line with another file from the images directory. You could totally make your own, I would love to see some cool ones.

We can see the main menu offering us a wide range of options.

The main areas of interest to us are the HID Management and Templates Features sections. The HID Management section allows you to browse your stored HID scripts, and run them as either a direct or background job.

In Templates Features, we can browse and apply any saved settings templates for Network, Bluetooth, Wifi, USB, and Trigger actions. This allows you to easily set up more complex attacks with complicated configurations without having to set them up via SSH or the web interface

Any settings applied through the menu should be instantly visible on the web interface.

The Case

I happen to be lucky enough to work an environment where I have a 3D printer at my disposal. However I did not design the case myself, but I found it on Thingiverse, and printed out from there.

Eventually, I would like to modify this design to incorporate some sort of rechargeable battery. However since the screen is occupying the GPIO pins and the USB A addon is occupying the connection pads, there are limited options when it comes to connecting it.

One could forgo the USB A capabilities and instead use the connectors for a Pisugar, which provides a solderless rechargeable battery.

Or if someone wanted to make a screenless version, they could opt for a LiPo hat that uses multiple PCBs and the GPIO pins. This seems like it would be the bulkiest method.

Conclusion

In this post, I showed you how to create a pocket sized pentest platform, with HID keystroke capabilities that are more robust than the USB rubber ducky, with the benefits of a powerful Kali Linux backbone for all your netsec needs.

In the near future I will elaborate on the effective P4wnP1 workflow, and some nifty examples of the powerful HID interface API.

Alava - A new single by the Dawn Bombs

Sat, 12 Oct 2019 00:00:00 +0000

I have been a huge fan of The Dawn Bombs since I met moved to Bellingham and my previous band Middle James started playing shows with them.

Fast forward a few years, and the band has moved into my apartment. My friend and ex-bassist for the dawn bombs had left to move on to other projects, and I happily stepped in to fill the void as best I could.

While I do appear on this recording singing and playing bass, I cannot take credit for this bassline. This was one of the last parts written by my predecessor, and in my opinion some of his finest work.

That being said, I thoroughly enjoyed being involved in the production of the rest of the song, from the mixing and mastering to the instrumentation and composition.

We were also fortunate enough to be provided with some amazing artwork, courtesy of my roommate. You can check out some of her other art here.

Alava is already on Bandcamp, and will appear on Spotify and other streaming platforms later this week.

You can check our Facebook page for information about upcoming shows and releases!

vim.reaper: 💀 A Hackable, Fully Featured, Rice Friendly Neovim Configuration

Wed, 09 Oct 2019 00:00:00 +0000

What is vim.reaper?

Over the years, I have slowly built up a configuration for neovim, adding functionality as the need arises.

With all my keybindings, plugins, and settings, my init.nvim file was well over 700 lines long. Debugging and editing settings was a chore.

With vim.reaper, I have split that configuration file into multiple modular pieces, that can easily be tweaked to fit the needs of the individual user. Not only that, but I adhere to my practice of keeping a theme consistent where possible. vim.reaper uses the wal theme, and will highlight syntax and icons appropriately.

Additionally, you get sane default keybindings that are easy to learn. Complex functions are available with just a few intuitive keypresses!

My goal was to create a configuration that is easy to modify, grants access to IDE like features, and can be updated and deployed with ease. As I update this configuration, I will inch further towards these goals.

Please feel free to submit a pull request. I would love to make the configuration more robust across all systems, and incorporate features that are wanted by users.

Genetically Engineering Yeast to Bioluminescence: Overview of steps

Sun, 22 Sep 2019 00:00:00 +0000

Intro

In a follow up to my previous posts, I will further explore the tools, steps, and biological processes that enable trivial gene editing using the CRISPR/Cas9 system. Future posts will dive deeper into each topic as I encounter them during my experiments.

The Yeast

In theory, this process should work with almost any strain of yeast. However I have plans for this yeast, which may or may not involve using it to ferment glowing wine and beer 😉.

Therefore I have opted to use D-47 Wine Yeast, due to it’s low price and ubiquity in brewing.

In this experiment, our goal is to inject the Green Fluorescent Protein into the genome of yeast, causing them to glow green when exposed to light in the blue to ultraviolet range.

The Plasmid

We can accomplish this by inserting the GFP genetic code into a plasmid. A plasmid is a circular piece of DNA that is separate from the main chromasomal DNA, and can replicate itself independently.

Plasmid from Wikipedia

This causes the DNA present inside the plasmid to be translated and expressed within the target organism.

It can be helpful to think of plasmids as “DNA Powerups” in the sense that they can provide traits such as antibiotic resistance, which will come in handy later in the process.

In a future post devoted to plasmids, I will further investigate the characteristics and properties of plasmids, as well the specific GFP pasmid used in this experiment.

Due to my limited tools and resources, I will be using the provided plasmid in the kit I am planning on purchasing. It has been optimized for this specific application, and can provide insight on to more general plasmid design principles.

The Equipment

Here I will break down the contents of the kit to better understand the purpose of each individual piece.

Containers
- Petri Plates
- Glass bottle for pouring plates
- 50mL Tube for measuring
- Microcentrifuge tubes
- Microcentrifuge tube rack
Misc/Disposable
- Nitrile Gloves
- Inoculation loops/Plate spreader
- Pipette tips
- UV Filter sheet and blue light
Bio/Perishable
- 1 mL Yeast transformation buffer 40% PEG 8000, 200mM LiAc, 0.1mg/mL Salmon Sperm DNA
- Preengineered strain of Yeast Containing GFP Plasmid in Agar stab
- Saccharomyces cerevisiae Mead Yeast Strain
- Yeast GFP Expression plasmid 100ng/uL
Experiment Supplies
- YPD Agar
- YPD Agar containing G418 Antibiotic
- 0-100uL professional lab grade variable volume adjustable pipette

The Timeline

Here is a general timeline of the steps for this experiment. I will go into more detail about the procedures when I actually do them.

Preparation
- Make Agar plates
- Streak yeast onto plates
- Allow yeast to grow overnight
Day of experiment
- Mix the transformation mix, plasmid, and yeast sample
- Heat shock sample for 60 mins
- add YPD media to cell solution
- incubate overnight at room temp
- Plate 400uL of the yeast solution and let dry
Incubation
- Incubate at room temp for 48+ hours

Conclusion

In my next post, I will look at each part of the plasmid and figure out why it works the way that it does. How can we make smart plasmid design choices?

Genetically Engineering Yeast to Bioluminescence: Introduction

Sun, 08 Sep 2019 00:00:00 +0000

Background in Biology

One of my research interests over the past years have been bioinformatics and genetic science.

The potential in this field is unprecedented, largely due to novel breakthroughs in accessible gene editing methods such as CRISPR/Cas9, as well as DNA storage drives and other fascinating applications of the technology.

The thing that truly inspired me to dig deeper into this field was the YouTube video by the channel The Thought Emporium, where the host hacks together a DIY cure for lactose intolerance using CRISPR and the tools available to him at a reasonable price.

To address the topic of this section, I have an extremely limited knowledge of biology. The last class I took was over 6 years ago as a sophomore in high school. This presents a unique challenge compared to my CS oriented content, which I can approach with a good amount of background knowledge.

I suspect my education in chemistry will ease my understanding of certain concepts, but I will be doing most of my research from the ground up.

I invite you to follow along as I research this project, design experiments, make mistakes, and learn new things.

The Goal

The overall goal for this project is to develop a strain of yeast that carries the green fluorescent protein or GFP, in its DNA. This protein is often used as a biomarker for other experiments, and causes the target organism to glow in the dark.

From what I can tell, a simple project like this is the biology equivalent of a Hello World program. By going through this process, I hope to become familiar with the following topics and concepts:

Proper procedures and use of basic bio lab equipment
Strategies for designing effective plasmids and DNA sequences
Streamlined physical lab/experiment workflow
Industry software and tools

Now, there are kits available that let you accomplish this very task, with all of the research done for you. This may teach me appropriate lab procedures, but I want do as much of the initial plasmid design myself for learning purposes, even if I do end up using a kit due to the convenience of getting all the gear I need.

Challenges

Ideally, I would have my own little home laboratory where I could experiment to my hearts content. Being a student, this is not possible in my current financial reality.

Each piece of equipment purchased for this project will be thoroughly researched and evaluated for value and efficacy.

Unlike a computer where I can simply recompile and try again, a failed bio experiment means depleted resources, lost time, and possibly inconclusive results.

Multiple failed experiments are pretty much a guarantee in this process, as I will be learning how to properly handle and execute experiments with many moving parts and points of failure.

CRISPR Crash Course

If you haven’t heard something about CRISPR in the last 5 years, welcome back from your coma. Most people know it as the subject of recent controversy where a Chinese scientist supposedly genetically modified a human embryo.

CRISPR, which stands for Clustered Regularly Interspaced Short Palindromic Repeats, is a collection of DNA sequences commonly found in bacteria and other prokaryotes.

When a bacteria survives an attack from another organism such as a bacteriophage, it can archive sections of the attacking organism’s DNA in a database that we can refer to as CRISPR.

When the same organism attacks the bacteria again, a special protein called Cas9 is activated. Cas9 will scan all DNA, and will surgically remove any DNA that matches the archived virus genome.

While this process provides a fantastic natural defense mechanism, scientists can leverage it in order manipulate the genetic code of practically any organism at will. The flexibility and accessibility of CRISPR has enabled new frontiers of genetic research previously impractical or impossible.

This is an extremely high level overview of the topic, and we will be going over each step in great deal as I encounter it.

Overview of Steps

Here, I will present a generalized set of steps I will be taking throughout the course of this project.

Preliminary Research:

What equipment and materials do I need?
Best delivery method for GFP?
Possible experimental roadblocks?

Experiment Design and Optimization

What are the important features in a plasmid?
Can we build a more effective GFP plasmid?
Do any strains of yeast promise better results?

Preform Experiments

Ensure repeatable procedure
Take careful notes and record data
Extrapolate conclusions from data

Conclusion

If you have any experience in the field or advice for my experiments, I would love to hear it. This is an undeniably ambitious task, that could take some time to complete.

Until then, thanks for tuning in! 😄

Quantum Katas #3: Measurement

Sat, 31 Aug 2019 00:00:00 +0000

Intro

In the previous katas, we learned how to send qubits through quantum gates, and how to prepare them in various states of superposition.

In this guide, we will be taking a look at measuring qubit states, and what effect measurement has on them.

Task 1.1: |0⟩ or |1⟩ ?

Here we are tasked with returning true if the input qubit is $|1\rangle$ and false if the qubit is $|0\rangle$.

To collapse the superposition and measure a qubit, we need to use the measurement gate M().

M() returns One if the qubit is $|1\rangle$ and Zero if it is $|0\rangle$.

The simplest way to do this task is to evaluate M(q) and test that it is equal to One. The result is a boolean, which we can then return.

operation IsQubitOne (q : Qubit) : Bool {
    return (M(q) == One);
}

Task 1.2: Set qubit to |0⟩ state

In this task we are given a qubit in an arbitrary superposition. Firstly, we can measure the qubit to narrow it down to $|0\rangle$ or $|1\rangle$.

We check if it measured to be $|1\rangle$, and if so, apply an X() gate to change it to $|0\rangle$.

operation InitializeQubit (q : Qubit) : Unit {
    if(M(q) == One){
        X(q);
    }
}

Task 1.3: |+⟩ or |-⟩ ?

We are now given a qubit which is in the state $|+\rangle$ or $|-\rangle$.

Let us first recall the mappings of the Hadamard gate.

$|0\rangle \rightarrow |+\rangle$ $|1\rangle \rightarrow |-\rangle$

Also, we can leverage the reversible properties of quantum gates to undo the Hadamard gate, returning our qubit to it's original state.

If the qubit measures to be zero, that means it was just $|+\rangle$ state and we can return true.

operation IsQubitPlus (q : Qubit) : Bool {
    H(q);
    if(M(q) == Zero){
        return true;
    }
    else{
        return false;
    }
}

Task 1.4: |A⟩ or |B⟩ ?

We are given an angle $\alpha$ in radians and a qubit which will be in one of the following states:

If we measure state $|A\rangle$ we return true, and false if we measure $|B\rangle$

We can think of these states as being some rotation of a qubit by $\alpha$ degrees.

operation AmplitudeChange_Reference (alpha : Double, q : Qubit) : Unit is Adj+Ctl {
    Ry(2.0 * alpha, q);
}

Task 1.5: |00⟩ or |11⟩ ?

Here we are given a two qubit state with equal probability of being in state $|11\rangle$ and $|00\rangle$.

Since there are no possible mixed states, we only have to measure one of the qubits to determine the state of the other.

operation ZeroZeroOrOneOne (qs : Qubit[]) : Int {
    if(M(qs[0]) == One){
        return 1;
    }
    else{
        return 0;
    }
}

Task 1.6: Distinguish four basis states

This task is the same as the task above, but we must add an additional check to account for the possibility of mixed states $|01\rangle$ and $|10\rangle$.

operation BasisStateMeasurement (qs : Qubit[]) : Int {
    if(M(qs[0]) == One){
        if(M(qs[1]) == One){
            return 3; // |11⟩ 
        }
        else{
            return 2; // |10⟩ 
        }
    }
    else{
        if(M(qs[1]) == One){
            return 1; // |01⟩ 
        }
        else{
            return 0; // |00⟩ 
        }
    }
}

Task 1.7: Distinguish two basis states given by bit strings

In this task we are given two bit strings, as well as $N$ qubits that are in a state described by one of the two bitstrings.

First, we need to find where the bitstrings diverge. We can reuse our helper function from the last kata to find that index.

function FindFirstDiff (bits1 : Bool[], bits2 : Bool[]) : Int {
    for (i in 0 .. Length(bits1) - 1) {
        if (bits1[i] != bits2[i]) {
            return i;
        }
    }
    return -1;
}

Then we can measure our qubit array at the FirstDiff index. We can turn the result of our measurement into a boolean by testing it's equality to One.

If our qubit measured $|1\rangle$, our variable will be true. If we measured $|0\rangle$, our variable will be false.

Now we can check if the bit we measured belongs to the state describe by the first or second bitstring.

If the boolean at bits1[firstDiff] is the same as our measured value, we know bits1[] describes our current state and we can return 0.

operation TwoBitstringsMeasurement (qs : Qubit[], bits1 : Bool[], bits2 : Bool[]) : Int {

    // find the first index at which the bit strings are different
    let firstDiff = FindFirstDiff(bits1, bits2);
    
    // res = true if the first different qubit measures to be one
    let res = M(qs[firstDiff]) == One;
    
    // If the measurement aligns with the value in the first bitstring, return 0
    // Otherwise it's the second state, return false.
    return res == bits1[firstDiff] ? 0 | 1;
}

Task 1.8: |0...0⟩ state or W state ?

In this task, we must determine if the supplied qubits are in a $W$ state or a $|0..0\rangle$ state.

We know from our last kata that the $W$ state looks like

\[ |W_N\rangle = \frac{\overbrace{|100...0\rangle + |010...0\rangle + ... + |00..01\rangle}^{N \text{ states}}}{\sqrt{N}} \]

Notice how every possible measurement in a $W$ state includes a single $1$. If we find even a single one in our measured state, it can no longer be in state $|0...0\rangle$ and thus it must be a $W$ state.

operation AllZerosOrWState (qs : Qubit[]) : Int {
    for (q in 0 .. Length(qs) - 1){
        if(M(qs[q]) == One){
            return 1; // We found a one, it's W
        }
    }
    return 0; // it's |0...0⟩
}

Task 1.9: GHZ state or W state ?

Like the last task, we are given a qubit register and must determine which of two possible states they are in.

They can either be in a $W$ state, or a $GHZ$ state. Recall that a $GHZ$ state is:

\[ GHZ_n = \frac{ \overbrace{|0...0\rangle}^{n} + \overbrace{|1...1\rangle}^{n} }{\sqrt{2}} \]

If our qubits are in a $GHZ$ state, we can expect them to be either all $|0\rangle$ or all $|1\rangle$. If we find two of our qubits measure as different values, we can assume they are in the $W$ state.

operation GHZOrWState (qs : Qubit[]) : Int {
    for(q in 1 .. Length(qs) - 1){
        if (M(qs[q-1]) != M(qs[q])){
            return 1;
        }
    }
    return 0;
}

Task 1.10: Distinguish four Bell states

As input we receive an array of two qubits that are guaranteed to be in a Bell state.

Let us recall the four Bell states, as well as the return values we use to distinguish them in this task.

$ 0: |\Phi^+\rangle = \frac{|00\rangle + |11\rangle}{\sqrt{2}} $ $ 1: |\Phi^-\rangle = \frac{|00\rangle - |11\rangle}{\sqrt{2}} $ $ 2: |\Psi^+\rangle = \frac{|01\rangle + |10\rangle}{\sqrt{2}} $ $ 3: |\Psi^-\rangle = \frac{|01\rangle - |10\rangle}{\sqrt{2}} $

The first step is to apply the H() gate to both of our qubits.

Then, we can reverse our Bell circuit by preforming a CNOT using the second bit as control, and then applying the H() gate to it.

This will change our state to one of the following: $|00\rangle$, $|01\rangle$, $|10\rangle$, $|11\rangle$.

We can then create two integer variables my measuring each of the qubits and testing their equality to Zero or One.

We can finally compute m2 * 2 + m1 and return it.

operation BellState (qs : Qubit[]) : Int {
    H(qs[0]);
    H(qs[1]);
    
    CNOT(qs[1], qs[0]);
    H(qs[1]);

    let m1 = M(qs[0]) == Zero ? 0 | 1;
    let  m2 = M(qs[1]) == Zero ? 0 | 1;
    return m2 * 2 + m1;
}

In this Quirk circuit, we can see that we input the $|\Psi^-\rangle$ state, and we received the value $|11\rangle$. $(1 * 2) + 1 = 3$, which corresponds to $|\Psi^-\rangle$.

Task 1.11: Distinguish four orthogonal 2-qubit states

We have the four following states, one of which represents the state of the supplied qubits:

$ |S_0\rangle = \frac{|00\rangle + |01\rangle + |10\rangle + |11\rangle}{2} $ $ |S_1\rangle = \frac{|00\rangle - |01\rangle + |10\rangle - |11\rangle}{2} $ $ |S_2\rangle = \frac{|00\rangle + |01\rangle - |10\rangle - |11\rangle}{2} $ $ |S_3\rangle = \frac{|00\rangle - |01\rangle - |10\rangle + |11\rangle}{2} $

We can again apply the Hadamard gate to both our qubits to collapse them into one of the two qubit basis states $|00\rangle$, $|01\rangle$, $|10\rangle$, or $|11\rangle$.

Then we can use the solution from task 1.6 to identify the correct basis state.

operation TwoQubitState (qs : Qubit[]) : Int {
    H(qs[0]);
    H(qs[1]);
    return BasisStateMeasurement(qs);
}

Task 1.12: Distinguish four orthogonal 2-qubit states, part two

The trick to this task is applying the Hadamard gate to transform the four orthogonal states into the Bell states.

After this is achieved, we can implement code similar to task 1.10 to determine what the bell state is.

operation TwoQubitStatePartTwo (qs : Qubit[]) : Int {
    H(qs[1]);
    
    CNOT(qs[0], qs[1]);
    H(qs[0]);

    let m1 = M(qs[0]) == One ? 0 | 1;
    let m2 = M(qs[1]) == One ? 0 | 1;
    return m2 * 2 + m1;
}

Task 1.13: Distinguish two orthogonal states on three qubits

As input, we are given three qubits stored in an array. The array is guaranteed to be in one of the following states:

$ \frac{|100\rangle + \omega|010\rangle + \omega^2|001\rangle }{\sqrt{3}} $ $ \frac{|100\rangle + \omega^2|010\rangle + \omega|001\rangle }{\sqrt{3}}$

where $\omega = \text{exp}(2\pi \frac{i}{3})$. We are to output 0 if we find our qubits in the first state, 1 if we find them in the second.

In order to solve this, we must map our first state to a $W$ state, and our second state to the first.

\[ \frac{|100\rangle + \omega|010\rangle + \omega^2|001\rangle }{\sqrt{3}} \rightarrow \frac{|100\rangle + |010\rangle + |001\rangle }{\sqrt{3}} \]

\[ \frac{|100\rangle + \omega^2|010\rangle + \omega|001\rangle }{\sqrt{3}} \rightarrow \frac{|100\rangle + \omega|010\rangle + \omega^2|001\rangle }{\sqrt{3}} \]

We can achieve this by applying a unitary operation of the form $I_2 \otimes R \otimes R^2$

$$ \begin{bmatrix} 1 & 0\\ 0 & 1\\ \end{bmatrix} \otimes \begin{bmatrix} 1 & 0\\ 0 & \omega^2\\ \end{bmatrix} \otimes \begin{bmatrix} 1 & 0\\ 0 & \omega\\ \end{bmatrix} $$

Now we can take the new $W$ state and undo it into the state $|000\rangle$, by utilizing the Adjoint if the operation that let us prepare the $W$ state in the last kata.

Our second state will then be mapped to some state guaranteed to be perpendicular to $|000\rangle$. In simple terms, the second state will never be measured to be $|000\rangle$.

Now we can measure our state, which is either $|000\rangle$ (state 1) or something else (state 2). To measure this, I prefer to implement the alternate solution provided in the solutions file.

operation ThreeQubitMeasurement (qs : Qubit[]) : Int {
    // map the first state to 000 state and the second one to something orthogonal to it
    // (as described in reference solution)
    R1(-2.0 * PI() / 3.0, qs[1]);
    R1(-4.0 * PI() / 3.0, qs[2]);
    Adjoint WState_Arbitrary_Reference(qs);

    // measure all qubits: if all of them are 0, we have the first state,
    // if at least one of them is 1, we have the second state
    return MeasureInteger(LittleEndian(qs)) == 0 ? 0 | 1;
}

Conclusion

I decided to omit part two of this kata as a challenge to readers. Anywas, the descriptions given in the solution are far better than I could write.

Again, thanks for reading and giving me feedback about my work 😄.

Quantum Katas #2: Superposition

Wed, 28 Aug 2019 00:00:00 +0000

Intro

If you read my last guide on basic quantum gates, you are familiar with the concept of superposition.

If not, let me briefly explain. A classical computer calculates on a system of binary "bits" that can either occupy the state $0$ or $1$, on or off.

A quantum computer on the other hand, uses qubits, or quantum bits which can occupy infinitely many states in between $0$ and $1$. When a qubit is in one of these "hybrid" states, we say that it is in superposition.

Consider the Bloch sphere below, which provides a visual reference for the state of a qubit.

Like a classical bit, the qubit can take states $0$ and $1$, which are usually referred to as $|0\rangle$ and $|1\rangle$ when describing qubits.

We also see states $|+\rangle$ and $|-\rangle$, which represent a superposition halfway in between $|0\rangle$ and $|1\rangle$.

Finally we see the state vector $|\Psi\rangle$, which can point to an arbitrary state on the surface of the bloch sphere. These arbitrary states are represented by angles $\theta$ and $\phi$.

The following programming challenges will have you manipulate qubits into various states of superposition. If you have not set up a Q# development environment, I suggest you take a look at my last post.

Task 1: Plus state

As input, we are given a qubit in the $|0\rangle$ state, and we need to output a qubit in the $|+\rangle$ state.

Recall that the Hadamard gate maps qubits like so:

$|0\rangle \rightarrow |+\rangle$ $|1\rangle \rightarrow |-\rangle$

Since we are only given qubits in the $|0\rangle$ state, we can convert them to $|+\rangle$ by simply applying the H() operation.

operation PlusState (q : Qubit) : Unit {
    H(q);
}

Task 2: Minus state

Like the previous task, we are given a qubit in the $|0\rangle$ state, but now we must map it to the $|-\rangle$ state.

Looking at the mappings for the Hadamard gate, we can see that applying H() to a qubit in the $|1\rangle$ state will result in the $|-\rangle$ state. We need to make use of the Pauli X gate to turn our $|0\rangle$ states into $|1\rangle$.

operation MinusState (q : Qubit) : Unit {
    X(q);    
    H(q);
}

Task 3: Unequal superposition

As I mentioned in the intro, the angles $\phi$ and $\theta$ on the Bloch sphere translate into an arbitrary superposition.

Let us consider the fact that we are trying to change the probability state of the qubit, which is represented by angle $\theta$ between the $|1\rangle$ state and the state vector $|\Psi\rangle$.

Here we are tasked with putting the qubit into a superposition given by the angle $\alpha$. A hint given in the comments reads:

Experiment with rotation gates from the Microsoft.Quantum.Intrinsic namespace.

Since we know that the probability state can be changed by rotating our state vector about the $Y$ axis, we can leverage the Ry() operation.

operation UnequalSuperposition (q : Qubit, alpha : Double) : Unit is Adj {        

    Ry(2.0 * alpha, q);
}

Task 4: Superposition of all basis vectors on two qubits

We finally reached our first 2-qubit gate of this kata.

Our goal is to take two qubits in state $|00\rangle$ and convert them into the state

\[ \frac{|00\rangle + |01\rangle + |10\rangle + |11\rangle }{2}\]

Which is an equal probability of all possible outcomes of measurement.

This state is also represented by the tensor product of the two states

\[ |+\rangle \otimes |+\rangle \]

Since this superposition of qubits is represented by the tensor product of the qubits, we can simply use the H() gate to change both our qubits to the $|+\rangle$ state.

operation AllBasisVectors_TwoQubits (qs : Qubit[]) : Unit {
    H(qs[0]);
    H(qs[1]);
}

Task 5: Superposition of basis vectors with phases

Our goal in this task is to take two qubits in the $|00\rangle$ state and convert them to the state

\[ \frac{|00\rangle + i|01\rangle - |10\rangle - i|11\rangle}{2} \]

This is another example of a $n$ qubit state that can be broken down into the tensor product of $n$ discreet states. In this case, the state above can be rewritten as

\[ \frac{|0\rangle - |1\rangle}{\sqrt{2}} \otimes \frac{|0\rangle + i|1\rangle}{\sqrt{2}} = |-\rangle \otimes |i\rangle\]

We already know how to put our qubits in these states! Take another look at the Bloch sphere to orient yourself. We are aiming for the $|-\rangle$ and $|i\rangle$ states, which both lie along the $XY$ plane.

We can use the Hadamard gate to move our state vector to the $|+\rangle$ state, and then use various rotations to get to the final states.

operation AllBasisVectorsWithPhases_TwoQubits (qs : Qubit[]) : Unit is Adj {
            
        // Qubit 0 is taken into |+⟩ and then z-rotated into |-⟩.
        H(qs[0]);
        Z(qs[0]);
            
        // Qubit 1 is taken into |+⟩ and then z-rotated into |i⟩.
        H(qs[1]);
        S(qs[1]);
}

Task 6: Bell state

Again, we are getting an input of two qubits $|00\rangle$. Our task is to create the bell state $|\Phi^+\rangle$.

If you need a refresher on Bell states, please refer to my last post, where I explain the concept in more detail.

For now, we can simply recall that the $|\Phi^+\rangle$ can be prepared by giving $|00\rangle$ as an input to the Bell preparation circuit $\beta$.

\[ |\beta_{00}\rangle \rightarrow |\Phi^+\rangle \]

Since we are already given the qubits in state $|00\rangle$, we can easily prepare the desired bell state.

operation BellState (qs : Qubit[]) : Unit is Adj {        
    H(qs[0]);
    CNOT(qs[0], qs[1]);
}

Task 7: All Bell states

Here the input is the same as the last few tasks. This time however, we are also given an index, that corresponds to one of the four Bell states.

We are given an input of $|00\rangle$, so unless the input is 0, we will have to alter our qubits after preparing a bell state.

First let's consider the sign of the Bell state. If the index supplied is 1 or 3, it means we are preparing $|\Psi^-\rangle$ or $|\Phi^-\rangle$. Since our input $ |00\rangle\rightarrow |\Phi^+\rangle$, we need to flip the sign.

If you remember task 1.7 from my last post, this can be accomplished with the Z() operation.

The only thing left to check for now is if we need to go from $|\Phi^{\pm}\rangle$ to $|\Psi^{\pm}\rangle$

Fill in the operation like so to complete the task.

operation AllBellStates (qs : Qubit[], index : Int) : Unit is Adj {
    
    H(qs[0]);
    CNOT(qs[0], qs[1]);
        
    // now we have |00⟩ + |11⟩ - modify it based on index arg
    if (index % 2 == 1) {
        // negative phase
        Z(qs[1]);
    }
    if (index / 2 == 1) {
        X(qs[1]);
    }
}

Task 8: Greenberger–Horne–Zeilinger state

The Greenberger–Horne–Zeilinger state, or GHZ state, is a special, maximally entangled case of 3 or more qubits.

It takes the following form:

\[ GHZ_n = \frac{ \overbrace{|0...0\rangle}^{n} + \overbrace{|1...1\rangle}^{n} }{\sqrt{2}} \]

For example, a three qubit GHZ state is as follows

\[ GHZ_3 = \frac{|000\rangle + |111\rangle}{\sqrt{2}} \]

The observant may notice the similarity to the two qubit Bell state $|\Phi^+\rangle$. In fact, the GHZ state can be prepared in a similar way.

Instead of preforming a Controlled NOT with the only two qubits, we can use the first qubit as the control and CNOT every additional qubit with it.

operation GHZ_State (qs : Qubit[]) : Unit {
    H(qs[0]);
    
    for(i in 1 .. Length(qs) - 1){
        CNOT(qs[0], qs[i]);
    }
}

In Q#, you can use the Rest() operation, which creates a copy of the input list removing the first element. This is the method shown in the solution, but you will have to add

open Microsoft.Quantum.Arrays;

to the top of your Tasks.qs file.

operation GHZ_State (qs : Qubit[]) : Unit {
    H(qs[0]);
    
    for(qubit in Rest(qs)){
        CNOT(qs[0], qubit);
    }
}

Task 9: Superposition of all basis vectors

In this task, we need to create an equal superposition of $n$ vectors. That will look like

\[ \overbrace{\frac{|0...0\rangle + ... + |1...1\rangle}{\sqrt{2^n}} }^{2^n \text{ permutations}} \]

For example:

\[ n =3 \rightarrow \frac{|000\rangle + |001\rangle + |010\rangle + |011\rangle + |100\rangle + |101\rangle + |110\rangle + |111\rangle}{\sqrt{2^3}} \]

This can be achieved by simply applying the Hadamard gate to every qubit. This is the same process we did for task 1.4, except with $n$ qubits.

operation AllBasisVectorsSuperposition (qs : Qubit[]) : Unit is Adj {
    for (q in qs) {
        H(q);
    }
}

Task 10. |00⟩ + |01⟩ + |10⟩ state

Here we are tasked with putting two qubits into the state

\[ |\Psi\rangle = \frac{|00\rangle + |01\rangle + |10\rangle}{\sqrt{3}} \]

This means our 2 qubit state has a $\frac{1}{3}$ chance of collapsing into each of the states $|00\rangle, |01\rangle, |10\rangle$, with no chance of being in the state $|11\rangle$

I recommend taking a look at the stack exchange solution for more detail, but I will explain briefly here.

The first qubit in our input can be considered a control bit. For example, if our input is $|10\rangle$, no action is needed and the state will measure $|10\rangle$.

If the first qubit is $|0\rangle$, we need to put our second qubit into the $|+\rangle$ state, which will give an equal probability of measuring $|00\rangle$ and $|01\rangle$.

Now we can think of our qubit in the following form:

\[ |\Psi\rangle = \frac{\sqrt{2}}{\sqrt{3}}|0\rangle|+\rangle + \frac{1}{\sqrt{3}}|1\rangle|0\rangle\]

Which gives it a 66% chance of giving the plus state (which covers $|01\rangle$ and $|00\rangle)$ and a 33% chance of going straight to $|10\rangle$. This neatly gives us our $\frac{1}{3}$ chance of measuring any of the states.

How do we replicate this probability with our qubits? Well we want our first qubit to be in the $|0\rangle$ state $\frac{2}{3}$ of the time, so we have to adjust our probability amplitude.

Then for our conditional logic, we can use a "reverse controlled Hadamard", which will put the second qubit in the $|+\rangle$ state if the first qubit is 0.

operation ThreeStates_TwoQubits (qs : Qubit[]) : Unit is Adj {
    
    // Follow Niel's answer at https://quantumcomputing.stackexchange.com/a/2313/
        
    // Rotate first qubit to (sqrt(2) |0⟩ + |1⟩) / sqrt(3) (task 1.4 from BasicGates kata)
    let theta = ArcSin(1.0 / Sqrt(3.0));
    Ry(2.0 * theta, qs[0]);
        
    // Split the state sqrt(2) |0⟩ ⊗ |0⟩ into |00⟩ + |01⟩
    (ControlledOnInt(0, H))([qs[0]], qs[1]);
}

Task 11: Superposition of |0...0⟩ and given bit string

In this task, we must create a superposition of $|0...0\rangle$ and an input given by a bit string.

The bit string:

Has the same length as the qubit register
Is guaranteed to start with a 1

For example, if the bit string was [true, false], the state we want to create would be

\[ \frac{|00\rangle + |10\rangle}{\sqrt{2}} \]

Here is a circuit that illustrates this on the input [true, false, true].

We apply the Hadamard gate to the first qubit, starting the superposition. For each following qubit, the CNOT gate is applied if the corresponding entry in the bit string is true.

The first qubit has a 50% chance of being $|0\rangle$ or $|1\rangle$. If it is zero, no more operations are applied, so the state $|000\rangle$ is measured.

If the state of the first qubit (qs[0]) is one, all possible states that start with $|0..\rangle$ are no longer possible.

In our above example, the second bit was false, meaning it will never get a NOT and will remain $|0\rangle$. Hence, any state $|.1.\rangle$ with 1 in the middle are not possible.

Similarly, a measurement of $|100\rangle$ would be impossible since if the first qubit was $|1\rangle$, the third qubit would have the NOT gate applied.

After eliminating the impossible measurements, the probability matrix gives us an equally probable output of $|000\rangle$ and $|101\rangle$.

Here I used the Length() function in a ranged for loop to check the value of the $i^{th}$ bit in the input and apply CNOT accordingly.

operation ZeroAndBitstringSuperposition(qs : Qubit[], bits : Bool[]) : Unit is Adj{
        
    // Hadamard first qubit
    H(qs[0]);
        
    // iterate through the bit string and CNOT to qubits corresponding to true bits
    for (i in 1 .. Length(qs) - 1) {
        if (bits[i]) {
            CNOT(qs[0], qs[i]);
        }
    }
}

Task 12: Superposition of two bit strings

In this task, we are given two arbitrary bit strings of length $n$. For example if we had two input strings [false, true, false] and [false, false, true], we would create an equal superposition of like this:

\[ \frac{|010\rangle + |001\rangle}{\sqrt{2}} \]

The first step in completing this task is to find the spot where the two bitstrings "diverge".

By putting this bit into superposition, we splits our state into two distinct possible outcomes. Here is a helper function that returns the index of the first differing qubit:

function FindFirstDiff (bits1 : Bool[], bits2 : Bool[]) : Int {
    mutable firstDiff = -1;
    for (i in 0 .. Length(bits1) - 1) {
        if (bits1[i] != bits2[i] and firstDiff == -1) {
            set firstDiff = i;
        }
    }
    return firstDiff;
}

Once we found where the strings diverge, we can apply the H() gate to it to create superposition. Next, we have to iterate through our bitstrings and put the qubits in their final state.

If the bit in both strings at index $i$ are the same, we have two possibilities:

They are both false, meaning no action is needed.
They are both true, and we need to NOT our qubit register at index $i$

If the bit in both strings at index $i$ are different, we use the first different bit as a control, and CNOT the qubit at index $i$.

If the first differing qubit measures as $|1\rangle$, then the $i^{th}$ qubit is flipped to $|1\rangle$.

Then, if the first bitstring at index $i$ is not equal to the value of the first different bit from the first string, we NOT the qubit at index $i$.

The pseudocode above can be realized with the following operation:

operation TwoBitstringSuperposition (qs : Qubit[], bits1 : Bool[], bits2 : Bool[]) : Unit is Adj {
    
    // find the index of the first bit at which the bit strings are different
    let firstDiff = FindFirstDiff (bits1, bits2);
        
    // Hadamard corresponding qubit to create superposition
    H(qs[firstDiff]);
        
    // iterate through the bit strings again setting the final state of qubits
    for (i in 0 .. Length(qs) - 1) {
        if (bits1[i] == bits2[i]) {
            // if two bits are the same apply X or nothing
            if (bits1[i]) {
                X(qs[i]);
            }
        } else {
            // if two bits are different, set their difference using CNOT
            if (i > firstDiff) {
                CNOT(qs[firstDiff], qs[i]);
                if (bits1[i] != bits1[firstDiff]) {
                    X(qs[i]);
                }
            }
        }
    }
}

Task 13: Superposition of four bit strings

This task is like the one above, except we must deal with 4 bitstrings, with $N$ elements each.

Elements can be accessed like bits[i][j] where $0 \lt i \lt 3$ and $0 \lt j \lt N - 1 $.

First we can use the ApplyToEach() function to apply the Hadamard gate to each of the ancilla qubits to put them in the $|+\rangle$ state.

Then we iterate through the rows and columns of our 2D array, touching each qubit. If bits[i][j] is true, we preform the following operation: (ControlledOnInt(i, X))(anc, qs[j]);

Then we have to iterate through each bitstring. If we are on bits[i] and $i$ is odd, we do

(ControlledOnBitString(bits[i], X))(qs, anc[0]);

If it's even, we do

(ControlledOnBitString(bits[i], X))(qs, anc[1]);

Here is the implementation.

operation FourBitstringSuperposition (qs : Qubit[], bits : Bool[][]) : Unit is Adj {
    let N = Length(qs);
        
    using (anc = Qubit[2]) {
        // Put two ancillas into equal superposition of 2-qubit basis states
        ApplyToEachA(H, anc);
            
        // Set up the right pattern on the main qubits with control on ancillas
        for (i in 0 .. 3) {
            for (j in 0 .. N - 1) {
                if ((bits[i])[j]) {
                    (ControlledOnInt(i, X))(anc, qs[j]);
                }
            }
        }
            
        // Uncompute the ancillas, using patterns on main qubits as control
        for (i in 0 .. 3) {
            if (i % 2 == 1) {
                (ControlledOnBitString(bits[i], X))(qs, anc[0]);
            }
            if (i / 2 == 1) {
                (ControlledOnBitString(bits[i], X))(qs, anc[1]);
            }
        }
    }
}

Task 14: W state on 2ᵏ qubits

A W state is a state of $N$ qubits that takes the following form:

\[ |W_N\rangle = \frac{\overbrace{|100...0\rangle + |010...0\rangle + ... + |00..01\rangle}^{N \text{ states}}}{\sqrt{N}} \]

In this task, we are challenged with creating the state $|W_N\rangle$ where $N = 2^k$.

We will implement a recursive operation to create the $W$ state.

In this operation, our base case is when $N = 1$, where the $W$ state is trivially prepared. If we are in this state, we can apply X(qs[0]) and return.

If not, we must preform the following steps:

Create new variable $K = \frac{N}{2} $
Recursively call the function passing it qs[0 .. K - 1] as input
Create an ancilla qubit and apply H() to it
For every qubit from 0 to K - 1, do a controlled swap using the ancilla, qs[i], and qs[i + K]
For every qubit from 0 to N - 1, preform a CNOT with qs[i] and the ancilla.

operation WState_PowerOfTwo (qs : Qubit[]) : Unit is Adj {
        
        let N = Length(qs);
            
        if (N == 1) {
            // base of recursion: |1⟩
            X(qs[0]);
        } 

        else {
            let K = N / 2;
                
            // create W state on the first K qubits
            WState_PowerOfTwo (qs[0 .. K - 1]);
                
            // the next K qubits are in |0...0⟩ state
            // allocate ancilla in |+⟩ state
            using (anc = Qubit()) {
                H(anc);
                    
                for (i in 0 .. K - 1) {
                    Controlled SWAP([anc], (qs[i], qs[i + K]));
                }
                for (i in K .. N - 1) {
                    CNOT(qs[i], anc);
                }
            }
        }
}

Task 15: W state on arbitrary number of qubits

Our task here is the same as in the previous, except now we cannot assume that $N = 2^k$.

Again, we find the length of are qubit register using the Length() function.

Our base case for this recursive function is when $N = 1$, where we can again preform X(qs[0]).

If we aren't at the base case, we want to preform a rotation on the first qubit to put it into the following state:

\[ \sqrt{\frac{N-1}{N}}|0\rangle + \frac{1}{\sqrt{N}}|1\rangle \]

We can accomplish this with an Ry() operation on the first qubit, rotating by $2\theta$, where

\[ \theta = sin^{-1}\left(\frac{1}{\sqrt{N}}\right) \]

Then we preform a zero controlled recursive call of the same function. A zero controlled gate can be preformed by applying a standard controlled gate to a qubit who's state has been flipped.

This recursive loop will continue until we are at the base state, and where we will preform X(qs[0]) and reach our target state.

operation WState_Arbitrary (qs : Qubit[]) : Unit is Adj + Ctl {
    
    let N = Length(qs);
        
    if (N == 1) {
        // base case of recursion: |1⟩
        X(qs[0]);
    } 

    else {
        // |W_N⟩ = |0⟩|W_(N-1)⟩ + |1⟩|0...0⟩
        // do a rotation on the first qubit to split it into |0⟩ and |1⟩ with proper weights
        // |0⟩ -> sqrt((N-1)/N) |0⟩ + 1/sqrt(N) |1⟩
        let theta = ArcSin(1.0 / Sqrt(IntAsDouble(N)));
        Ry(2.0 * theta, qs[0]);
            
        // do a zero-controlled W-state generation for qubits 1..N-1
        X(qs[0]);
        Controlled WState_Arbitrary (qs[0 .. 0], qs[1 .. N - 1]);
        X(qs[0]);
    }
}

Conclusion

In this kata, we learned about the various special states and properties of quantum superposition, as well as some nontrivial implementations of these states with Q#.

As always, feel free to reach out with questions or feedback 😄

Quantum Katas #1: Basic Quantum Gates

Wed, 21 Aug 2019 00:00:00 +0000

I have been fascinated with quantum computing since the day I learned of its existence.

I picked up the Microsoft Quantum Development Kit as soon as it was released, but being far from a quantum physicist, I was understandably lost.

Since then, I have done a fair bit of research on the subject, and Microsoft has released a set of tutorials called Quantum Katas to help developers get their feet wet in the world of quantum computing.

Using these Katas, I hope to introduce you to some fundamental concepts in quantum computing and quantum mechanics.

Getting ready to write quantum code

In order to work with Quantum programs, you need to have your quantum development environment set up. This is as easy as installing Visual Studio or VS Code and installing the Microsoft Quantum Development Kit extension with the included extension manager.

In addition to this, you will need to have the dotnet runtime and development kit available to your system. On Arch Linux, this can be accomplished with

sudo pacman -Syu dotnet-runtime dotnet-sdk code

If everything went well, you should be able to run the quantum examples that come with the QDK.

Getting the Katas

The Katas can be acquired from their github repository.

Usually I would do my development in Neovim, but I opted to stick with VS Code for this project because it is better integrated with the language (also being a Microsoft product). Syntax highlighting and other hints are nice to have when working with an unfamiliar language.

The Katas assume a test driven workflow, so you can track your progress through each tutorial. In the BasicGates folder, run the dotnet test command to run the Tests.qs file and test your solutions.

Before starting this guide, I recommend checking Microsoft's readings on the qubit, which goes over some of the mathematical and visual representations I will be using in this tutorial.

Task 1.1: State flip

This is possibly the most basic quantum gate, with a counterpart in classical computation. Analogous to the classical NOT gate, the Pauli X gate takes a single qubit and flips its state.

That is, a qubit in state $|0\rangle$ will be flipped to state $|1\rangle$ and vice versa. This operation can be represented with the following linear transformations.

$$ \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix} $$

$$ \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \end{bmatrix} $$

To generalize, the effect of the $X$ gate on an arbitrary qubit is as follows:

$$ \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} \alpha \\ \beta \end{bmatrix} = \begin{bmatrix} \beta \\ \alpha \end{bmatrix} $$

Where $\alpha$ and $\beta$ are complex numbers that represent the probability of measuring the qubit in state $|0\rangle$ and $|1\rangle$ respectively. Simply fill the code in like so to complete this task.

operation StateFlip (q : Qubit) : Unit is Adj {
    X(q);
}

Quantum gates have the unique property of being reversible. For every unitary transformation $ U $, there exists an adjoint transformation $ U^\dagger $ that reverses the operation, such that

$$ U^\dagger \cdot U = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $$

Like the classical NOT gate, the Pauli X gate has the property of being self-adjoint, meaning it is its own inverse. Applying the operation twice will result in the initial state. Multiple quantum operations can be represented as the matrix product of all the transformations.

$$ \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $$

Here we can see applying the X gate twice results in the identity matrix, meaning no transformation occurred.

Task 1.2: Basis change

This is the first truly quantum gate we have encountered in this tutorial. This task requires the use of the Hadamard gate, which can be represented by the following unitary matrix.

$$ H = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} $$

In ket notation, the Hadamard gate looks like this.

\[ H(|0\rangle) = \frac{|0\rangle + |1\rangle}{\sqrt{2}} = |+\rangle \]

\[ H(|1\rangle) = \frac{|0\rangle - |1\rangle}{\sqrt{2}} = |-\rangle \]

And in matrix notation

$$ H(|0\rangle) = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \frac{|0\rangle + |1\rangle}{\sqrt{2}} $$

$$ H(|1\rangle) = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \frac{1}{\sqrt{2}} \begin{bmatrix} 1 \\ -1 \end{bmatrix} = \frac{|0\rangle - |1\rangle}{\sqrt{2}} $$

This transformation takes the qubit from the computational basis $(|0\rangle |1\rangle)$ to the polar basis $(|+\rangle |-\rangle)$. To see the effect on the state of a qubit, let us consider the following qubit state

\[|\Psi \rangle = \alpha |0\rangle + \beta |1\rangle\]

where $\alpha$ and $\beta$ again represent the normalized probability amplitudes of the quantum state. By applying the Hadamard gate to this state, we can reduce the quantum state like so.

\[H(|\Psi \rangle) = H(\alpha |0\rangle + \beta |1\rangle) = \alpha \left(\frac{|0\rangle + |1\rangle}{\sqrt{2}}\right) + \beta \left(\frac{|0\rangle - |1\rangle}{\sqrt{2}}\right) = \frac{\alpha + \beta}{\sqrt{2}}|0\rangle + \frac{\alpha - \beta}{\sqrt{2}}|1\rangle \]

This transformation alters the probability amplitudes of the state. Like the X gate, the Hadamard gate is self adjoint.

To complete task two, fill in the operation like so

operation BasisChange (q : Qubit) : Unit is Adj {
    H(q);
}

Task 1.3: Sign flip

The objective in this task is to take a qubit in the $|+\rangle$ state and flip it to $|-\rangle$, and vice versa.

By looking at the Bloch sphere, we can see that the state $|+\rangle$ or $\frac{|0\rangle + |1\rangle}{\sqrt{2}}$ is represented by a vector pointing directly along the x axis in the positive direction. To negate this, we must make our quantum state vector point along the x axis in the negative direction.

This can be accomplished by rotating out vector $\pi$ radians or $180^\circ$ about the $Z$ axis. Like the Pauli X gate rotated our state vector about the $X$ axis, the Pauli Z gate will rotate our vector about the $Z$ axis.

Like all of the Pauli gates, Pauli Z is self adjoint.

The effect of this quantum gate looks like this:

Although $\beta \rightarrow -\beta$, $|-\beta|^2 + |\alpha|^2 = 1$ so normalization holds.

Fill in the operation like so to complete the task.

operation SignFlip (q : Qubit) : Unit is Adj {
    Z(q);
}

Task 1.4: Amplitude change

In this section, our task is to change the amplitude of our qubit.

Looking at the Bloch sphere, we see that amplitude is represented by the angle $\theta$, the angle measured between our state vector $\Psi$ and the $Z$ axis.

To change this angle, we must rotate out state vector about the $Y$ axis. Looking at the documentation on the $R_y$ operation, we can see it takes the form of the following unitary matrix.

$$ \begin{bmatrix} cos \frac{\theta}{2} & -sin \frac{\theta}{2}\\ sin \frac{\theta}{2} & cos \frac{\theta}{2}\\ \end{bmatrix} $$

The comments above this task provide the following hints:

If the qubit is in state |0⟩, change its state to cos(alpha)*|0⟩ + sin(alpha)*|1⟩. 
If the qubit is in state |1⟩, change its state to -sin(alpha)*|0⟩ + cos(alpha)*|1⟩. 
If the qubit is in superposition, change its state according to the effect on basis vectors.

operation AmplitudeChange (alpha : Double, q : Qubit) : Unit is Adj+Ctl {
    Ry(2.0 * alpha, q);
}

Task 1.5: Phase Flip

On the Bloch sphere, this would map our qubit's state from $|X\rangle$ to $|Y\rangle$, introducing an imaginary component to our $|1\rangle$ term.

The perceptive may have noticed that this is simply a rotation about the $Z$ axis by $90^\circ$ or $\frac{\pi}{2}$ radians. In fact, Quantum Inspire's page about the S gate explicitly says

The S gate is also known as the phase gate or the Z90 gate, because it represents a 90 degree rotation around the z-axis.

In physical terms, this changes the angle $\Phi$ without altering $\theta$, the angle between the vector and the $Z$ axis.

In mathematical terms, we introduce an imaginary element into our $|0\rangle$ component.

$$ \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{bmatrix} \rightarrow \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{i}{\sqrt{2}} \end{bmatrix} $$

The $S$ gate can be represented by the following matrix:

$$ S = \begin{bmatrix} 1 & 0\\ 0 & i \end{bmatrix} $$

Fill in the following code to complete the operation:

operation PhaseFlip (q : Qubit) : Unit is Adj+Ctl {
    S(q);
}

Task 1.6: Phase change

Let us remember that the phase of a qubit is represented by the angle $\Phi$ on the bloch sphere. This is the angle made between the $X$ axis and the projection of our state vector on the $XY$ plane.

To alter this angle, we must rotate our state vector about the $Z$ axis by $\alpha$ degrees.

Using the knowledge we have gained so far, one might assume the $\sigma_z$ or $Z$ rotation gate could accomplish this. However, the following snippet will result in the test failing.

operation PhaseChange (alpha : Double, q : Qubit) : Unit is Adj+Ctl {
    Rz(alpha, q);
}

Looking at the solutions file, we see that the intended solution is the $R1$ operation. This operation rotates the vector about the $|1\rangle$ state.

operation PhaseChange (alpha : Double, q : Qubit) : Unit is Adj+Ctl {
    R1(alpha, q);
}

To be completely truthful, I am unsure why this is the case. The documentation for the $R1$ operation says that it equivalent to R(PauliZ, theta, qubit);, or a rotation about the $Z$ axis by theta degrees.

However, the test will fail if Rz or R operations are used. If anyone knows the reason behind this, please do enlighten me.

Bell States

Before we continue to the next task, let us briefly unpack an important concept in quantum information.

To put it simply, a Bell state or EPR pair is a pair of maximally entangled qubits. The four Bell states are as follows.

\[ |\Psi^+\rangle = \frac{|01\rangle + |10\rangle }{\sqrt{2}} = |\beta_{01}\rangle \]

\[ |\Psi^-\rangle = \frac{|01\rangle - |10\rangle }{\sqrt{2}} = |\beta_{11}\rangle \]

\[ |\Phi^+\rangle = \frac{|00\rangle + |11\rangle }{\sqrt{2}} = |\beta_{00}\rangle \]

\[ |\Phi^-\rangle = \frac{|00\rangle - |11\rangle }{\sqrt{2}} = |\beta_{10}\rangle \]

The arbitrary Bell state $|\beta_{xy}\rangle$ can be prepared with the following quantum circuit.

Diagram made with LaTex and Qcircuit

This diagram shows qubit $x$ being put into the hadamard gate, which gives it a $\frac{1}{2}$ chance of being found in the 0 and 1 position. Then, this qubit is used as the control in a CNOT gate. The state of qubit $y$ will only be flipped if qubit $x$ is in state $|1\rangle$

To see how this works in detail, check out this quantum circuit tool. Here I have prepared the $|\Psi^+\rangle$ Bell state. Mess around with the inputs to generate the different states.

Task 1.7: Bell State Change 1

Our goal in this task is to take two qubits prepared in the bell state $|\Phi^+\rangle$ and transform them to the bell state $|\Phi^-\rangle$

We recall from task 1.3 that a sign flip of a single qubit state can be accomplished with the Pauli Z gate.

Can we extrapolate this to work on a multi-qubit state?

operation BellStateChange1 (qs : Qubit[]) : Unit is Adj+Ctl {
    Z(qs[0]);
    // alternatively Z(qs[1]);
}

Indeed we can. Applying the Z gate to either of the qubits will change them to the $|\Phi^-\rangle$ state.

In Q#, a quantum register is treated like an array. Elements within the register can be indexed and operated upon for multi-qubit states and operations.

Task 1.8: Bell State Change 2

In this task, we take the Bell state $|\Phi^+\rangle$ to the state $|\Psi^+\rangle$. Applying the Pauli X gate to one qubit in a Bell state will keep the sign the same but alter the possible measured values.

In simple terms, this would take $|\Phi^\pm\rangle \rightarrow |\Psi^\pm\rangle$ and vice versa.

The $$|\Phi^+\rangle$$ Bell state

Applying the X gate gives us the $$|\Psi^+\rangle$$ state

Fill in the operation like so to complete the task:

operation BellStateChange2 (qs : Qubit[]) : Unit is Adj+Ctl {
    X(qs[0]);
    // alternatively X(qs[1]);
}

Task 1.9: Bell State Change 3

In our final Bell state task, we must combine the things we have learned so far to transform $|\Phi^+\rangle$ to $|\Psi^-\rangle$.

As you may have guessed, we can combine the X and Z gates to change the basis and sign.

operation BellStateChange3 (qs : Qubit[]) : Unit is Adj+Ctl {
    X(qs[0]);
    Z(qs[0]);
}

As the comment in the code mentions, this operation is not equivalent to Y(qs[0]), because the Y gate adds a global phase that the X and Z gates do not.

Task 2.1: Two-qubit gate - 1

Let's closely examine the spec for this task to isolate what the true goal is here.

Goal: Change the two-qubit state to α |00⟩ + β |11⟩.

The keen eyed might recognize the measurement basis of $|00\rangle$ and $|11\rangle$ being the same as the Bell states $|\Phi^\pm\rangle$

The first qubit will be in state |ψ⟩ = α |0⟩ + β |1⟩, the second in state |0⟩

Since our first qubit is already in superposition, there is no need to apply the H gate. All that is left to do is apply a CNOT gate, using our first qubit as a control.

operation TwoQubitGate1 (qs : Qubit[]) : Unit is Adj {
    CNOT(qs[0], qs[1]);
}

We have just created a pair of entangled qubits. Unlike the Bell states, these qubits are not necessarily maximally entangled, with amplitudes $\frac{1}{\sqrt{2}}$. The amplitudes $\alpha$ and $\beta$ can represent any complex normalized probability amplitude.

Task 2.2: Two-qubit gate - 2

We have been given two qubits prepared in the $|+\rangle$ state. Recall that

\[ |+\rangle = \frac{|0\rangle + |1\rangle}{\sqrt{2}} \]

Since all of our $n$ qubits are in a superposition of $|0\rangle$ and $|1\rangle$, it makes sense that measuring our qubits would result in $2^n$ possible outcomes.

What's more, the $|+\rangle$ state represents an exact superposition between $|0\rangle$ and $|1\rangle$, with a $\frac{1}{2}$ chance of being measured in either state.

A probability of 25% for all states

Simple statistics tell us that the probability of measuring any of the states would be $\frac{1}{2}^n$, which in this case is $\frac{1}{4}$.

The goal of this task is to change our state like so:

\[ \frac{|00\rangle + |01\rangle + |10\rangle + |11\rangle}{2} \rightarrow \frac{|00\rangle + |01\rangle + |10\rangle - |11\rangle}{2} \]

Looking at the circuit above, this is the same thing as negating the val: field for the amplitude of $|11\rangle$

As we learned in task 1.7, changing the sign of the state can be accomplished with the Z gate. Let's see what happens when we apply the Z gate to our qubit.

Placing the Z gate on first and second qubit negated the value of the second column and second row respectively. However, we just want to isolate the $|11\rangle$ state.

This can be done with a Controlled Z gate. Similar to CNOT gate, CZ(x, y) preforms a Z operation on qubit $y$ if qubit $x$ is in the $|1\rangle$ state.

To understand this, let's look at the matrix representation of all possible inputs. If we represent our two-qubit state as a 4-dimensional vector $|\Psi\rangle$, where

$$ |\Psi\rangle = \begin{bmatrix} \alpha_{00} \\ \alpha_{01} \\ \alpha_{10} \\ \alpha_{11} \\ \end{bmatrix} $$

and $\alpha_{xy}$ is the normalized probability amplitude of measuring our qubits in the state $|XY\rangle$, we can see the effect of the Controlled Z gate on all our possible basic inputs.

$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & -1 \\ \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \\ \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \\ 0 \\ 0 \\ \end{bmatrix} $$

$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & -1 \\ \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \\ 0 \\ 0 \\ \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \\ 0 \\ 0 \\ \end{bmatrix} $$

$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & -1 \\ \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 0 \\ 1 \\ 0 \\ \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 1 \\ 0 \\ \end{bmatrix} $$

$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & -1 \\ \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 0 \\ 0 \\ 1 \\ \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ 0 \\ -1 \\ \end{bmatrix} $$

We can see that only the $|11\rangle$ state was changed, just as intended.

Fill in the operation as shown below. You can use the intrinsic CZ gate or the Z gate with the Controlled modifier.

operation TwoQubitGate2 (qs : Qubit[]) : Unit is Adj {
    Controlled Z([qs[0]], qs[1]);
    // alternatively: CZ(qs[0], qs[1]);
}

Task 2.3: Two-qubit gate - 3

Again, we are given an array of two qubits, this time in some arbitrary state

And we are tasked with transforming it to the state

Which effectively swaps the probability of finding our qubits in state $|01\rangle$ and $|10\rangle$. In vector form, this can re represented like this:

$$ \begin{bmatrix} \alpha_{00} \\ \alpha_{01} \\ \alpha_{10} \\ \alpha_{11} \\ \end{bmatrix} \rightarrow \begin{bmatrix} \alpha_{00} \\ \alpha_{10} \\ \alpha_{01} \\ \alpha_{11} \\ \end{bmatrix} $$

If you recently took a linear algebra course, you may have been able to derive the matrix to achieve this transformation. This is the matrix representation of the SWAP gate.

$$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

The task also hints that this test can be passed with "several (possibly controlled) Pauli gates". Much like the classical SWAP gate, a quantum SWAP gate can be expressed with CNOT gates.

Diagram from 'Quantum Computation and Quantum Information'

Write the operation like so to pass the test.

operation TwoQubitGate3 (qs : Qubit[]) : Unit is Adj {
    CNOT(qs[0], qs[1]);
    CNOT(qs[1], qs[0]);
    CNOT(qs[0], qs[1]);
    // alternatively: SWAP(qs[0], qs[1]);
}

Task 2.4: Toffoli Gate

For this task, we have been given a generous three qubit state $|\Psi\rangle$ to work with.

The goal is as follows:

Flip the state of the third qubit if the state of the first two is |11⟩

This means changing the state to

In previous tasks, we have used Controlled gates to implement condition based operations. How might we add another conditional layer? By simply adding another control bit!

This is the Toffoli gate, a universal gate in both classical and quantum computing.

Diagram from 'Quantum Computation and Quantum Information'

The Toffoli gate negates the third qubit if and only if the first two qubits are in the $|11\rangle$ state.

Here is the correct code for the task:

operation ToffoliGate (qs : Qubit[]) : Unit is Adj {
    CCNOT(qs[0], qs[1], qs[2]);
    // alternatively (Controlled X)(qs[0..1], qs[2]);
}

Task 2.5. Fredkin gate

Like the Toffoli gate from the last task, the Fredkin gate is a universal gate in classical and quantum computing.

The Fredkin gate, also known as the Controlled Swap, swaps the 2nd and 3rd qubit only if the first qubit is in the $|1\rangle$ state.

\[ \downarrow \]

Fredkin circuit with matrix representation

Although the Fredkin gate is extremely common in larger quantum circuits, the easiest way to express it here is with the following code:

operation FredkinGate (qs : Qubit[]) : Unit is Adj {
    Controlled SWAP([qs[0]], (qs[1], qs[2]));
}

Conclusion

In this article, I covered some of the fundamental concepts of quantum mechanics by using example problems from Microsoft's Q# Quantum Katas.

I went over some of the basic quantum gates and how they can be applied to your quantum programs.

I am still new to this field, so please do not hesitate to contact me with questions or corrections.

Setting Up An Automated Home Media Server With Docker

Tue, 13 Aug 2019 00:00:00 +0000

With streaming services becoming more fragmented, getting access to all your favorite content has become almost as inconvenient and expensive as it was in the days of cable packages.

Even if you happen to find your favorite content on a streaming platform, there’s no guarantee it will be around forever. A single discontinued licensing contract could mean losing access to countless pieces of media!

For these reasons, I have built up a fairly large music collection on my hard drive. I recently realized I needed a proper backup solution for my music and other important files.

I figured while I was at it, I would make my music collection streamable, because there’s lots of stuff Spotify and others don’t have. Then I figured since I was hosting music, I might as well start building a collection of movies and television for when I inevitably tire of paying for Netflix and Hulu.

Here’s a breakdown of the server I built:

PCPartPicker Part List

Type	Item	Price
CPU	AMD Ryzen 3 2200G 3.5 GHz Quad-Core Processor	$79.89 @ OutletPC
Motherboard	Gigabyte B450M DS3H Micro ATX AM4 Motherboard	$73.99 @ Amazon
Memory	Corsair Vengeance LPX 8 GB (1 x 8 GB) DDR4-2400 Memory	$39.99 @ Amazon
Storage	Toshiba X300 8 TB 3.5" 7200RPM Internal Hard Drive	$167.99 @ Amazon
Case	Deepcool TESSERACT BF ATX Mid Tower Case	$49.99 @ B&H
Power Supply	Silverstone 300 W 80+ Bronze Certified SFX Power Supply	$61.99 @ Amazon
Prices include shipping, taxes, rebates, and discounts
Total	$473.84
Generated by PCPartPicker 2019-08-13 13:51 EDT-0400

I went cheap for this build, because all it needs to do is reliably store media files. I gave it a decent CPU and 8GB of memory so streaming this media to a couple clients should be no problem.

Software Stack

This diagram should give an overview of how our software talks to each other. Basically;

All programs are run in docker for portability and convenience
Sonarr watches usenet for TV shows I want
Couchpotato watches usenet for movies I want
SABnzbd downloads the NZBs found by Sonarr and Couchpotato
Jellyfin serves these media files with a pretty interface
Traefik generates SSL certs for the desired containers

Step 1: Setting up Docker

In this tutorial I will assume you are a docker novice. If you already have some familiarity with docker, you can probably skip to the next section.

Since I am running my machine with Ubuntu Server, I am going to install the necessary packages with the following command:

sudo apt install docker.io docker-compose

docker is a program that runs applications in containers. These containers share the kernelspace with the host machine, but isolate their own dependencies and runtime environments. This is great for application security and portability.

docker-compose is a functionality within docker that allows a series of containers to be preconfigured, so they can all be launched, maintained, and updated easily and conveniently.

I set up my “programs” folder with a docker-compose.yaml file, along with subfolders for every application to store configurations and metadata.

Step 2: Setting up Traefik

Traefik is a reverse proxy that integrates with Let’sEncrypt to dynamically provide SSL certificates to running applications. It will be able to securely direct HTTP requests to our server to the correct container.

If you aren’t planning on using these services outside your home network, it’s safe to skip this step. If you own a domain and wish to access your media anywhere, read on.

Traefik needs two files to work correctly. The Traefik configuration file, traefik.toml, and the configuration defining the container behaivior in docker-compose.yaml. Let’s take a look at that first.

version: '3'

services:      
   reverse-proxy:
      image: traefik:latest
      restart: always
      container_name: traefik
      ports: 
         - 80:80
         - 443:443
         - 8081:8081
      expose:
         - 8080
      networks:
         - traefik
      environment:
         - DO_AUTH_TOKEN=<or another DNS provider if you use one>
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - ./Traefik/traefik.toml:/traefik.toml
         - ./Traefik/acme.json:/acme.json
      labels: 
         - "traefik.backend=traefik"
         - "traefik.docker.network=traefik"
         - "traefik.enable=false"
         - "traefik.frontend.rule=Host:Host:monitor.mydomain.com"
         - "traefik.port=8080"
         - "traefik.frontend.headers.forceSTSHeader=true"
         - "traefik.frontend.headers.STSSeconds=315360000"
         - "traefik.frontend.headers.STSIncludeSubdomains=true"
         - "traefik.frontend.headers.STSPreload=true"

Some things to take away:

We need to bind ports 80 and 443 to receive HTTP/S traffic. These ports must be forwarded to your server from your router.
We mount our docker node to the traefik container so it can see our other containers
We mount traefik.toml and acme.json, the file used to store our SSL configurations.
traefik.enable is set to false because I don’t need to use it outside my network.

Then we have to define some more settings in traefik.toml

defaultEntryPoints = ["http", "https"]
logLevel = "DEBUG"

[entryPoints]
  [entryPoints.dashboard]
    address = ":8081"  # I changed from 8080 because SAB uses it
  [entryPoints.http]
    address = ":80"
      [entryPoints.http.redirect]
        entryPoint = "https"
  [entryPoints.https]
    address = ":443"
      [entryPoints.https.tls]

[api]
entrypoint="dashboard"

[acme]
   email = "me@mydomain.com"
   storage = "acme.json"   
   onHostRule = true
   entryPoint = "https"

   [acme.dnsChallenge]
      provider = "digitalocean"
      delayBeforeCheck = 0   

   [[acme.domains]]
      main = "*.mydomain.com"
     
[docker]
domain = "mydomain.com"
watch = true
network = "traefik"

The entrypoints section forces HTTPS and enables the dashboard on locahlhost:8081.
The acme section defines the settings for Let’sEncrypt, which may need to be tweaked for your setup.

The above file must be created:

touch acme.json && chmod 600 acme.json

To test out what we have so far, go to the root of your docker directory and run docker-compose up. Traefik was the hardest for me to configure, so read the output carefully for errors. If everything went OK, we should see our dashboard at localhost:8081.

Step 3: Docker Configurations

The other applications in this setup are mostly controlled through web interfaces. That means that after the container is up and running, you can change the settings via your web browser just how you like them.

I will go over the basic steps to get things running, but after that it’s up to you!

Here is the full docker compose file for my current setup

version: '3'
services:      
   reverse-proxy:
      image: traefik:latest
      restart: always
      container_name: traefik
      ports: 
         - 80:80
         - 443:443
         - 8081:8081
      expose:
         - 8080
      networks:
         - traefik
      environment:
         - DO_AUTH_TOKEN=
      volumes:
         - /var/run/docker.sock:/var/run/docker.sock
         - ./Traefik/traefik.toml:/traefik.toml
         - ./Traefik/acme.json:/acme.json
      labels: 
         - "traefik.backend=traefik"
         - "traefik.docker.network=traefik"
         - "traefik.enable=false"
         - "traefik.frontend.rule=Host:Host:monitor.gideonwolfe.com"
         - "traefik.port=8080"
         - "traefik.frontend.headers.forceSTSHeader=true"
         - "traefik.frontend.headers.STSSeconds=315360000"
         - "traefik.frontend.headers.STSIncludeSubdomains=true"
         - "traefik.frontend.headers.STSPreload=true"

   jellyfin:
      image: linuxserver/jellyfin
      container_name: jellyfin
      environment:
         - PUID=1000
         - PGID=1000
         - TZ=America/Los_Angeles
      volumes:
         - /home/gideon/Data/Programs/Docker/Jellyfin/ProgramData/:/config
         - /home/gideon/Data/Media/TV/:/data/tvshows
         - /home/gideon/Data/Media/Movies/:/data/movies
         - /home/gideon/Data/Music/:/data/music  
      ports:
         - 8096:8096
      labels:
         - "traefik.enable=true"
         - "traefik.port=8096"
         - "traefik.frontend.rule=Host:server.gideonwolfe.com"
         - "traefik.backend=JellyFin"
         - "traefik.frontend.entryPoints=https"
         - "traefik.frontend.headers.forceSTSHeader=true"
         - "traefik.frontend.headers.STSSeconds=315360000"
         - "traefik.frontend.headers.STSIncludeSubdomains=true"
         - "traefik.frontend.headers.STSPreload=true"
         - "traefik.docker.network=traefik"
      networks:
         - traefik
      restart: unless-stopped


   couchpotato:
      image: linuxserver/couchpotato
      container_name: couchpotato
      environment:
         - PUID=1000
         - PGID=1000
         - TZ=America/Los_Angeles
         - UMASK_SET=022
      volumes:
         - /home/gideon/Data/Programs/Docker/CouchPotato/config:/config
         - /home/gideon/Data/Programs/Docker/CouchPotato/Downloads:/downloads
         - /home/gideon/Data/Media/Movies:/movies # Where movies end up
      ports:
         - 5050:5050
      restart: unless-stopped
      labels:
         - "traefik.enable=false"

   sabnzbd:
      image: linuxserver/sabnzbd
      container_name: sabnzbd
      environment:
         - PUID=1000
         - PGID=1000
         - TZ=America/Los_Angeles
      volumes:
         - /home/gideon/Data/Programs/Docker/Sabnzbd/config:/config
         - /home/gideon/Data/Media/:/downloads
      ports:
         - 8080:8080
         - 9090:9090
      restart: unless-stopped
      labels:
         - "traefik.enable=false"

   sonarr:
      image: linuxserver/sonarr
      container_name: sonarr
      environment:
         - PUID=1000
         - PGID=1000
         - TZ=America/Los_Angeles
         - UMASK_SET=022 #optional
      volumes:
         - /home/gideon/Data/Programs/Docker/Sonarr/config:/config
         - /home/gideon/Data/Media/TV:/tv # Final folder where shows end up
         - /home/gideon/Data/Media/:/downloads # Same as SAB downloads
      ports:
         - 8989:8989
      restart: unless-stopped
      labels:
         - "traefik.enable=false"
networks:
  traefik:
    external: true   
  internal:

Each service level represents a containerized docker application.
The image level tells docker where to pull the image from.
The environment level allows you to pass variables into the container.
The volumes level mounts directories from the host into docker containers. This allows for persistant settings and access to media locations.
ports defines the port to access the services
labels allow us to define additional settings, such as disabling SSL certs with traefik.

I would recommend creating a seperate docker-compose.yaml for each sevice, and testing them ony by one before combining them into a master compose file.

Step 4: Web configuration

Jellyfin: localhost:8096
- As long as your desired media is mounted in docker-compose.yaml, You can use the dashboard page to set up libraries with these folders. Piece of cake.
SABnzbd: localhost:8080
- You need to connect SAB with a usenet provider using your account credentials and API key sent by your provider. This is done in the “Servers” settings tab.
- Set up the Categories tab to sort downloads into appropriate folders. For example, tv goes to ./TV, movies to ./Movies. These puts them in the folders Jellyfin looks in.
Sonarr: localhost:8989
- Allow Sonarr to rename episodes under ‘Media Management’
- Under the “Indexers” tab, enter the url and API key from any indexers you have accounts for. Sign up for as many as possible for maximum success finding media.
- Under “Download Client” tab, add SAB. You will need the local IP address of your SAB server as well as the API key found in the “General” section of the SAB settings.
- Optionally enable metadata fetching
CouchPotato: localhost:5050
- Add your indexers in the “Searchers” tab
- Add SAB in the “Downloaders” tab
- Allow Movie Renaming

After all this is set up, we should be able to test it with some media.

Try finding a single TV episode through Sonarr using the magnifying glass.
This should be sent to SABnzbd to download
It should be in your media directory TV/Series/Season #/Episode
Refresh Jellyfin and view the media

Once this pipeline is working correctly, add all the shows you want, and set up monitors so new episodes are automatically downloaded.

For CouchPotato, try adding a movie and seeing if the same behavior happens. Are your movies showing up in Jellyfin?

Conclusion

In this guide, I detailed the steps for setting up an end to end automated pipeline for downloading and streaming your favorite media. Although the initial setup can be tricky at times, the portability and convenience offered by docker makes this setup awesome!

Containers and services can easily be added to the configuration, such as Syncthing or Nextcloud for file backups and sharing.

Automating Linux Desktop Theming With Chameleon

Tue, 30 Jul 2019 00:00:00 +0000

In my previous posts, I have mentioned my use of tools like pywal to automatically generate colorschemes for my terminal based programs.

Since then, I have developed a couple scripts that extend these colorschemes to programs I use regularly.

At that time, my solution to dynamic configuration generartion was to replace the program executable with my own executable, which would generate a configuration on the fly and then launch the intended program.

However, after gathering enough wal plugins, I realized I no longer had a one command solution to theme my system. Wal worked as intended, but I would manually generate GTK themes with oomox, and run commands to generate new skins for programs like Steam and Spotify.

I finally caved and decided it was time for a script to handle all this tedium. Chameleon is that script.

Chameleon works by putting the desired desktop background through wal, and then using the palette generated by wal, creates themes and colorschemes for all supported programs detected on the users system.

For example, if the user has installed Zathura-Pywal, A new zathura configuration file will be generated for this colorscheme. This means the user can run the zathura command to launch the program, instead of the zth command I had previously implemented to hook into the colorschemes.

While not every plugin works seamlessly to theme their intended programs, (see issues I have opened in various repos) My script detects if you are trying to use them (with some configuration required) and runs the appropriate commands.

For a full and up to date list of supported programs, see the github repo. In the mean time, feel free to submit a feature request (or even better, a pull request 😄)

Here is some sample output of the script running

#######################
# Running Wal On Image#
#######################
[I] image: Using image acastro_190117_dekstop_wallpaper.0.jpg.
[I] theme: Set theme to _home_gideon_Wallpapers_acastro_190117_dekstop_wallpaper_0_jpg_dark_None_None_1.1.0.json.
[I] colors: Found cached colorscheme.
[I] wallpaper: Set the new wallpaper.
[I] sequences: Set terminal colors.

[I] export: Exported all files.
[I] export: Exported all user files.
[I] reload: Reloaded environment.

##########################
# Updating Zathura Color #
##########################
Zathura Theme Set

##########################
# Updating Gnuplot Color #
##########################
Gnuplot Theme Set

##########################
# Updating GTK theme     #
##########################

Building theme at /home/gideon/.themes/oomox-xresources-dark

rm -rf "gtk-3.0/dist"
rm -f "gtk-3.0/gtk.gresource"
rm -rf "gtk-3.20/dist"
rm -f "gtk-3.20/gtk.gresource"
rm -rf "/home/gideon/.themes/oomox-xresources-dark/dist"
mkdir -p gtk-3.0/dist
sassc -I "gtk-3.0/scss" "gtk-3.0/scss/gtk.scss" "gtk-3.0/dist/gtk.css"
sassc -I "gtk-3.0/scss" "gtk-3.0/scss/gtk-dark.scss" "gtk-3.0/dist/gtk-dark.css"
glib-compile-resources --sourcedir="gtk-3.0" "gtk-3.0/gtk.gresource.xml"
mkdir -p gtk-3.20/dist
sassc -I "gtk-3.20/scss" "gtk-3.20/scss/gtk.scss" "gtk-3.20/dist/gtk.css"
sassc -I "gtk-3.20/scss" "gtk-3.20/scss/gtk-dark.scss" "gtk-3.20/dist/gtk-dark.css"
glib-compile-resources --sourcedir="gtk-3.20" "gtk-3.20/gtk.gresource.xml"
mkdir -p cinnamon
sassc -I "cinnamon/scss" "cinnamon/scss/cinnamon.scss" "cinnamon/cinnamon.css"

#######################
# Updating Icon theme #
#######################
== Template was copied to /home/gideon/.icons/oomox-xresources-dark-flat
== Theme was generated
Icon Theme Generated

##########################
# Updating Telegram Skin #
##########################
:: Palette generated succesfully.

#######################
# Updating Steam Skin #
#######################
Wal Steam cache found
Wal Steam config found
Wal Steam skin found
Reading colors
Patching new colors
No file to remove
Wal colors are now patched and ready to go
If this is your first run you may have to
enable Metro Wal Mod skin in steam then
simply restart steam!

###########################
# Updating Keyboard Color #
###########################
Keyboard Color Set

##########################
# Updating Spotify Color #
##########################
about.spa
artist.spa
browse.spa
buddy-list.spa
chart.spa
collection-album.spa
collection-artist.spa
collection-songs.spa
collection.spa
concert.spa
concerts.spa
error.spa
findfriends.spa
full-screen-modal.spa
genre.spa
glue-resources.spa
hub.spa
licenses.spa
login.spa
lyrics.spa
playlist-folder.spa
playlist.spa
profile.spa
queue.spa
search.spa
settings.spa
show.spa
station.spa
stations.spa
zlink.spa
[sudo] password for gideon: 
Spotify Theme Set

##################################
# Updating IntelliJ Color Scheme #
##################################
IntelliJ Theme Set

Deploying a static Hugo site with NGINX

Mon, 01 Jul 2019 00:00:00 +0000

Many people (such as myself) try to avoid web development as much as humanly possible. I find myself less frustrated staring down 1000 lines of assembly than I do trying to center some text on a website.

In the past, I have used Squarespace and the like to build nice sites for my non tech related projects. This may be an easy “it just works” solution, but for my personal website I wanted to do things myself.

That being said, I don’t have time to spend fiddling with a LAMP stack and making a complicated site when all I really needed was a platform to post and share projects. That’s where Hugo comes in. With Hugo, you can write simple markdown files and dynamically generate a static html based site, like this one.

Prerequisites:

Linux OS: I recommend a dedicated system just for hosting if this is going to production. Either a system you aren’t using or a VPS on a provider such as Digital Ocean or Linode.
A domain: You can’t really expect to run a website without one. There are plenty of good registrars out there, but I personally like NameCheap because you can host email for your domain without hosting the site itself.

Installing Hugo

Visit the Hugo quickstart guide for distro specific information. I was able to install hugo on my Ubuntu Server system with

sudo apt install hugo

and on Arch with

sudo pacman -Syu hugo

If Hugo is not in your distro’s repositories, you can always download it with the tarball.

Making a quick site

Find a directory where you want to store your site, and run

hugo new site <site name>

where site name is the name of the folder your site will be created in.

Adding a theme

Browse this extensive list of Hugo themes until you find one you like. Find it on github, and simply clone it into your themes directory. Here, I chose the m10c theme.

cd mysite
git clone https://github.com/vaga/hugo-theme-m10c themes/m10c

Then add the following line to your config.toml file in your website’s root folder

theme = "m10c"

Adding a post

To create a new post, simply run

hugo new posts/my-first-post.md

This will create a post called my-first-post in the content/posts folder. Edit this as you please.

Finally, we need to run the site. Change directory into the projects root and run

hugo server -t m10c -D

Finally, visit localhost:1313 to see the site.

Configuring NGINX

Firstly, make sure you have installed NGINX, which is available on pretty much every default repository.

Now we are going to create and edit a configuration file for our site.

cd /etc/nginx/sites-available/
cp default mysite
vim mysite

There are a lot of comments, but your server configuration should look something like this:

server {
       listen 80;
       listen [::]:80;

       server_name mysite.com www.mysite.com;

       root /home/username/mysite/public/; #Absolute path to where your hugo site is
       index index.html; # Hugo generates HTML

       location / {
               try_files $uri $uri/ =404;
               add_header 'Access-Control-Allow-Origin' '*'; # Allow access to resources (for www and non www)
       }
}

To enable this site, we need to create a symlink from your site into sites-enabled. Use absolute filepaths to avoid symlink confusion.

sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/mysite

We can run and enable NGINX with the following commands:

sudo systemctl start nginx
sudo systemctl enable nginx

Quick Checks

Did you set your registrars nameservers to point at the server you’re using to host?
Did you create A records with your hosting provider to direct requests to the correct server? (add a separate A record for the www subdomain)

Adding SSL

SSL is a must these days, and it’s never been easier to implement. I went with Certbot, because it’s easy to use and hey, who doesn’t love the EFF?

Visit the certbot site to get customized instructions based on your setup, but here is the process for NGINX running on Ubuntu 18.04:

First we add the certbot PPA:

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository universe
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update

Then we install certbot itself

sudo apt-get install certbot python-certbot-nginx

And finally we run the setup script:

sudo certbot --nginx

and go through the steps.

Conclusion

Assuming you have set things up correctly, you should have an easy to use site built by hugo, but running with the powerful NGINX suite.

To update the site, simply write more markdown files and run hugo from the site root to rebuild the site.

Using Lambda Functions in Python λ

Thu, 20 Jun 2019 00:00:00 +0000

What are lambda functions?

In Python, lambda functions are a syntax that allows the use of anonymous functions in ones code.

Lambda functions take any number of inputs, perform some computation on these inputs, and return the result of this computation.

To best illustrate the uses of lambda functions, let’s first examine a normal function which takes in two numbers, x and y, multiplies them, and returns the result.

# Function that multiplies two numbers
def multiplyTwoNumbers(x, y):
    return x*y

Now let’s see how this looks as a lambda function:

>> g = lambda x, y: x*y
>> print(g(6, 3))
>> 18

Lambda functions contain an implicit return call
Lambda functions contain only one expression

Why use lambda functions?

Lambda functions work best when a computation needs to be done but does not necessitate a separate function.

In our last example we only saved one line of code. Let’s showcase the true strengths of anonymous functions.

For our example, say we need to find the value of x % 10 for every element x in a list. Using traditional code, it may look something like this.

# This code preforms modulus 10 on every element in a list

def getMod(x):
    return(x%10)

myList = [44, 12, 98, 34, 17]                              
newList = []

for i in myList:
    newList[i] = getMod(i)

print(newList)

This gives us [4, 2, 8, 4, 7]. Let’s clean this up with a lambda expression.

# This code preforms modulus 10 on every element in a list

myList = [44, 12, 98, 34, 17]
newList = list(map(lambda x: x%10, myList))
print(newList)

The result is [4, 2, 8, 4, 7], just as expected.

The map function takes two inputs: A function, and a list of arguments to pass to it.
map returns a map object, so we cast it to a list in order to read it.
Lambda functions can declared inline and thrown away. No declaration or variable assignment required.

Lambda functions and `filter()`

Here is an example of the filter() function in python. filter() takes in two arguments:

A function object, (in this case a lambda function) which returns a boolean
Iterable list of aguments

filter() works similarly to map() except only the values for which our lambda function returns True are added to the list

Let’s say we wanted to filter out numbers that are not a multiple of three.

myList = [44, 12, 98, 34, 17]
newList =list(filter(lambda x: x%3 == 0, myList))
print(newList)

This returns [12], which makes sense because it is the only multiple of three in our list. This is certainly better than a for loop with a traditionally defined function!

Lambda functions and `reduce()`

The reduce function is used when one wishes to perform a rolling computation on an iterable. For example, let’s say you wanted to find the product of numbers in a list. Using traditional python, it might look something like this:

# code to find the product of all elements in a list

myList = [2, 3, 4, 5, 6]                                            
total = 1
for i in myList:
	total = total*i
print(total)

The output here is 720.

The same process using a lambda expression and reduce() looks something like

from functools import reduce  
myList = [2, 3, 4, 5, 6]                                            
print(reduce(lambda x,y: x*y , myList))

Again, the output is 720.

Basically, the reduce function is performing the following operation:

$$ ((((2 * 3) * 4 ) * 5) * 6) $$

Pretty nifty, huh? 😄

Conclusion

Lambda functions. Used by CS students trying to flex on their classmates? Indeed. Valuable and useful tool to optimize quick calculations? Hell yes.

We learned how to leverage lambda expressions with the map(), reduce(), and filter() functions to make quick data processing a breeze. These are some pretty trivial examples but the concept can be extrapolated to fit real world applications.

Theming gnuplot

Tue, 18 Jun 2019 00:00:00 +0000

Overview

If you have read any of my other posts, you know how important it is to have a cohesive looking system. I recently developed Zathura-Pywal, a tool that recolors PDFs in zathura based on your colorscheme.

In that same vein, I have whipped up a set of scripts to do the same thing for gnuplot. If you haven’t heard of gnuplot, check out my short tutorial here.

How to install

git clone https://github.com/GideonWolfe/Gnuplot-Pywal

cd Gnuplot-Pywal

sudo ./install.sh

Now here’s the thing to remember. Instead of running gnuplot, you now have to run gplot to get the freshly themed config. You can mess with the aliases however you like, but the config generation script must be called.

$$ \frac{sin(x^2 + y^2)}{x^2 + y^2} $$

How it works

These are pretty short scripts, so I’ll be brief. When the command gplot is run, it immediately calls another program gengnuplotconfig. Aside from some optional tinkering, you should never have to interact with this program.

When gengnuplotconfig is called, it imports colors from a cache created by pywal. It then pipes out a gnuplot configuration, using color values from your colorscheme. These can be tweaked to your preference.

in gplot, the output of gengnuplotconfig is being redirected into the file ~/.gnuplot, which is the default configuration file used by gnuplot.

Finally with this new configuration in place, gnuplot is launched. The delay is imperceptible and the colorscheme usually comes out looking pretty good.

$$ (sin(x) * cos(y)) - tan(y^2)$$

Conclusion

Could this have been done better? Probably 😄. But that’s what happens when you do an entire project between one and three in the morning.

Do not hesitate to contact me with any ideas for projects you would like to see!

$$ cos(u) * (cos(v)+3), sin(u) * (cos(v)+3), sin(v) $$

Introduction to gnuplot

Sat, 15 Jun 2019 00:00:00 +0000

Have you ever been frustrated in math classes due to lack of visual examples of abstract topics? Have you ever wanted to quickly visualize a dataset without going through cumbersome software like Excel?

If so, you should know about gnuplot.

Gnuplot is a command line based graphing and plotting utility. In this post, I’ll outline some practical applications of gnuplot as well as some tips.

Install gnuplot

Most distos should have gnuplot in their repositories. I installed gnuplot on Arch linux by running sudo pacman -Syu gnuplot

To start using gnuplot, fire up your favorite terminal and run gnuplot. You should be faced with a prompt like so:

	G N U P L O T
	Version 5.2 patchlevel 7    last modified 2019-05-29 

	Copyright (C) 1986-1993, 1998, 2004, 2007-2018
	Thomas Williams, Colin Kelley and many others

	gnuplot home:     http://www.gnuplot.info
	faq, bugs, etc:   type "help FAQ"
	immediate help:   type "help"  (plot window: hit 'h')

Terminal type is now 'qt'
gnuplot>

2D Plotting

Now we are ready to start graphing. Let’s start with the simple function.

To plot this in gnuplot, simply enter the following into your prompt:

plot x**3

\mathring{g}

This looks pretty much like we would expect it to. If we want to plot multiple functions to compare, we can simply add it to the plot command.

plot sin(x), cos(x)

$$f(x) = sin(x)$$

$$g(x) = cos(x)$$

3D plotting

Plotting functions in two dimensions is great, but what about three dimensional functions? Let’s see what

$$ f(x) = sin(x)*cos(y) $$

looks like in 3D. Enter the following into gnuplot:

set isosamples 50
set hidden3d
splot [0:2*pi] [0:2*pi] sin(x)*cos(y)

The set hidden3d command disables the wireframe, making it easier to see the contours of the 3D objects. Try it both ways with the unset hidden3d command, use whichever you prefer.
Setting our isosamples to 50 gives us a much smoother plot.
In gnuplot, the splotis identical to the plot command, except it will make a 3-dimensional plot.
The [0:2*pi] [0:2*pi] section specifies a domain for each of these functions, which is zero to 2π (the period of the cosine and sine functions)

After running these commands, we get

$$ f(x) = sin(x)*cos(y) $$

I would recommend checking out the gnuplot demos page, which has tons of functions and scripts to play around with. There are many more features than I could ever hope to cover here, so this is a great place to explore.

Plotting data

Say that we ran some experiments and we wanted to quickly plot the results. To keep things simple, let’s start out with a 2D graph that might be commonplace on homework assignments and research projects. First, we need our data file. Data files in gnuplot use the .d extension by default. In the case of 2 dimensions, there will be two space-separated columns for our points, with two line breaks signifying a new data block.

For this example, I using a classic classroom experiment of tracking the growth of plants over time. The left column represents the day, and the right hand is the height in inches.

# Flower number one
1 1.3
2 1.8
3 2.1
4 2.4
5 2.4
6 2.9
7 3.2


# Flower number two
1 1.0
2 1.1
3 1.6
4 2.1
5 2.6
6 3.0
7 3.3

To plot this data, we can simply run

plot 'flowers.d'

This looks OK, but it doesn’t really tell the whole story and it certainly isn’t up to research (or even science fair) standards. Let’s spruce this plot up a bit!

set title font "Helvetica,14"
set title 'Flower Growth Over Seven Days'
set xlabel 'Time (Days Elapsed)'
set ylabel "Height (Inches)"
plot 'flowers.d' index 0 w lp lw 3 title 'Flower 1', 'flowers.d' index 1 w lp lw 3 title 'Flower 2'

index chooses the block from the dataset to plot (each block is separated by two blank lines)
w lp: short for with linepoints, which styles the lines
lw: short for line width
title sets the key label for the corresponding data block

Looks much better!

Conclusion

We learned how to use gnuplot to quickly create a graph out of experimental data, as well as plot equations in two and three dimensions. I have barely scratched the surface of this program, and I will share my knowledge with you as I continue to learn. In the future, I want to explore more advanced gnuplot features such as parametric equations and data fitting. I also hope to develop an automatic gnuplot theming script, that would act like my Zathura-Pywal utility and automatically color gnuplot based on the system colors. Keep an eye out for updates!

Workflow Optimization Part 4: Theming Your System

Mon, 03 Jun 2019 00:00:00 +0000

Why care about themes?

Why spend so much time theming your system when you could actually be getting work done? I get this question a lot, and I’ll attempt to explain my reasoning.

When you look at a typical system doing many tasks, you can see a mosaic of different programs floating around and cluttering your desktop. Each of these programs has their own UI, colorscheme, and style. Just take a look at all of the right click context menus in stock Windows 10. If you can’t unify design within a single OS, how do you expect to get an entire system looking cohesive?

Windows 10 is a particularly egregious example of bad UI/UX consistency in software, and I’ll leave it at that before I launch into an involuntary anti-Windows rant.

My point is, that having a system that looks like the result of a long list of poor design choices makes the system feel worse to use. Sure, it may not effect raw speed, but it makes the overall system feel unpolished. Imagine I had a Toyota, with the bumpers of a Corolla, the dashboard of a Prius, the bed of a Tacoma, and the interior of a Rav 4. Sure it may run fine, but it’s going to be hell to work on. Even though they are part of the same “system”, none of the parts fit together quite right. Nothing feels natural, because you can’t rely on anything working the same way twice.

As Linux users, how can we do better?

Good news on that front. I can’t think of a single mainstream distro that does a worse job at UI/UX than Windows.

On Linux, most of the UI is handled by your desktop environment, or DE. If you’re looking for a traditional GUI experience, you can turn to the likes of Cinnamon, KDE, Unity, GNOME, Budgie, and countless others. Most of these DEs are OS agnostic, meaning they can be installed on top of any operating system. This gives you the ability to experiment with different DEs without changing your whole OS, and even run multiple DEs simultaneously.

Theming My System

When I initially started theming my system, I envisioned something reminiscent of a sci-fi computer system.

Every program looks like it was a handmade for that particular system, and feels like a cohesive state of the art machine. However, a setup like the one above isn’t as versatile as I need my systems to be. If you want to try one out for kicks, I recommend a eDEX-UI.

As you may have guessed, I do not prefer a traditional GUI experience. I use i3, which is a window manager as opposed to a desktop manager. Since I don’t have a desktop with icons, or a main settings menu, I simply manage my windows using i3, no DE required. To get some of the nice features that come with DEs (such as a statusbar and tray), I opt to use an array of helper programs to add this functionality. Since I mainly use terminals for my day to day work, theming programs in the terminal is my primary concern.

The cornerstone of my theme is a program called pywal. pywal takes an image as input, and generates a colorscheme based on the colors in the image. Additionally, pywal has an assortment of built in colorschemes, no image required.

pywal can also be used with different image backends to alter the final colorscheme, giving you a lot of ways to get the colors you want. In the future I want to look into wpgtk, which is a theming framework that utilizes pywal. This can be used to generate themes for GUI apps too!

When pywal generates colors, it stores them in a few places.

Firstly, there is your cache, where wal stores the current colors in many formats. Many programs can be pointed directly at these files to set the theme.

$ tree .cache/wal/

├── colors
├── colors.css
├── colors.hs
├── colors.json
├── colors-kitty.conf
├── colors-konsole.colorscheme
├── colors-oomox
├── colors-putty.reg
├── colors.reg
├── colors-rofi-dark.rasi
├── colors-rofi-light.rasi
├── colors.scss
├── colors.sh
├── colors-speedcrunch.json
├── colors-sway
├── colors-tty.sh
├── colors-wal-dmenu.h
├── colors-wal-dwm.h
├── colors-wal-st.h
├── colors-wal-tabbed.h
├── colors-wal.vim
├── colors-waybar.css
├── colors.xresources
├── colors.Xresources
├── colors.yml
├── firefox.css

Many programs take their colors from .Xresources. I simply add the section below to my .Xresources to make the colorscheme available to programs.

*.color0: {color0}
*color0:  {color0}
*.color1: {color1}
*color1:  {color1}
*.color2: {color2}
*color2:  {color2}
*.color3: {color3}
*color3:  {color3}
*.color4: {color4}
*color4:  {color4}
*.color5: {color5}
*color5:  {color5}
*.color6: {color6}
*color6:  {color6}
*.color7: {color7}
*color7:  {color7}
*.color8: {color8}
*color8:  {color8}
*.color9: {color9}
*color9:  {color9}
*.color10: {color10}
*color10:  {color10}
*.color11: {color11}
*color11:  {color11}
*.color12: {color12}
*color12:  {color12}
*.color13: {color13}
*color13:  {color13}
*.color14: {color14}
*color14:  {color14}
*.color15: {color15}
*color15: {color15}

This should cover most TUI programs, because they generally just inherit colors from your terminal. Sometimes there are programs that don’t run in the terminal, or don’t automatically take colors from .Xresources. In the case of the former, your best bet is to look for a wal theme or skin. If the program in question is easily themable, you can probably plug wal colors into it. You can check this page to see all of the programs wal works with.

I even wrote my own script for the Zathura document viewer that dynamically pulls colors from the current wal colorscheme. Solutions like this are pretty easy to implement with a bit of know how.

Finally, the browser. I am using FireFox, which allows you to edit the CSS of how the browser actually looks. Take a look at my repo for the files. I wrote my own startpage, which is dynamically themed using the current colorscheme. The same scheme is applied to the core UI, but you need to restart the browser for this to take effect.

This about sums up the themes I use in my system. Take a look at my helper programs post for info on how I configure specific programs.

Programming in Neovim

Tue, 28 May 2019 00:00:00 +0000

Why Vim?

When people in the development community hear the word vim, they generally think one of three things:

That nightmare program that I can never seem to exit, much less learn. mainly used by elitists.
A powerful and customizable text editor that is worth the steep learning curve.
The fact you are not using Emacs sickens me to my very core

Today we are going to look at vim, and see why when utilized correctly, it can be an essential tool in your toolbox.

Don’t get me wrong, emacs is cool, and I have colleagues who use it and love it. Emacs can certainly do many things that vim cannot, which is both a blessing and a curse. If you recall my first post, I refer to the Unix philosophy as a guiding principle to my workflow. Unfortunately, when used to it’s fullest capacity, emacs goes against that philosophy.

Emacs is a great operating system… All it needs is a good text editor.

This is a quip I hear often around developers, many who use emacs themselves. Like vim, emacs is quite extensible, taking advantage of a custom scripting language called Emacs-lisp, or elisp. The combination of elisp with a GUI emacs client has lead to a large ecosystem of user created applications and extensions that work right within emacs. Inherently, there is nothing wrong with this. Extensibility like this is part of makes vim great. However, at a certain point I have to question the scope of how a text editor should be used in one’s workflow,

On Milkypostman’s Emacs Lisp Package Archive or MELPA, one can browse over 4,000 emacs packages, from calendars to fully fledged multimedia systems. If you really set your mind to it, you could build an emacs that contains every tool you need in your workflow, and work without every having to leave emacs. As you can see, this is quite contrary to the unix philosophy of every program being very good at one thing.

I use separate programs to handle separate tasks, and I wouldn’t really use the apps available to emacs. Without these features, there is nothing that sets emacs apart over vim, besides more difficult keybinds. At that point they are both text editors and do that job well.

To answer the original question I posed in this section; I use vim because it is efficient, and it is universal. No matter the job, whether that be spinning up virtual machines or SSHing into remote servers to work, I can count on a vim-like program being installed. You may not have all your customizations, but that is quickly fixed by copying over your .vimrc file.

If you haven’t used vim before, I reccommend getting used to the commands. If you are running classic vim, you can run vimtutor in your terminal, and you can work through these examples. If not, there are fun ways to get acquainted with the system.

Learning the language of vim commands allows you to surgically edit files at lightning speed, all without leaving the comfort of your terminal. In this post, I’ll go over some of the customizations that really take vim to the next level.

Customizing Vim

I use a fork of vim called neovim. Neovim adds a lot of much needed features to vim, and has even better support for extensions and customization. However, most of these tips should translate to legacy vim as well.

To see my full and up to date configuration, take a look at my init.vim file.

Changing some defaults

" Statusline Config: Show filename and leave room for extensions to show output
set statusline+=%*
set statusline+=%F
set cmdheight=2

" Tab Settings: 1 tab = 4 spaces
set expandtab           
set tabstop=4           
set shiftwidth=4

" Misc
set showmatch "shows matching braces/parens
set formatoptions+=o
set number
set nocompatible
filetype off
set background=dark
colorscheme wal
set clipboard=unnamed 

" Relative line numbers
augroup numbertoggle
  autocmd!
  autocmd BufEnter,FocusGained,InsertLeave * set relativenumber
  autocmd BufLeave,FocusLost,InsertEnter   * set norelativenumber
augroup END

" Show next 3 lines while scrolling.
if !&scrolloff
    set scrolloff=3       
endif

" Show next 5 columns while side-scrolling.
if !&sidescrolloff
    set sidescrolloff=5   
endif

Managing plugins

For a plugin manager, I chose Vundle. Most plugin managers for vim are similar in use, and instructions for popular ones can be found on the pages for most vim plugins.

To enable vundle, we add the following to our vim configuration:

set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
Plugin 'VundleVim/Vundle.vim'



call vundle#end()

The third line of this excerpt declares Vundle, which itself is a plugin. However, we can follow this format and add any plugin using it’s github repo (username/repo). Here is a list of some of the must have plugins I use.

Vim snippets / utilsnips: Together, these plugins provide an extensible engine for quick code snippets. Quickly write boilerplate code so you can focus on the tough stuff.
deoplete: Great autocompletion engine
ALE: All purpose linter. Highlights errors and warnings in most programming languages.

After you add the plugins you want to your config, open any vim buffer and run :PluginInstall, which will attempt to… Well, install your plugins.

Look in my init.vim to see all of the specific plugins I use and how I configure them.

This is only the tip of the iceberg. I am constantly discovering new plugins to enhance vim, all without leaving it’s scope as a text editor.

Workflow Optimization Part 3: Sidekick Programs for i3

Tue, 02 Apr 2019 00:00:00 +0000

i3 comes perfectly usable out of the box, but If there’s one thing you should have learned by now, I like customizing everything. By default, i3 comes with basic tools to facilitate a tiling WM workflow. That being said, there are some programs that are must haves in my opinion, that don’t come with i3.

Polybar

i3 comes with i3bar installed and enabled by defult. i3bar is a perfectly fine status bar, and if you’re interested in using it, I recommend checking out Luke Smith’s overview of his i3bar setup. However, I prefer polybar, mostly for aesthetic reasons.

Configuring polybar

Configuring polybar is pretty straightforward. By default, there a config file located at ~/.config/polybar/config. The name of the default bar is example. Simply run polybar example to start the bar. I recomend starting the bar from a terminal so you can see any errors polybar may throw at you. These errors will mostly warn you about misconfigurations in your config file, missing fonts, and the like.

To get started with your custom config, I recommend looking at reddit, and the Arco Linux dotfiles for interesting modules. The creator of polybar also has some slick looking themes on his repo. Basically, you can copy the modules from any polybar config, and edit the specifics to suite your needs.

For example, here is my configuration for the download speed module, seen in the top right of my statusbar.

[module/networkspeeddown]
;https://github.com/jaagr/polybar/wiki/Module:-network
type = internal/network
interface = wlp3s0
;interface = enp14s0
;interface = enp0s31f6
;interface = enp4s0
label-connected = "%downspeed:7%"
format-connected = <label-connected>
format-connected-prefix = ""
format-connected-prefix-foreground = ${colors.foreground-alt}
format-connected-foreground = ${colors.foreground}
format-connected-background = ${colors.background}
;format-connected-underline = #62FF00

This module is a stock polybar module, but it won’t work if you don’t uncomment the correct interface for your system! I use the following lines to import my colorscheme to polybar.

[colors]           
background = ${xrdb:color0:#222}    
background-alt = ${xrdb:color3:#222}
foreground = ${xrdb:color7:#222}    
foreground-alt = ${xrdb:color2:#555}
highlight = ${xrdb:color2:#f00}
urgent = #f00

I have this section to enable a system tray, for programs that use it. Nextcloud, Steam, etc.

tray-detached = false
tray-position = right
tray-offset-x = 0
tray-offset-y = 0
tray-maxsize = 16
tray-padding = 0
tray-scale = 1
tray-background = ${colors.background}

Keep running polybar from the terminal until it looks the way you want, without any fatal errors. Then, I add the following line to my i3 config: exec_always pkill -9 polybar ; polybar main. This kills and restarts polybar everytime you refresh i3. This is an easy way to quickly reload your polybar config to test things, or if it glitches for some reason.

Like i3bar, polybar supports many custom modules and scripts. For example, my cmus module runs a bash command every second to check the status of my music player, cmus. I added some logic for click behavior, and now it is a fully functioning music widget.

[module/player-cmus]
type = custom/script
exec = ~/.config/polybar/player-cmus.sh
interval = 1
click-left = cmus-remote -n
click-right = cmus-remote -r
click-middle = cmus-remote -u

It really is that simple. The script mentioned above can be found in my dotfiles.

Rofi

In the default i3 config, there is a built in hotkey to launch dmenu, a program launcher. dmenu is great. I personally prefer rofi, because I’ve been usig it for a long time now. Both rofi and dmenu serve basically the same purpose. They are general menu applications. This means they can be used for the default purpose, which is to choose and launch programs, but also so much more. You can integrate rofi or dmenu into your statusbar, and create interactive menus for any possible purpose.

Configuring rofi

Admittedly, I have been using rofi exclusively as an application launcher. Since I rarely touch the mouse in my workflow, having GUI click-through menus seems counterintuitive for my purpose. The configuration of rofi is pretty straightforward. I literally have a single line in the file ~/.config/roficonfig.

rofi.theme: ~/.cache/wal/colors-rofi-dark.rasi

This will make rofi check the colorscheme cached by wal the last time I generated a theme. More on that later. Interestingly enough, most of the real configuration is done in ~/.Xresources. The below section gives rofi a pretty slick look in my opinion. Simply edit these settings to your preferences and run sudo xrdb -merge .Xresources to apply the changes. Arch users may have to download the xorg-xrdb package in order to run the last command.

! rofi setup      
rofi.color-enabled:         true
rofi.separator-style:       solid
rofi.font:                  curieMedium 20
rofi.bw:                    2
rofi.columns:               1
rofi.yoffset:               0
rofi.fake-transparency:     false
rofi.hide-scrollbar:        true
!rofi.opacity:               90
rofi.location:              0
rofi.width:                 50
rofi.lines:                 10
rofi.fixed-num-lines:       true

With rofi, there is no need for a taskbar, start menu, or any other cluttered GUI elements. I then edited the keybind in my i3 config to lauch rofi instead of dmenu.

Other programs:

ranger: Terminal based file manager. Great for easy browsing and previewing of files.

xcwd: I add bindsym $mod+Shift+Return exec urxvt -cd "``xcwd``" to my i3 config, and now I have the “Open terminal in current directory” functionality.

gotop: Terminal task manager/resource monitor. Functional and cool looking.

Here is a list of awesome command line apps. Pick out the ones that you need and suite your workflow! If You’re running arch, don’t forget to check the AUR for these packages, it makes managing all these programs much easier.

Next time, we take a closer look at theming.

Markdown test page

Wed, 27 Mar 2019 00:00:00 +0000

Name	Age
Bob	27
Alice	23

~~Crossed Out~~

This is a footnote.¹

Definitions

Cat: Fluffy animal everyone likes
Internet: Vector of transmission for pictures of cats

TODO - tasks

[ ] a task list item
[ ] list syntax required
[ ] incomplete
completed

I ❤️ Hugo! 😗

Math

$\require{\mhchem}$

$$\ce{Zn^2+ <=>[+ 2OH-][+ 2H+]
$\underset{\text{zinc hydroxide}}{\ce{Zn(OH)2 v}} $ <=>[+ 2OH-][+ 2H+] $\underset{\text{tetrahydroxozincate(II)}}{\ce{[Zn(OH)4]^2-}}$}$$

the footnote text. ↩︎

Workflow Optimization Part 2: The i3 Window Manager

Wed, 27 Mar 2019 00:00:00 +0000

Why use a tiling window manager?

As I touched on in my introductory post, there are many reasons I prefer tiling window managers over traditional desktop environments. Of these reasons, let me highlight three. Efficiency, flexibility, and usability.

A quick note about i3 and other tiling WMs. While tiling WMs support regular GUI programs, the benefits are most apparent while using the terminal. Most tiling WMs set “open terminal” as the most fundemental keybind. Most traditional desktop environments are configured with GUI settings menus, but i3 has none. i3 settings and the settings of many programs used with i3 are configured with plain text files. I will be talking about i3 in this series, but most of this applies to all tiling window managers. Check out alternatives like xmonad, or dwm.

Efficiency

If tiling window managers excel in one category, it would be efficiency. When we think about how we interact with a traditional desktop environment, we notice that much of our time is spent switching between the mouse and keyboard, or simply rearranging floating windows.

Tiling window managers allow you to do pretty much everything from the keyboard, eliminating the need to constantly be reaching for the mouse. Since all windows are tiled, they can be manipulated with keybinds. In i3, windows can be stacked, tiled, tabbed, and even floated like in a traditional DE.

It may seem ludicrous to spend so much effort on shaving a couple seconds off menial tasks. The reality is when you spend hours a day on the computer doing 1000’s of small tasks, the time saved really starts to add up.

Flexibility

i3 is an extremely flexible window manager. Depending on your installation method, your i3 configuration file is located at either $HOME/.i3/config or $HOME/.config/i3/config. Within the file, you will find various i3 settings as well as comments describing their purpose. These settings allow you to easily understand and configure i3’s core functionality. Applying these changes is as easy as refreshing i3, no recompile required.

Usability

To a new user, i3 may look and feel unusable. Like anything unfamiliar, i3 takes a bit of getting used to. After a couple days of glancing at the i3 reference card, I was pretty much up to my previous work speed. Much like going to a multimonitor setup, once you go to a tiling WM, it is extremely difficult to go back. With every possible action or program just a few keystrokes away, i3 beats digging through endless GUI menus in a traditional DE.

My i3 configuration

My full i3 configuration is available here, but I will highlight a few of the changes I made to make i3 more usable.

Basic keybinds

The first thing I do is change the keybindings for window control to use the Vim keys “hjkl” instead of the i3 default of “jkl;”. This makes sense because many terminal based programs incorporate vim keys, and having a single set of keybinds super convenient!

Gaps

I added the snippet below to my i3 config to define some window gaps. I find having no gaps feels claustraphobic, and you can’t see your wallpaper! You may want to change these values based on your resolution.

for_window [class="^.*"] border pixel 3
new_window pixel 4
gaps inner 17
gaps outer 5

Setting colors from .Xresources

The .Xresources file is a configuration file shared between many applications to define common settings such as colors and fonts. Luckily, i3 gives us an easy way to access these values and use them in our setup.

set_from_resource $bg           i3wm.color3
set_from_resource $bg-alt       i3wm.color0
set_from_resource $fg           i3wm.color15
set_from_resource $fg-alt       i3wm.color2
set_from_resource $hl           i3wm.color4

Now that we have these values defined, let’s apply them to our UI elements.

set_from_resource $bg           i3wm.color3
# class                 border  backgr. text indicator      child_border
client.focused          $fg-alt $bg     $fg  $fg-alt        $hl
client.focused_inactive $bg     $bg-alt $fg  $bg            $bg
client.unfocused        $bg     $bg-alt $fg  $bg            $bg
client.urgent           $bg     $bg-alt $fg  $bg            $bg
client.placeholder      $bg     $bg-alt $fg  $bg            $bg
client.background       $bg

If you look in your .Xresources file (create it if it doesn’t exist), you should see something like this:

*.color0: {color0}
*color0:  {color0}
*.color1: {color1}
*color1:  {color1}
*.color2: {color2}
*color2:  {color2}
*.color3: {color3}
*color3:  {color3}
*.color4: {color4}
*color4:  {color4}
*.color5: {color5}
*color5:  {color5}
*.color6: {color6}
*color6:  {color6}
*.color7: {color7}
*color7:  {color7}
*.color8: {color8}
*color8:  {color8}
*.color9: {color9}
*color9:  {color9}
*.color10: {color10}
*color10:  {color10}
*.color11: {color11}
*color11:  {color11}
*.color12: {color12}
*color12:  {color12}
*.color13: {color13}
*color13:  {color13}
*.color14: {color14}
*color14:  {color14}
*.color15: {color15}
*color15: {color15}

! Black color that will not be affected by bold highlighting.
*.color66: #131811

If not, add it. This defines colors for other programs to use.

Startup Commands

i3 makes it easy to define custom commands that run when i3 restarts (with a logout/login) or refreshes (Mod+Shift+r by default). Here are the important commands I run.

exec-always wal -R tells pywal to reload my last colorscheme every time I refresh i3.

exec xset s off
exec xset -dpms
exec xset s noblank

This setting will disable automatic screen blanking, which I prefer on my machines.

exec_always urxvt -name math -e bash -c 'qalc' 
for_window [instance="math"] floating enable;
for_window [instance="math"] move scratchpad; move position center;
bindsym $mod+m [instance="math"] scratchpad show

I added this calculator scratchpad which gives me acces to qalculate, my favorite calculator. Really handy!

Custom Keybinds

Keybinds in i3 are defined as follows: bindsym $mod+f exec urxvt -e ranger would launch ranger, a terminal file manager, in my terminal urxvt.

As long as your program is callable from the terminal, it can be bound to a keybind.

Misc

bindsym XF86AudioRaiseVolume exec pactl set-sink-volume @DEFAULT_SINK@ +2%; exec pactl set-sink-mute @DEFAULT_SINK@ 0
bindsym XF86AudioLowerVolume exec pactl set-sink-volume @DEFAULT_SINK@ -2%; exec pactl set-sink-mute @DEFAULT_SINK@ 0
bindsym XF86AudioMute exec pactl set-sink-mute @DEFAULT_SINK@ toggle
bindsym XF86AudioPlay exec playerctl play-pause
bindsym XF86AudioNext exec playerctl next
bindsym XF86AudioPrev exec playerctl previous
bindsym XF86MonBrightnessUp exec light -A 10 # increase screen brightness
bindsym XF86MonBrightnessDown exec light -S 10 # decrease screen brightness

These are the settings that allow me to use the built in media keys on my macbook.

And that about wraps it up for this post. Next time I’ll discuss some of the programs that are needed to make i3 as usable as a standard DE.

Workflow Optimization Part 1: The System

Tue, 26 Mar 2019 00:00:00 +0000

I could write dozens of pages on why I choose linux over any other operating system. Let me save us both the time and break it down here.

Customization. If you can dream it, you can make it. From purely aesthetic enhancements to low level system changes, linux gives you choices that simply are not available on other platforms.
Philosophy. Every program does one thing and does it well. Linux comes with free and open source software, and respects your privacy.
Control. When you run Linux, you truly own your system. There is no corporate overlord renting you a license for your OS. Any possible system action and change is only a few commands away.

On my main machine, I run a distrobution called Arch Linux. This distro is a rolling release, meaning the repositories are constantly updated with new packages. In addition to this, it is a lightweight distro, and comes with practically nothing installed. By the time your workflow is complete, it will be a full fledged system.

If you take a look at the above screenshot, you will notice that the screen is divided into multiple black panes. Each of these panes are terminals, that allow me to launch and use pretty much any program on my system. Instead of managing a bunch of shortcuts and launchers, I simply spawn a terminal and tell it what to do.

In addition to this, I am using a tiling window manager called i3-gaps. Tiling window managers allow you to more efficently move and interact with windows. Ever gotten frustrated when you have a million windows open and you’re just dragging them around frantically trying to find the right one? Tiling window managers like i3 let you tile all programs to prevent this. Works great with a terminal workflow, because terminals scale well and work fine at a small size.

In my next post, I will be going into more detail about some of the configuration I’ve done to i3 to improve my workflow. Until then, check out my dotfiles for the configs.

The Importance of Building an Optimized Workflow

Mon, 25 Mar 2019 00:00:00 +0000

There are a few things that set apart true professionals from the rest in their trade. Among these things are a refined workflow and a mastery of one’s tools.

No matter the industry, this philosphy holds true. A great carpenter will know the purpose and location of every tool in their shop. A great photographer will be intimately familiar with the settings and behavior of their camera, as well as a suite of photo editing tools and programs.

A lot of times, we can expect programs to ship with what are commonly referred to as “sane defaults”. This means that for a new user, there shouldn’t be very much to configure in order to get up and running. The settings are preconfigured in a way that makes sense for the average use-case. This, combined with a general apathy towards customization/optimization from the average user, has left many of the most powerful tools, programs, and features unexplored.

This is a tragedy, because if we fully harnessed the power of the software we already use, our productivity would greatly increase. Take document writing as an example. There are a million programs out there designed to type documents, all with varying features and specializations. However, a good portion of the population will never move beyond Microsoft Word, or whatever their first document editor was.

Time after time, I see students (and professors) attempt to format a math heavy document in Word, completely unaware that LaTex or R Markdown exists. Or even worse, preform tedious and repetitive text editing tasks that probably could have been automated with a single vim command.

Everytime I find myself thinking “there has got to be a better way to do this…”, I actually take the time to find and master that better way. And if it doesn’t exist, I do my best to make it. Some might consider such obsessive configuration a waste of time. Why spend hours writing config files when you could just be writing code?

There are lots of benefits of building your own workflow, from increasing raw productivity to passively gaining knowledge by exposure to the granular settings offered by most tools and programs. In this series of posts, I am going to explain this philosophy and how I apply it to my own systems.

About Me

Sat, 23 Mar 2019 00:00:00 +0000

RSS feed with full post text

My Resumé

~$ ./gideon.sh

Output:
------------------------

Position: Researcher, Employee, and Student at WWU
Languages: Python, JavaScript, Java, C, Racket, bash, Powershell
Interests: Infosec, Quantum Computing, Biotech, Music
Github: https://www.github.com/GideonWolfe

gpg file

153B 370B 0E00 4CDE 261C
CFC9 A97C 79D4 0564 E167

Projects

Mon, 01 Jan 0001 00:00:00 +0000

Linux

Zathura-Pywal: Dynamically generate colorschemes for Zathura based on colors from Pywal
Gnuplot-Pywal: Dynamically generate colorschemes for Gnuplot based on colors from Pywal
My Dotfiles: My system configuration files
CodeOverTime: A bash utility that logs the lines of code in a git repo over the years
Chameleon: A script to automate desktop theming on Linux
canvas-tui: A TUI for the Canvas education system

Science

pTable: A terminal based chemistry toolkit
Single and Pairwise Mutations and Their Impact on SARS-CoV-2 Proteins: Research paper on SARS-Cov-2 protein mutations.

Programming

vim.reaper 💀 A Hackable, Fully Featured, Rice Friendly Neovim Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Linux:

Workflow

Math

gnuplot
- Introduction to Gnuplot
- Theming Gnuplot

Programming

Security

Creating a pocket pentest platform with P4wnP1: Part 1

System Administration

Science

Music

Alava - A new single by the Dawn Bombs

Gideon Wolfe

How to add a picom toggle to polybar

Intro

Solution

Conclusion

xmenu: Desktop Agnostic Menu Utility

Introduction

Installation

Usage

Theming

polybar integration

My config

Conclusion

Single and Pairwise Mutations and Their Impact on SARS-CoV-2 Proteins

Intro

The Class

The Project

Results

Conclusion

Canvas in the terminal with canvas-tui

Intro

History

canvas-tui

Dashboard

Course Overview

Other Pages

Conclusion

Agents Standing By: Notifications through Telegram

Create your Telegram bot

Setting up the Telegram Agent

Sending Data

Email in the Terminal: Configuring Neomutt

Intro

Files

General Settings

Colors

Mappings

Pager

Sidebar

Mailcap

Accounts

Conclusion

Agents Standing By: Monitoring National Emergencies

NOTE!

Conclusion

Agents Standing By: Monitoring Website Status

Intro

Creating our Status Agent

Creating our Alert Agent

Conclusion

Agents Standing By: Setting Up Huginn

Intro

Docker setup

Agents Standing By: Introduction to Huginn

What is Huginn?

What about Grafana?

Conclusion

Keeping your docker containers updated with Watchtower

Who needs this?

Setting it up

Using preset layouts in i3

How to create a layout

Deploying a layout

Conclusion

Adding RSS to your Hugo Site

Editing config.toml

Showing all your content

Conclusion

Adding Google Analytics to your Hugo Site

Set up your Google Analytics account

Configuring Hugo

header.html

config.toml

Note!

Conclusion

vim.reaper: 💀 What's new?

What is vim.reaper?

Lazygit

Lazydocker

vimtex

Chat like it's 2000 2020: Using IRC in the modern world