Friday, May 27, 2022

In Search of a Reliable Gmail Permalink

One of my favorite features of Gmail is that every message is backed by a unique URL. I frequently use this URL to easily refer back to a message. This works great, until it doesn't.

The Problem

Suppose this intriguing message from Mr. Helms is something I want to follow-up on. After all, seven million dollars is a lot of money. Normally, I'd grab the URL to this message and add it to a card on my 'On Deck' Trello Board. But in the Android version of the Gmail App, there's no obvious way to get a URL to a message.

(Incidentally, the above screenshot is from a Samsung DeX session. DeX is amazing and is on my list of topics to blog about.)

A Solution

The obvious work-around is to visit in the phone's browser. This will bring up the Desktop Gmail interface, which does have a URL to the message:

Depending on your device's screen size, you may end up at an alternative version of Google's web mail UI.

In this case it took some fiddling to land at the Desktop version of GMail. First I had to convince Google that I wanted to see the Basic HTML interface, and from there I could click over to the Standard interface:

So while it's possible to convince Chrome on my phone to navigate to the Desktop Gmail UI, in practice this can be maddeningly difficult to do. The multiple Google Accounts on my phone, multiple Gmail UIs and the 'smart' logic that guesses what I want to see, means that I often end up clicking in circles in search of the right page.

A Better Solution. Maybe.

An obvious replacement for all this clicking would be to leverage the Gmail API. I even have an existing command line tool, gmail_tool, that interacts with this API.

Currently, I can use gmail_tool to get me the details of the message in question:

$ gmail_tool -a list -q "label:SPAM Helms"
180df45360340ba9:.GREG HELMS, Director. Airport Storage and Cargo Unit Erie International Airport (Pennsylvania) PA 16505, USA eMAIL.

All that's need is to map the above message to the Gmail URL:

But alas, that's where things get tricky. The magic token FMfcgzGpFzwKLQTspjZvpjCvNxLXMQjz is neither a message ID nor a thread ID. Apparently, this is a 'view token' and while you can decode it in some respects, there's no obvious mapping from information in the API to this token.

A Better Solution. For Sure.

But all is not lost. This article suggests another way forward to uniquely identify a message within Gmail. The answer, which is obvious in hindsight, is to use the search operator rfc822msgid.

This is quite sensible. Each message comes with it's only unique Message-Id header. Searching by this value you should always bring up the one message in question.

So while I can't get the token FMfcgzGpFzwKLQTspjZvpjCvNxLXMQjz from the Gmail API, I can get the headers for a given message, from there search out the Message-Id value.

$ gmail_tool -a list -q "label:SPAM Helms"
180df45360340ba9:.GREG HELMS, Director. Airport Storage and Cargo Unit Erie International Airport (Pennsylvania) PA 16505, USA eMAIL.

# Pull the full JSON for all messages associated with thread id: 180df45360340ba9
$ gmail_tool -a get -i 180df45360340ba9 -v |  head -4
  "id": "180df45360340ba9",
  "historyId": "357728706",
  "messages": [

# Dump out all the headers associated with this thread
$ gmail_tool -a get -i 180df45360340ba9 -v | \
  jq  '.messages[] | .payload.headers[] | .name ' | gmail_tool -a get -i 180df45360340ba9 -v |  jq '.messages[] | .payload.headers[] | .name '

Once I combined my knowledge that you can search by rfc822msgid and how to access the message headers using the Gmail API, I was able to put that together into a simple option for gmail_tool:

# set the shell variable $tid to the matching thread id
$ tid=$(gmail_tool -a list -q "label:SPAM Helms" | cut -d: -f1)

# look up the URL to for $tid
$ gmail_tool -a url -i $tid

Visiting this URL takes me a Gmail search result page with the one message I'm seeking:


Here's the latest version of gmail_tool with both url and header options added:


## command line tools for working with Gmail.
AUTH_TOKEN=`gapi_auth -i $CLIENT_ID -p $CLIENT_SECRET -s $API_SCOPE token`

usage() {
  cmd="Usage: $(basename $0)"
  echo "$cmd -a init"
  echo "$cmd -a list -q query [-v]"
  echo "$cmd -a get -i id [-v]"
  echo "$cmd -a labels"
  echo "$cmd -a update -i id  -l labels-to-add -r labels-to-remove"
  echo "$cmd -a headers -i id"
  echo "$cmd -a url -i id"
  echo "$cmd -a messages -q query [-v]"


filter() {
  if [ -z "$VERBOSE" ] ; then
    jq "$@"

listify() {
  expr="[ "
  for x in "$@" ; do
    expr="$expr $sep \"$x\""
  expr="$expr ]"
  echo $expr

while getopts ":a:r:q:i:l:vp" opt ; do
  case $opt in
    a) ACTION=$OPTARG             ;;
    v) VERBOSE=yes                ;;
    q) QUERY="$OPTARG"            ;;
    l) LABELS_ADD=$OPTARG         ;;
    r) LABELS_REMOVE=$OPTARG      ;;
    i) ID=$OPTARG                 ;;
    p) PAGING=yes                 ;;
    \?) usage                     ;;

invoke() {
  root=$1 ; shift
  curl -s -H "Authorization: Bearer $AUTH_TOKEN" "$@" > /tmp/yt.buffer.$$
  next_page=`jq -r '.nextPageToken' < /tmp/yt.buffer.$$`

  if [ "$PAGING" = "yes" ] ; then
    if [ "$next_page" = "null" -o -z "$next_page" ] ; then
      cat /tmp/yt.buffer.$$
      rm -f /tmp/yt.buffer.$$
      jq ".$root"  < /tmp/yt.buffer.$$ | sed 's/^.//'> /tmp/yt.master.$$
      while [ "$next_page" != "null" ] ; do
        curl -s -H "Authorization: Bearer $AUTH_TOKEN" "$@" -d pageToken=$next_page |
          tee /tmp/yt.buffer.$$ |
          ( echo "," ; jq ".$root" | sed 's/^.//' ) >> /tmp/yt.master.$$
        next_page=`jq -r '.nextPageToken' < /tmp/yt.buffer.$$`
      rm -f /tmp/yt.buffer.$$
      echo "{ \"$root\" : [ "
      cat /tmp/yt.master.$$
      echo ' ] }'
      rm -f /tmp/yt.master.$$
    cat /tmp/yt.buffer.$$
    rm /tmp/yt.buffer.$$

case $ACTION in
    gapi_auth -i $CLIENT_ID -p $CLIENT_SECRET -s $API_SCOPE init
    if [ -z "$QUERY" ] ; then
      echo "Uh, better provide a query"
    invoke threads -G $API_BASE/users/me/threads \
           --data-urlencode q="$QUERY" \
           -d maxResults=50 |
      filter -r ' .threads[]? |  .id + ":" + (.snippet | gsub("[ \u200c]+$"; ""))'

    if [ -z "$ID" ] ; then
    invoke messages -G $API_BASE/users/me/threads/$ID?format=full \
           -d maxResults=50 |
      filter -r ' .messages[] | .id + ":" + (.snippet | gsub("[ \u200c]+$"; ""))'

    invoke labels -G $API_BASE/users/me/labels |
      filter -r ' .labels[] | .id + ":" + .name'

    if [ -z "$ID" ] ; then
      echo "Missing -i id"

    if [ -z "$LABELS_ADD" -a -z "$LABELS_REMOVE" ] ; then
      echo "Refusing to run if you don't provide at least one label to add or remove"

    body="{ addLabelIds: $(listify $LABELS_ADD),  removeLabelIds: $(listify $LABELS_REMOVE) }"

    invoke messages -H "Content-Type: application/json" \
           $API_BASE/users/me/threads/$ID/modify \
           -X POST -d "$body" |
        filter -r '.messages[] | .id + ":" + (.labelIds | join(","))'

    if [ -z "$ID" ] ; then
      echo "Missing thread ID"

    message_id=$(gmail_tool -a get  -i $ID -v |
                   jq -r '.messages[0].payload.headers[] | select(.name | ascii_downcase == "message-id") | .value| @uri' |
                   sed -e 's/^%3C//' -e 's/%3E$//' )

    echo "$base_url:$message_id"


    if [ -z "$ID" ]; then
      echo "Missing thread ID"

    gmail_tool -a get -i $ID -v |
      jq -r '.messages[] | .id + ":" + (.payload.headers[] | .name + ":" + .value)'


Monday, May 23, 2022

Duct Tape Engineering

G. has never met a ball he didn't want to befriend. So last night I decided we needed to build a ramp to give his friends a bit of new terrain to play on. Mind you, I didn't plan for this, so we had to improvise. Ultimately, I grabbed an Amazon box and a big 'ol roll of duct tape. And thus began our first experience in Red Neck Engineering. The 'ramp' we constructed wasn't much, but it got the job done.

I'm proud to report that G now knows two of life's most important phrases: Duct Tape and More Duct Tape.

I'm hardly an expert on these matters, but surely the next lesson we need to tackle is WD-40, right? Because really, what else do you need besides WD-40 and Duct Tape? Oh yeah,channel lock pliers.

Wednesday, May 18, 2022

Gotcha: Emacs on Mac OS: Too Many Files Open

Generally, the MacPorts version of Emacs works great on my Mac Mini. But every so often, I'd hit a Too many open files error.

The *Messages* buffer was little help as it just repeated what I already knew:

insert-directory: Opening process input file: Too many open files, /dev/null [2 times]

I'd attempt to close buffers or shut down projectile projects, but there was nothing I could reliably do to recover from this error. Ultimately, I had to do the unthinkable and restart emacs.

I tried the obvious fix: telling my Mac to allow processes to open more files. I followed this recipe:

$ sudo cat /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  <plist version="1.0">

$ sysctl kern.maxfiles
kern.maxfiles: 20480
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 64000

It didn't help. Perhaps I was setting the limit incorrectly for my particular version of the OS. Or maybe I was setting the value too high, and the system was reverting it to something smaller. Or maybe all was good at the OS level, and it was a bash ulimit problem.

I considered all of these scenarios, but no matter how I set the file limit or what I set the file limit too, emacs kept hitting a 1024 open files limit. I could easily confirm this with lsof:

$ ps auxww|grep Emacs
ben              15944   0.0  0.7 412688032 111280   ??  S    Mon06AM  71:07.20 /Applications/MacPorts/
$ lsof -p 15944 | wc -l

After much frustration and searching, I finally stumbled on this reddit thread where a fellow emacs user complained about the 1024 max file limit:

Been trying to figure out a way to get out of this trap… I got a new Mac as a work laptop and I can’t seem to update the file descriptors. I updated it for the system but whenever eMacs is opened, ulimit is still at 1024.

Thankfully, there was a helpful reply:

I hit this all the time, as I work on a large monorepo with lsp-mode (and sometimes treemacs, which also watches stuff).

Whenever it happens I run M-x file-notify-rm-all-watches and things go back to normal for a while.

Last time I looked into this, you could not work around it by adjusting ulimits or literally anything. It's a core limitation of a low-level API used internally and is not configurable. I will try to find the previous discussion.

Aha! This made sense, as I've added lsp-mode to my workflow recently.

I was delighted to find I wasn't the only one having this problem, and more importantly, there was an easy work around.

Alas, when I tried to execute M-x file-notify-rm-all-watches I found that my version of emacs didn't have this function.

A quick Google search turned up this source file for filenotify.el. It does have file-notify-rm-all-watches defined as follows:

(defun file-notify-rm-all-watches ()
  "Remove all existing file notification watches from Emacs."
   (lambda (key _value)
     (file-notify-rm-watch key))

I copied the code, untouched, into my emacs configuration. Next time I got the dreaded Too many open files error, I ran M-x file-notify-rm-all-watches and just like that, emacs was happy again. And so was I.

Wednesday, May 11, 2022

Faith, Patience and Careful Observation - Lessons From A Plant Scavenger Hunt

The Air Force's SERE Handbook, the text designed to teach downed pilots how to survive under the most hostile of conditions, calls out 12 plants with useful medicinal properties. I took that list as a challenge to seek out these plants. Here's what I've learned along the way.

Common Plantain. Holy smokes is this plant common. It's everywhere in the DC area. The lesson: if there's one medicinal plant worth mastering in this area, it's plantain.

Dandelion. Dandelion's are a lesson in patience. Here's a common plant that's super easy to identify. And yet, for most of the year, they're invisible. As prolific as all these plants are, finding them always comes down to being in the right place, at the right time.

Dog Rose. Dog roses are what this project is all about. If you had told me that wild roses grew along a running and walking route I frequented, I'd tell you that you were crazy. And yet, that's exactly where I found them. I also learned about the multiflora rose, which is so common it's considered an invasive species. I now see them all the time. While multiflora rose isn't considered the ideal rose to harvest, it's still an excellent source of vitamin C and other nutrients.

Wild Garlic. Wild garlic serves as a reminder that plant naming and identification can get fuzzy. Wild onion and garlic are related and look so alike when growing in your yard that they are often grouped together. It doesn't help that a synonym for wild onion is 'cow garlic'. This article bypasses the differences and offers this advice:

If a plant looks like an onion and smells like an onion you can eat it. If a plant looks like a garlic and smells like a garlic you can eat it. If you do not smell a garlic or an onion odor but you have the right look beware you might have a similar-looking toxic plant.

Wild Onions. See wild garlic. Grown up wild onion (or is it wild garlic?) is crazy looking. It has a Dr Suessian looking nodule that develops into flowers; what I believe is called an 'umbel.' The first time I came across wild onion that had this nodule, it was outside of a gas station along Washington Boulevard. I found myself snapping pics like I'd found some rare species. It was awesome.

Mullein. Years ago I discovered mullein, so I knew it was in the area. However, I had to be patient before I could log a specimen. If you're not familiar with mullein, you're in for a treat. This is a fascinating looking and feeling plant, and you're going to be amazed at how common it is.

White Willow. It took longer than I'd like to admit to remember the collection of massive willow trees growing along the Potomac river, not far from our house. The lesson: sometimes a plant can be so obvious, it's hidden.

Sweet Gum. Finding a sweet gum tree in the area was going to be like finding a needle in a haystack. Luckily, I had a magic needle finder. Aka, the Internet. Turns out, Arlington, VA publishes a list of notable trees. Sure enough, two of them were gum trees. I added the address of one of them to a running route, and in no time, I found myself face to face with a splendid looking gum tree. Once I knew what the distinctive star like leaves looked like, I had no problem identifying another gum tree just a few blocks way. The lesson: be creative.

Yarrow. I looked high and low for yarrow, keeping an eye out on every hike I went on. I finally found it two streets over in a neighbor's yard, and in a landscaping feature at a local park. The lesson: the plant detection game doesn't stop just because you're traipsing through suburbia.

Jewelweed. Jewelweed is a lesson in hope and faith. I so wanted to find this plant, yet in hike after hike it eluded me. That is, until one day when I was running and took a random trail into the woods. It led down to the Potomac river where I found a massive stand of Jewelweed. It was gorgeous. The lesson: these plants are out there, so don't give up.

Aloe Vera. Aloe Vera doesn't grow locally, so the lesson here was about taking this little project on the road. I came across a brilliant example of Aloe Vera while hiking in Florida.

I've have two plants left on my list: balsam and peppers. Baslam, like Aloe Vera, doesn't grow locally, so I may have to catch it while out of town. Alternatively, they are frequently used for Christmas trees, so maybe I'll get lucky and find an example during the Christmas season.

The last plant on the list is peppers. With a little reflection, I realize I've left them for last because I've got no expectation of finding them. This is based on some flimsy reasoning: peppers are very distinctive, and I've never seen one growing in the wild. Why should I find one now?

With just a few minutes of research, I can see that my logic is faulty. Wild peppers are absolutely a thing, and most importantly they grow in my climate. I need to change my mindset: wild peppers do exist and I shall find one!

In short, This little exercise has taught me the same lessons over and over: have faith, be patient and stay sharp; what you're looking for is out there.

Thursday, May 05, 2022

The Set to Watch from Ultra Miami Musical Festival

After a two year hiatus, the Ultra Musical Festival returned to Miami. For a few weeks after the event, YouTube was popping with musicians publishing their sets.

I know what you're thinking: Ben, I'd like to get a flavor for this year's festival, but I'm not sure which set I should watch? Not to fear, I've got you covered.

If you listent to a single set from this year's festival, make it Fisher's.

Mind you, I don't know who 'Fisher' is, and much of the music he played isn't my preferred style. But that's the sort of the point. Here's an unknown (to me) DJ playing new (to me) tunes, yet he ended up taking me on a delightful musical journey.

All I could think when he finished his set was, how the heck did he just do that?

Here, give his set a listen and see hear what I mean:

Banging, right?

Wednesday, May 04, 2022

Special Delivery: From A Random Linux Box to My Wrist

I've got a phpunit test suite on a Linux server that now takes a few minutes to run. When it's done, I'd like to get a message on my watch telling me whether it completed successfully or not. This let's me kick off the tests and go on to my next task (probably: eating) and will give me just the nudge I need to either return to work on the tests or keep going with the new task.

Here's the recipe I used to accomplish this feat. The goal here is to keep the script that runs on the Linux server nice and simple so that I can move it around to other locations as I decide to spread around this behavior.

The Linux Script

On the Linux side, I've got a simple script that I can invoke like so:

$ phpunit --stop-on-failure tests/ ; andnotify "Test are done! Status: $?"

The magic variable $? will be zero if tests completed without error, or a non-zero value if there was a problem.

The andnotify script is little more than a wrapper around curl which makes a request to Tasker's AutoRemote:


## Send data to an android device using Tasker's Auto Remote
key='<Your AutoRemote Key>'

if [ $# -eq 0 ] ; then
  echo "Usage: `basename $0` message"
  exit 1

echo "Notify=:=$@" > $HOME/.andnotify
curl -s $base -d "key=$key" --data-urlencode "message@$HOME/.andnotify"  > /dev/null

The Tasker Code

andnotify sends a message to the AutoRemote API endpoint, which can be received by a Tasker Profile. The magic happens when you create a profile under: Event » Plugin » AutoRemote.

This profile will receive the text sent via curl above. I'm using the variable %arcomm, which will grab all the text after =:= in the AutoRemote message.

Once the text is received, it's handed to a trivial Tasker Task, which contains a single Notify Action in the body of the task.

The Notify Action causes the message to appear as an Android Notification.

To The Watch

Finally, I've set my Garmin VivoActive 4 up to pass AutoRemote notifications to my watch.

The ensures that notification that are posted to my device, show up on my watch.

Tuesday, May 03, 2022

Boat Tree

It's simple. Trees need dirt.
Wrong. Said the tree on a boat.
What else says I'm wrong?

Monday, May 02, 2022

Bat Mitzvah Awesomeness!

I'm trying to imagine what it would be like if I could have turned to Jared in college, 25 years ago, and said:

I've got good news, and I've got bad news. The bad news is that due to a deadly pandemic, we're going to have to attend your Son's Bar Mitzvah virtually. The good news is, your Daughter's Bat Mitzvah was off the hook!

This past weekend we attended Maya's Bat Mizvah, and it was an all around pleasure. We relished seeing her parents, and she did a fantastic job with both her Torah and Haftarah portion. We even got in an 8 mile stroll along the Ballenger Creek Trail. The party in the evening was excellent, with food and desserts that were top notch.

I'm telling you, this new trend of getting together in person to celebrate events may have staying power. It's pretty awesome.

To many more simchas together!