Friday, May 27, 2022

In Search of a Reliable Gmail Permalink

One of my favorite features of Gmail is that every message is backed by a unique URL. I frequently use this URL to easily refer back to a message. This works great, until it doesn't.

The Problem

Suppose this intriguing message from Mr. Helms is something I want to follow-up on. After all, seven million dollars is a lot of money. Normally, I'd grab the URL to this message and add it to a card on my 'On Deck' Trello Board. But in the Android version of the Gmail App, there's no obvious way to get a URL to a message.

(Incidentally, the above screenshot is from a Samsung DeX session. DeX is amazing and is on my list of topics to blog about.)

A Solution

The obvious work-around is to visit in the phone's browser. This will bring up the Desktop Gmail interface, which does have a URL to the message:

Depending on your device's screen size, you may end up at an alternative version of Google's web mail UI.

In this case it took some fiddling to land at the Desktop version of GMail. First I had to convince Google that I wanted to see the Basic HTML interface, and from there I could click over to the Standard interface:

So while it's possible to convince Chrome on my phone to navigate to the Desktop Gmail UI, in practice this can be maddeningly difficult to do. The multiple Google Accounts on my phone, multiple Gmail UIs and the 'smart' logic that guesses what I want to see, means that I often end up clicking in circles in search of the right page.

A Better Solution. Maybe.

An obvious replacement for all this clicking would be to leverage the Gmail API. I even have an existing command line tool, gmail_tool, that interacts with this API.

Currently, I can use gmail_tool to get me the details of the message in question:

$ gmail_tool -a list -q "label:SPAM Helms"
180df45360340ba9:.GREG HELMS, Director. Airport Storage and Cargo Unit Erie International Airport (Pennsylvania) PA 16505, USA eMAIL.

All that's need is to map the above message to the Gmail URL:

But alas, that's where things get tricky. The magic token FMfcgzGpFzwKLQTspjZvpjCvNxLXMQjz is neither a message ID nor a thread ID. Apparently, this is a 'view token' and while you can decode it in some respects, there's no obvious mapping from information in the API to this token.

A Better Solution. For Sure.

But all is not lost. This article suggests another way forward to uniquely identify a message within Gmail. The answer, which is obvious in hindsight, is to use the search operator rfc822msgid.

This is quite sensible. Each message comes with it's only unique Message-Id header. Searching by this value you should always bring up the one message in question.

So while I can't get the token FMfcgzGpFzwKLQTspjZvpjCvNxLXMQjz from the Gmail API, I can get the headers for a given message, from there search out the Message-Id value.

$ gmail_tool -a list -q "label:SPAM Helms"
180df45360340ba9:.GREG HELMS, Director. Airport Storage and Cargo Unit Erie International Airport (Pennsylvania) PA 16505, USA eMAIL.

# Pull the full JSON for all messages associated with thread id: 180df45360340ba9
$ gmail_tool -a get -i 180df45360340ba9 -v |  head -4
  "id": "180df45360340ba9",
  "historyId": "357728706",
  "messages": [

# Dump out all the headers associated with this thread
$ gmail_tool -a get -i 180df45360340ba9 -v | \
  jq  '.messages[] | .payload.headers[] | .name ' | gmail_tool -a get -i 180df45360340ba9 -v |  jq '.messages[] | .payload.headers[] | .name '

Once I combined my knowledge that you can search by rfc822msgid and how to access the message headers using the Gmail API, I was able to put that together into a simple option for gmail_tool:

# set the shell variable $tid to the matching thread id
$ tid=$(gmail_tool -a list -q "label:SPAM Helms" | cut -d: -f1)

# look up the URL to for $tid
$ gmail_tool -a url -i $tid

Visiting this URL takes me a Gmail search result page with the one message I'm seeking:


Here's the latest version of gmail_tool with both url and header options added:


## command line tools for working with Gmail.
AUTH_TOKEN=`gapi_auth -i $CLIENT_ID -p $CLIENT_SECRET -s $API_SCOPE token`

usage() {
  cmd="Usage: $(basename $0)"
  echo "$cmd -a init"
  echo "$cmd -a list -q query [-v]"
  echo "$cmd -a get -i id [-v]"
  echo "$cmd -a labels"
  echo "$cmd -a update -i id  -l labels-to-add -r labels-to-remove"
  echo "$cmd -a headers -i id"
  echo "$cmd -a url -i id"
  echo "$cmd -a messages -q query [-v]"


filter() {
  if [ -z "$VERBOSE" ] ; then
    jq "$@"

listify() {
  expr="[ "
  for x in "$@" ; do
    expr="$expr $sep \"$x\""
  expr="$expr ]"
  echo $expr

while getopts ":a:r:q:i:l:vp" opt ; do
  case $opt in
    a) ACTION=$OPTARG             ;;
    v) VERBOSE=yes                ;;
    q) QUERY="$OPTARG"            ;;
    l) LABELS_ADD=$OPTARG         ;;
    r) LABELS_REMOVE=$OPTARG      ;;
    i) ID=$OPTARG                 ;;
    p) PAGING=yes                 ;;
    \?) usage                     ;;

invoke() {
  root=$1 ; shift
  curl -s -H "Authorization: Bearer $AUTH_TOKEN" "$@" > /tmp/yt.buffer.$$
  next_page=`jq -r '.nextPageToken' < /tmp/yt.buffer.$$`

  if [ "$PAGING" = "yes" ] ; then
    if [ "$next_page" = "null" -o -z "$next_page" ] ; then
      cat /tmp/yt.buffer.$$
      rm -f /tmp/yt.buffer.$$
      jq ".$root"  < /tmp/yt.buffer.$$ | sed 's/^.//'> /tmp/yt.master.$$
      while [ "$next_page" != "null" ] ; do
        curl -s -H "Authorization: Bearer $AUTH_TOKEN" "$@" -d pageToken=$next_page |
          tee /tmp/yt.buffer.$$ |
          ( echo "," ; jq ".$root" | sed 's/^.//' ) >> /tmp/yt.master.$$
        next_page=`jq -r '.nextPageToken' < /tmp/yt.buffer.$$`
      rm -f /tmp/yt.buffer.$$
      echo "{ \"$root\" : [ "
      cat /tmp/yt.master.$$
      echo ' ] }'
      rm -f /tmp/yt.master.$$
    cat /tmp/yt.buffer.$$
    rm /tmp/yt.buffer.$$

case $ACTION in
    gapi_auth -i $CLIENT_ID -p $CLIENT_SECRET -s $API_SCOPE init
    if [ -z "$QUERY" ] ; then
      echo "Uh, better provide a query"
    invoke threads -G $API_BASE/users/me/threads \
           --data-urlencode q="$QUERY" \
           -d maxResults=50 |
      filter -r ' .threads[]? |  .id + ":" + (.snippet | gsub("[ \u200c]+$"; ""))'

    if [ -z "$ID" ] ; then
    invoke messages -G $API_BASE/users/me/threads/$ID?format=full \
           -d maxResults=50 |
      filter -r ' .messages[] | .id + ":" + (.snippet | gsub("[ \u200c]+$"; ""))'

    invoke labels -G $API_BASE/users/me/labels |
      filter -r ' .labels[] | .id + ":" + .name'

    if [ -z "$ID" ] ; then
      echo "Missing -i id"

    if [ -z "$LABELS_ADD" -a -z "$LABELS_REMOVE" ] ; then
      echo "Refusing to run if you don't provide at least one label to add or remove"

    body="{ addLabelIds: $(listify $LABELS_ADD),  removeLabelIds: $(listify $LABELS_REMOVE) }"

    invoke messages -H "Content-Type: application/json" \
           $API_BASE/users/me/threads/$ID/modify \
           -X POST -d "$body" |
        filter -r '.messages[] | .id + ":" + (.labelIds | join(","))'

    if [ -z "$ID" ] ; then
      echo "Missing thread ID"

    message_id=$(gmail_tool -a get  -i $ID -v |
                   jq -r '.messages[0].payload.headers[] | select(.name | ascii_downcase == "message-id") | .value| @uri' |
                   sed -e 's/^%3C//' -e 's/%3E$//' )

    echo "$base_url:$message_id"


    if [ -z "$ID" ]; then
      echo "Missing thread ID"

    gmail_tool -a get -i $ID -v |
      jq -r '.messages[] | .id + ":" + (.payload.headers[] | .name + ":" + .value)'


No comments:

Post a Comment