Monday, May 10, 2021

ShulCloud on the command line: Now with form support

Our shul is in the process of switching to In-Person and Zoom based services from being strictly Zoom based (whoo!). To power this process, we're making use of ShulCloud's forms. Naturally, I wasn't excited about logging into ShulCloud's interface to work with these forms, so I decided to add on to the command line tool I'd previously built.

The shulcloudassist tool, which you can find on github, has the following new options:

$ shulcloudassist -a forms
...generate a command line friendly list of all forms...

$ shulcloudassist -a export-form -i 94848
...dump the submissions for form 94848 as CSV...

ShulCloud offers a form submission export URL that I was able to tap into without issue. Using this URL and theauthentication functionality I'd previously built, it was trivial to pull the form submissions given a form ID.

Getting the list of forms, however, proved to be trickier. There wasn't a CSV export option that I could find that gave me this info. It looked like my best bet was to scrape the HTML <table> that contained the list of forms.

To accomplish this, I used a new tool: pup. Pup is inspired by jq, a tool that makes working with JSON a far simpler process. In fact, pup will output JSON, no doubt for further processing by jq.

It took a little experimentation, but ultimately, I got pup to do what I wanted. For example, to pull out the 3rd form's name, I run the following pup command:

curl -s -G ... $BASE/admin/forms.php > $DOC_SNAPSHOT

name=$(cat $DOC_SNAPSHOT | \
  pup "table.listing tr:nth-child(3) td:nth-child(1) text{}")

echo $name

By making creative use of nth-child(...) I can navigate the HTML document and get back just the data I need.

While the process is messy, it does seem reliable. And any tool that brings data to the command line is a win in my book.

Friday, May 07, 2021

3 Plant Mysteries, 1 Simple Tricks for Solving Them

Last weekend we hiked the BROT. Peeking out at the base of a tree I saw this little guy:

What the heck is it?

Or consider this beauty right here. It's obviously what would be called a wildflower, but what plant is it specifically?

And check out the funky leaves of this plant. The dark green and pronounced vein pattern reminded me of spotted wintergreen. Was this spotted wintergreen or a relative?

I wouldn't consider a hike a success if I didn't walk away with a heap of question like these. And thanks to Plant Snap and Google Images it's usually possible to get them answered.

Using a new tool, however, I was able to untangle these mysteries in a single click. Here's what I learned:

Pic 1 is Conopholis americana, known as American Cancer Root, Squawroot or Bear cone. I was right to be confused as it "is a fully parasitic plant that occurs only where it can grow attached to the roots of some species of oak." It has no chlorophyll, hence the lack of any green. Technically, the plant is edible, though reports suggest it tastes awful.  The name 'squawroot' is derived from the medical benefit related to treating menopause and other lady ailments.

Pic 2 is the aptly named Virginia Spring Beauty. Apparently, they are tasty:

In his book "Stalking The Wild Asparagus" Gibbons wrote about eating them daily if not twice a day for several weeks. He said: "We tried them fried, mashed, in salads, and cooked with peas, like new potatoes. All these ways were completely successful, but, as regular fare, we preferred them just boiled 'in the jackets.' My friend grew so fond of this food that he was afraid he would experience withdrawal symptoms when the supply was exhausted." While Gibbon's friend thought they tasted like potatoes Gibbons thought they were sweeter, closer to chestnuts in flavor.

Pic 3 is most definitely not Spotted Wintergreen, and when comparing the two photos they don't look anything alike. What I found is Hieracium Venosum, known as Hawkweed or Rattlesnake Weed. Apparently, it gets it name as it thrives where rattle snakes occur. (Though no, there are no rattle snakes on the BORT). While it may have some medical benefits, they appear to be relatively obscure.

So how did I figure all this out so quickly? Why Google, of course. Specifically using Google Lens. Below are what happens when I click the "lens" button in Google Photos for these examples:

I suppose I shouldn't be surprised that Google can find web pages that contain photos that are a close match to these examples. But still, to see it in action it's hard not to be blown away.

So next time you're out and about and see some strange looking plant, give Google Lens a try. I think you'll be amazed with what you can learn.

Tuesday, May 04, 2021

A Long Walk to Image-Based-Map Story Telling

I did a bit of Yak Shaving this morning. See, I wanted to write a post about a recently discovered app feature that does an amazing job of identifying wildflowers. But, to give that post context I thought it would be helpful to put the wildflowers pics on a map. But to get photos on a map, I figured I needed to extract the lat/long from each of the images' metadata. And to extract the image metadata, I needed re-learn how to pull exif data using ImageMagick.

So while I had intended to write about wildflower identification, I found myself writing the shell script below that extracts the latitude and longitude from an image.

But my Yack Shaving didn't stop there.

My Maps Awesomeness

With the latitude and longitude derived, I then went over to Google's My Maps to figure out how I could render the photos on a map.

I opened up My Maps, clicked Import and was shocked to see that there's a Google Photos option:

I selected a bunch of wildflower pics from our recent hike and to my delight, Google precisely placed them on the map. All the work I did to extract geo location data from images was unnecessary (but I'm confident it may be handy in the future).

Wrestling with Garmin Connect

To complete the wildflower map I needed to export our route from connect.garmin.com and import it into a new layer on My Maps. Garmin, to their credit, includes GPX and KML export options and My Maps reads both these formats. This was going to be a breeze!

The export from Garmin went smoothly, but the import into My Maps failed. The export file was over 5 Megs, the size limit for My Maps. Ugh.

Looking at the export file, I noticed at least two optimizations I could make to shrink the file:

<trkpt lat="38.72506211511790752410888671875" lon="-77.3301221989095211029052734375">
  <ele>78.40000152587890625</ele>
  <time>2021-05-02T11:00:34.000Z</time>
  <extensions>
    <ns3:TrackPointExtension>
      <ns3:hr>105</ns3:hr>
    </ns3:TrackPointExtension>
  </extensions>
</trkpt>
<trkpt lat="38.725062198936939239501953125" lon="-77.33012236654758453369140625">
  <ele>78.40000152587890625</ele>
  <time>2021-05-02T11:00:35.000Z</time>
  <extensions>
    <ns3:TrackPointExtension>
      <ns3:hr>105</ns3:hr>
    </ns3:TrackPointExtension>
  </extensions>
</trkpt>
...

Those lat/lon values are crazy. There's no reason to have that many decimal points of precision. Also, the tags prefixed with n3: appear to be heartrate metadata which My Maps won't use. Running the following unix commands brought the 9.2 meg GPX file down to 4.9 meg, perfect for My Maps:

$ sed -E 's/([0-9]{6})[0-9]+/\1/g' bort.gpx |  grep -v ns3 > bort.smaller.gpx

A BROT Wildflower Map

And check it out, here's my wildflower map of the Bull Run Occoquan Trail (BROT). Pretty sweet, right?

Image Based Maps for Storytelling

The more I think about this My Maps capability, the more I'm both impressed by it and amazed I hadn't considered searching for it in the past. Viewing pictures on a map adds an impressive amount of context. It turns what may be an un-inspiring one-off picture into a fascinating collection.

Consider the difference between a photo of a meal you took while on vacation, and a map showing every meal you ate as you traveled the country. While none of the pictures themselves may be particularly special, as a whole, they would be a fun and unique way to visualize your entire trip.

I'm glad to add map based image collections to my blogging toolbox.

Bonus: Image Metadata Extraction

Here's the script I wrote to extract latitude and longitude metadata from images. It was inspired by this discussion.

#!/bin/bash

##
## help out with manipulating images
##

usage() {
  cmd=$(basename $0)
  echo "Usage: $cmd -a lat -f <img>"
  echo "Usage: $cmd -a lng -f <img>"
  echo "Usage: $cmd -a latlng -f <img>"
  echo "Usage: $cmd -a exif -f <img>"
  exit
}


while getopts ":a:f:h" o; do
  case "${o}" in
    a)
      action=$OPTARG
      ;;
    f)
      file=$OPTARG
      ;;
    * | h)
      usage
      ;;
  esac
done

case "$action" in
  lat | lng)
    if [ -z "$file" ] ; then
      usage
    fi
    
    case $action in
      lat)
        prop=GPSLatitude
        ;;
      lng)
        prop=GPSLongitude
        ;;
    esac


    raw=$(identify -format "%[exif:$prop]" $file)
    if [ -z "$raw" ] ; then
      echo "exif:$prop not found in image"
      exit
    fi

    ref=$(identify -format "%[exif:${prop}Ref]" $file)
    if [ -z "$ref" ] ; then
      echo "exif:${prop}Ref not found in image"
      exit
    fi

    deg_n=$(echo $raw | cut -d, -f1 | cut -d/ -f1)
    deg_d=$(echo $raw | cut -d, -f1 | cut -d/ -f2)
    min_n=$(echo $raw | cut -d, -f2 | cut -d/ -f1)
    min_d=$(echo $raw | cut -d, -f2 | cut -d/ -f2)
    sec_n=$(echo $raw | cut -d, -f3 | cut -d/ -f1)
    sec_d=$(echo $raw | cut -d, -f3 | cut -d/ -f2)

    loc=$(convert xc: -format "%[fx:($deg_n/$deg_d) + ($min_n/$min_d)/60 + ($sec_n/$sec_d)/3600]" info:)


    if [ "$ref" = "S" -o "$ref" = "W" ] ; then
      echo -n "-"
    fi
    
    echo $loc
  ;;

  latlng)
    if [ -z "$file" ] ; then
      usage
    fi
    
    echo -n $(imgassist -a lat -f $file)
    echo -n ","
    echo -n $(imgassist -a lng -f $file)
    echo
    ;;

  exif)
    if [ -z "$file" ]; then
      usage
    fi
    
    identify -verbose $file
    ;;
  *)
    usage
esac

Oh, and I really do plan on posting about wildflower identification. That is if I can avoid the Yak.

Monday, May 03, 2021

Hiking (nearly all) The Bull Run Occoquan Trail

Yesterday we hiked the Bull Run Occoquan Trail (aka BROT). I classify this as a One Day Trail, because every time I'd seen it mentioned I remarked that one day I was going to hike it. And yesterday was that day.

What distinguishes BROT from other local area trails is its combination of length and back-country feel. At a little over 17 miles, it's one the area's longest woodland trails. Yesterday, the weather cooperated, and we had an outstanding day of hiking.

We opted to traverse the trail from Fountain Head Regional Park to Bull Run. This was a wise decision because the Fountain Head Section has a heap of elevation gain and loss which was nice to accomplish while we were fresh. The trail was well marked, and at only two points did we find that the blue trail markings and the All Trails route disagreed. And even those cases were easy to reconcile.

Some of the AllTrails reviews painted a pretty bleak picture of the trail: claiming that the trail was overgrown, required bushwhacking or in at least two instances, suggested that you were required to make a waist-deep stream crossing. None of this is true. Except for one recently downed tree, the trail was smooth sailing the whole way. All the stream crosses had bridges or stepping stones.

We did run into one logistical glitch. When we arrived at Bull Run to drop off our car it was 6:15am and the park wasn't scheduled to open until 7am. Our fix was to park at an access point off of Route 28 about a mile into the hike. I was bummed to be losing a mile of the trail, but preferred that to scrubbing the whole adventure. Fountain Head Regional Park opened at 6am (despite Google claiming it too opened at 7am), so we had no problem parking there. We were on the trail by 7am as we hoped.

While the trail didn't offer much in terms of views, and we didn't have time to explore the history of the area as much as I'd liked, it more than made up for these losses. Much of the trail takes you along water which was both picturesque and quite inviting. There were also numerous types of wildflowers in bloom, including Virginia blue bells, Virginia spring beauty, Meadow buttercup, Golden ragwort and more. There was a variety of Azaleas blooming that looked downright spectacular. Heck, even the Skunk Cabbage looked impressive.













I also came across some unique finds, including a Pink Lady's Slipper which is a variety of orchid(!) and the odd looking and equally oddly named American Cancer Root. It was a definitely a wonderful day to be in the woods.


I also spent quite a bit of time pondering this plant right here:


She may not look like much. However, unlike most of her plant neighbors, she's destined for greatness. That's a baby Tulip Poplar. Once mature, it'll be among the tallest trees in the forest. Tulip Trees are so large that I didn't notice their distinctive leaves and flowers until one pooped on me.

Think about all countless things that have to go right for this tiny seedling to turn into a mighty tree. It seems impossible. And yet, as we hiked through the forest, we were surrounded by mature Tulip Trees. That's amazing.

Overall, the BORT is a splendid trail and one I hope to come back and explore in the future.
















Friday, April 30, 2021

Failing (An Insignificant) Stress Test

Earlier this week I was sitting with Shira while she completed a fitness-related test at the gym. From my perspective, there were two unique features of the test: one, Shira was obviously not a happy camper while doing it and two, she couldn't talk.

The test was just a few minutes long, so I immediately came up with the idea of distracting her with YouTube clips. The only problem: in the moment I'd forgotten every single TV show and movie we'd ever watched. For the life of me, I couldn't think of a single piece of media we'd enjoyed in the last 22 years of being married.

Grasping for anything, I finally searched for the one show that did come to mind: This Is Us. What came up were clips from a recent episode where two of the main characters have an incredibly intense and awkward scene. It was awful. If anything, it slowed time down further.

My mind raced as I tried to think of literally any search that would bring up a funny and familiar clip. Shira then held up her hands to get my attention. I counted the fingers she held up: eight. Eight she flashed again. And again. What did eight mean?

Finally I had a useful thought: I had a pen and notepad in my pocket! Shira couldn't talk, but she could write. I pulled them out and handed them to her. She wrote: 9 9.

Huh? Oh, 99, as in Brooklyn 99! That's a goofy show we watch and enjoy. A quick search brought up a stream of cold opens which we used to pass the time.

When the test was over, I confessed to Shira that I was amazed at how much my brained had turned to mush. She agreed. Her proof: she wasn't holding up eight fingers, she was holiding up nine. She was flashing me nine-nine, the only TV show she could think to describe with hand signals. You're telling me, in the moment, I'd lost the ability to count? Yikes.

As I type this, I'm confident I could rattle off a dozen terms that would bring up distracting content on YouTube. Heck, if I had just added "funny" to my This is Us search I'd have found an appropriate video.

Stress makes you dumb. I know this. I've read about this. But to experience it first hand, man, is it humbling. I feel like I was the one tested this last week, and it did not go well.

Wednesday, April 28, 2021

Decoding a Slap Happy Street Artist's Work

Last week I was running across a bridge over Four Mile Run when I noticed a sticker. And then another. And then another. It so grabbed my attention that I ran back to the start of the bridge and captured this wobbly video:

Here's a few stills of the sticker:

I couldn't help but wonder, what was I looking at? Did this image mean something? Was it an ad? Was this done by a board high schooler?

I figured a tineye search would put the matter to bed quickly. But no, the image wasn't found anywhere on the web. I found a list of 40 common stickers in DC and it's not listed among them.

As I dug around for evidence of the sticker, I began to read up on sticker street art (aka sticker slaps, sticker tagging and sticker bombing) and came to appreciate that it's a respected art form. It's sort of like graffiti's faster, more nimble cousin. There's no doubt it's a clever hack for getting your message seen by the masses, and for those in the know, it's a chance to experience art in unexpected places. It's also vandalism and can mar public spaces. If I take the optimistic view and choose to look at the stickers on the bridge as art and not noise, I can begin to see meaning.

The image calls to mind Sherlock Holmes, the detective known for his next-level observational skills. Isn't it deliciously ironic then, that most of the users of the bridge will pass without noticing the dozens of likenesses of Mr. Holmes. What else do we pass by in life because we miss the clues?

And while it's possible (and probably likely) that my lofty explanation is giving this collection of stickers too much credit, I think it's worth noting that the stickers appear to be hand cut. To me, that implies someone put in a fair bit of effort into to placing these stickers. Mainly, this isn't merely the work of a 7 year old haphazardly emptying a book of stickers he found on the nearest convenient surface.

So that was supposed to be the end of the post. I saw an interesting sticker, got an appreciation for a new genre of art, and practiced my art appreciation skills. Last night on my run through DC, I figured I'd take some pics of the stickers I encountered to use in future posts. What I found blew my mind.

First, I saw almost no stickers. The blank canvas that is the back of most signs were pristine. I was running in a touristy part of DC, so perhaps this isn't surprising. But it wasn't a complete bust. As I ran over the 14th Street Bridge into DC I saw one particular sign that was stickered up:

Zooming in on one of the stickers I saw:

Whoa! The border of the sticker matches the Holmes one from above. Here are the two stickers overlaid:

That's got to be the same artist, right? And what are the odds that I'd run by another example of his or her work at just the right moment?

All this raises more questions. But one thing is for sure, the declaration the second sticker makes, Art Is Dead, is greatly exaggerated. Apparently it's alive and well, and being slapped on signs and bridges all over the area.