Wednesday, February 08, 2023

Same Map, Less Megs: Optimizing USGS Map File Size, Part 1

But Why?

I love the idea of downloading a large area's worth of USGS maps, dropping them on a Micro SD card, and keeping them in my 'back pocket' for unexpected use. Sure, Google and Back Country Navigator's offline map support is more elegant and optimized, but the there's just something reassuring about having an offline catalog at your fingertips.

Getting and downloading maps in bulk is easy enough to do. For example, I can ask my USGS command line tool for all the maps that define Virginia:

$ usgsassist -a topos -l "Virginia, USA" | wc -l

The problem is that each map is about 50 megs. I confirmed this by looking at the 4 maps that back Richmond, VA:

$ wget $(usgsassist -a topos -l "Richmond, VA"  | cut -d'|' -f3)
$ ls -lh
total 427440
-rw-------  1 ben  staff    53M Sep 23 00:20 VA_Bon_Air_20220920_TM_geo.pdf
-rw-------  1 ben  staff    56M Sep 17 00:17 VA_Chesterfield_20220908_TM_geo.pdf
-rw-------  1 ben  staff    48M Sep 23 00:21 VA_Drewrys_Bluff_20220920_TM_geo.pdf
-rw-------  1 ben  staff    51M Sep 23 00:22 VA_Richmond_20220920_TM_geo.pdf

Multiplying this out, it will take about 84 gigs of space to store these maps. With storage space requirements like these, I'll quickly exhaust what I can fit on a cheap SD card.

This begs the question: can we take any action to reduce this disk space requirement? I think so.

But How?

Inside each USGS topo is an 'Images' layer that contains the satellite imagery for the map. By default, this layer is off, so it doesn't appear to be there:

But, if we enable this layer and view the PDF, we can see it:

$ python3 ~/dt/i2x/code/src/master/pdftools/pdflayers \
   -e "Images" \
   -i VA_Drewrys_Bluff_20220920_TM_geo.pdf  \
   -o VA_Drewrys_Bluff_20220920_TM_geo.with_images.pdf

My hypothesis is that most of the 50 megs of these maps go towards storing this image. I rarely use this layer, so if I can remove it from the PDF the result should be a notable decrease in file size and no change in functionality.

But Really?

To test this hypothesis, I decided I'd extract the image from the PDF. If it was as hefty as I thought, I'd continue with this effort to remove it. If the file isn't that large, then I'd stop worrying about this and accept that each USGS map is going to take about 50 megs of disk space.

My first attempt at image extraction was to use the poppler PDF tool's pdfimages command. But alas, this gave me a heap of error messages and didn't extract any images.

$ pdfimages VA_Bon_Air_20220920_TM_geo.pdf images
Syntax Error (11837): insufficient arguments for Marked Content
Syntax Error (11866): insufficient arguments for Marked Content
Syntax Error (11880): insufficient arguments for Marked Content
Syntax Error (11883): insufficient arguments for Marked Content

Next up, I found a useful snippet of code in this Stack Overflow discussion. Once again, PyMuPDF was looking like it was going to save the day.

I ended up adapting that Stack Overflow code into a custom python pdfimages script.

When I finally ran my script on one of the PDF map files I was surprised by the results:

$ python3 ~/dt/i2x/code/src/master/pdftools/pdfimages -i VA_Drewrys_Bluff_20220920_TM_geo.pdf -o extracted/
page_images: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 174/174 [00:18<00:00,  9.51it/s]
pages: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:18<00:00, 18.31s/it]
$ ls -lh extracted/ | head
total 361456
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-100.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-101.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-102.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-103.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-104.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-105.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-106.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-107.png
-rw-------  1 ben  staff   1.0M Feb  8 07:22 VA_Drewrys_Bluff_20220920_TM_geo_p0-108.png
$ ls extracted/ | wc -l

Rather than extracting one massive image, it extracted 174 small ones. While not what I was expecting, the small files do add up to a significant payload:

$ du -sh extracted
176M    extracted

Each of these image files is one thin slice of the satellite photo. Here's an example:

I find all of this quite promising. There's over 170 Megs worth of image data that's been compressed into a 50 Meg PDF. If I can remove that image data, the file size should drop significantly.

Next up: I'll figure out a way to remove this image data, while still maintain the integrity of the map files. I'm psyched to see just how small these file can be!

Friday, February 03, 2023

Organize it or trash it? Taming My Download Directory Mess

Every so often my S22's Downloads folder becomes a tangled thicket of files. When I looked yesterday, it had over a hundred files and it was impossible to know if any were worth keeping.

Rather than sort this mess out, I prefer an easier route: push the files to an Amazon S3 bucket where the mess can live safely out view. Amazon's pricing makes it so that I can store a gig of questionable files for .4 cents(!) a month. At that price, it's much easier to sweep the files under the rug, knowing I can get back to them if needed, then to spend any real time doing cleanup.

Using FolderSync, this process happens in a reliable and automated way. Here's how.

First off, I defined a Amazon S3 account in FolderSync. I then created a new FolderPair with these properties:

The most useful settings are:

  • Sync type: Remote. This is a one way push from my phone into Amazon S3.
  • Sync deletions: (unchecked). It's important to leave this unchecked, as we'll be asking FolderSync to delete the files after they are uploaded.
  • Move files to target folder: (checked). This setting ensures the phone's download directory will be empty after the sync procedure is done.
  • Copy files to a time-stamped folder: (checked). This helps organize our clean up efforts, making a directory in S3 for each time we perform the sync process.

Once set up, I can click the 'Sync' button and have FolderSync work its magic. When my last cleanup effort finished, here were the results:

On the S3 side of things, I can access the files by browsing a date of interest:

One helpful feature of S3 is the ability to set up life-cycle policies. I'm using this, for example, to tell Amazon that the items in this archive bucket aren't going to be accessed frequently. Amazon will then store my files at a much cheaper rate.

I could use this facility to delete these files after an extended period, say, 1 year. While that's tempting, I'm interested to see what I make of my Downloads directory in say, another decade. Will I find some hidden gem in there that at the time seemed like junk, but is now invaluable? Time will tell.

If only IRL de-cluttering could be this easy!

Thursday, February 02, 2023

OCGs meet CLI | GeoPDF Layer Management from the Command Line

As soon as I learned that USGS GeoPDF maps contained multiple layers, I wanted to find a way to toggle these layers from the command line. Combined with my USGS download script, I figured I could retrieve and prepare maps in bulk. While I found many pdf command line tools, surprisingly, I couldn't find any that worked with layers.

My luck finally turned when I learned that PDF Layers are also known as 'Optional Content Groups', or OCGs. Using this term, I found a number of promising libraries and tools.

The most obvious one was the python based pdflayers script. But alas, it failed to identify any layers when I pointed it to a topo PDF.

Next up, I investigated the impressive PyMuPDF library. It offered functions like get_layers() and get_ocgs() which made it quite promising. I installed the library with the command:

$ python3 -m pip install --upgrade pymupdf

and kicked off a REPL to experiment with these functions. My Python skills are weak, but I did confirm that PyMuPDF identifies layers within a USGS GeoPDF.

I hacked together my own version of pdflayers* and before I knew it, I had just the tool I was after. Let's see it in action:

# Look at the existing layers of a freshly downloaded USGS topo map
$ pdflayers -l VA_Alexandria_20220927_TM_geo.pdf
on:237:Map Collar
on:238:Map Elements
on:239:Map Frame
on:241:Federal Administrated Lands
on:242:Department of Defense
on:243:National Park Service
on:244:National Cemetery
on:245:Jurisdictional Boundaries
on:246:County or Equivalent
on:247:State or Territory
off:250:Shaded Relief
on:258:Road Features
on:259:Road Names and Shields
on:261:Geographic Names
on:262:Projection and Grids

# 'Enable' specific layers. When enabling, all other layers are
# implicitly disabled
$ pdflayers -e 237,238,239,249,251,252,253,262 \
   -i VA_Alexandria_20220927_TM_geo.pdf \
   -o t.VA_Alexandria_20220927_TM_geo.pdf

# Check my work: success!
$ pdflayers -l t.VA_Alexandria_20220927_TM_geo.pdf
on:237:Map Collar
on:238:Map Elements
on:239:Map Frame
off:241:Federal Administrated Lands
off:242:Department of Defense
off:243:National Park Service
off:244:National Cemetery
off:245:Jurisdictional Boundaries
off:246:County or Equivalent
off:247:State or Territory
off:250:Shaded Relief
off:258:Road Features
off:259:Road Names and Shields
off:261:Geographic Names
on:262:Projection and Grids

While these results were encouraging, the question remained: would the updated files continue to behave like proper GeoPDFs? That is, would messing with the layers remove the geographic metadata that lets these PDF files work as interactive maps. I turned to Avenza Maps, an Android App, to answer this question.

Avenza is a map viewer that is powered by PDF files. When loaded with a GeoPDF, Avenza becomes location aware and can display a blue dot on the document at your current position. It can also report the precise coordinates of any spot on the map.

The moment of truth came when I loaded my modified PDF file. Would the blue dot show up? Would it remain geo aware?

It does! The two screenshots above are of the same map, the only difference being the visible layers. The second map has many layers turned off, optimizing it for viewing terrain. You can see contour lines in the first map, but the labels and other features make this harder to do.

One limitation of Avenza maps is that it has no ability to toggle PDF layers. Using my command line tool, I can now sidestep this issue. I can prepare maps using pdflayers and import them into Avenza for optimized use.

*My pdflayers command should really be packaged as a pip module. If this would be helpful, let me know in the comments.

Monday, January 30, 2023

Map Skills in the Hood

Last year, I realized I could brush up my map and compass skills by printing out a local topo map and using it to explore my neighborhood. I recently gave this a try and was richly rewarded.

Here was the first stop on my little adventure. One of the side streets near our home dead ends at a fence. It's hard to see, but on the other side of that fence is the Army Navy Golf Course with the clubhouse in the distance. I wondered if I could plot the location of this building on my map using my current location and terrain clues.

I grabbed my current location from the UTM/MGRS widget on my Garmin vĂ­voactive 4 watch.* Alternatively, I could have used the GPS Test app to learn my UTM coordinates from my phone.

I plotted these coordinates using the map's grid lines and a Map Tools Mini Corner Ruler. I noted the location as 'C'.

A quick glance at the compass told me that the clubhouse was due South from my position. Looking at the contour lines, I plotted point '1' in purple as my guess of the building's location.

Once at home, I used Google Maps to determine the latitude and longitude of the clubhouse. I plugged these values into Backcountry Navigator to reveal the equivalent UTM values. Finally, I recorded these coordinates in purple on my map by again using the Mini Corner Ruler.

Success! As you can see below, my guess of the building's location agrees with the actual value. I repeated this process a handful of times: plotting known values in black, guesses in purple and actual values in red.

Most of my guesses were fairly accurate. I ran into issues with guess '2' because I incorrectly converted miles to kilometers. Curse you imperial units!

*Alas, I didn't grab a photo of the actual coordinates at the fence-line. The coordinates in the picture are from another part of my stroll, but the concept is still the same.

Friday, January 27, 2023

The Goldilocks Problem, Backup Phone Edition

I've shared why I carry a backup phone, and the plan I use to power it. Now comes the fun part: talking about the device itself.

Here are three examples of what I consider top notch backup phones, but as you'll see one is clearly the winner. All three phones make use of the same old-school operating system which will be familiar to those of us who grew up on turn of the century Ericssons and Nokias. For everyone else, these phones will feel primitive. Remember: what they lack in pizzazz they more than make up for in functionality.

Finally, all three phones rely on 2G networks. One day, T-mobile & US Mobile may stop supporting this type of network. When that happens, these phones will be useless and I'll need to search out new devices. But until then, these devices rock. One just happens to rock more than the others.

The Slide: Too Little

The CR1102GD Slide, aka the M5 mini phone, is a remarkable device. It's easily one of the best phones I've ever owned. Weighing in at just over an ounce (31g!), having form factor of a credit card and costing $19.95, it's the just about the perfect minimalist communication tool.

The Slide has you covered for calls and text messages and includes extras like speakerphone, Bluetooth and a physical off switch to avoid power drain while not in use. If you pair it with a Micro USB to 3.5mm audio jack cable, you also get access to FM radio. You can even pair the device with your main cellphone, letting the Slide serve as a 'Bluetooth dialer.' This lets you handle this unassuming phone in public, while calls are routed through the $1,000 cell phone in you pocket.

The Slide isn't without its quirks. It doesn't pickup the current date & time from the cell network, so you have to set it yourself. And the screen and keyboard are obviously sub par. But still, as backup devices go, it's gold.

But, we can do better!

The A10: Too Much

The full metal construction and comparatively expansive color screen of the A10 1.77 HD mini phone makes it feel luxurious compared the Slide. The built in 3.5mm audio jack and support for capturing voice memos is a nice bonus, making it useful for on the go journaling.

Alas, the all metal construction is also the phone's Achilles' heel. The phone weighs in at a massive 66g (for context: my S22 Ultra, with case, weighs 300g), which I can't justify when there are lighter options out there. Like the Slide, it doesn't appear to pick up the date from the network either. Finally, for the life of me, I couldn't figure out how to get the battery removed to install an SD card.

Again, we can do better!

The Soyes 7S+: Just Right

The Soyes S7+, aka the PUSOKEI Mini Cell Phone, is a delightful step up from the Slide without the weight penalty of the A10. Weighing in at 36g, and being slightly thicker, yet narrower than the Slide, it maintains a tiny footprint. The phone adds a larger and more functional screen than the Slide, a very low resolution camera, flashlight and primitive Micro SD card support. And hallelujah, it gets its date and time from the network, so you don't need to set it manually.

The flat keyboard is handy for storage, but not as pleasant to use as the A10. The voice recorder of the A10 is probably more practical than the camera from S7+, but that hardly justifies the weight increase.

Overall, the Soeys S7+ is the way to go if you're looking for a backup phone that disappears in a murse, yet can be a lifesaver when you need it.

Thursday, January 26, 2023

Phone Plan Perfection: Picking a Backup Phone Service Provider

I like carrying a backup phone. While the phone handset gets all the glory, I realized that without a cell plan my little backup phone would be a lot less useful.

I've found US Mobile elegantly answers the question of which cell provider should I use. What makes them unique is their custom plans can be scaled way down to match my backup phone usage.

I currently pay US Mobile $2.00 a month for 75 minutes of talking, $1.50 a month for 50 text messages and $0.00 a month for data I don't use. Of course there are fees--there's always fees--but they are relatively minimal. For about $5.00 a month I can turn any random phone into a invaluable communication device.

Another wrinkle is that my backup phone requires a 2G network. US Mobile uses T-mobile as a network provider. While T-mobile has shut down their 3G networks, 2G continues to live on. This means that the quirky device I use as a backup phone (more on that soon), work seamlessly with US Mobile SIM cards.

I don't usually find myself heaping praise on cell phone companies, but US Mobile nails it this time around.

Wednesday, January 25, 2023

5 Reasons I Carry a Backup Phone

Here's my top five reasons I carry an itty bitty backup cell phone:

1. To find my 'main' phone. In that moment of panic when I realize my regular phone isn't in my pocket, I can use my backup phone to find out where it's located.

2. To let me communicate when my main phone gets lost, broken or separated from me. We ran into this over the summer when I handed my phone to my nephew to take pics. Shira and a few kids got separated from me and the rest of the little ones. Of course my nephew was in Shira's crew, safely holding my phone. My backup phone let us gracefully reconnect.

3. To access FM radio in an emergency. In a disaster, the humble FM radio can become a critical lifeline. That's why a radio is mentioned on Arlington County's build an emergency kit page as a 'basic emergency' item. Having the radio as part of my backup phone, rather than built into my primary phone, means that I can get info in a disaster while not running down the battery of my main device.

4. To avoid the distractions of a cell phone while still maintaining a safety net. I carry my backup phone on Shabbat, a day when I relish and recharge by being offline. By carrying my backup phone, I know I can still reach 911 or Shira in an emergency. The backup phone also disappears into a small zippered pocket on my running shorts; so it's ideal for taking on run when I'd like to go ultralight and leave my clunky phone behind.

5. To serve as a 2-Factor Authentication backup device. For critical accounts, like Google or Apple, I like being able to provide my backup phone's cell number as a yet another way to get a one time code delivered to me.

And here's 10 bonus reasons:

  1. To Test out SMS based Tasker routines.
  2. To avoid carrying a flashy phone in a high crime area.
  3. To avoid being a target for hackers.
  4. To experiment with low resolution photography.
  5. To hand to stranger so they can make a call.
  6. To share with someone so we can be temporarily in contact.
  7. To sneakily store names, numbers of other details in the phone's contact section.
  8. To access data stored on an SIM or SD card.
  9. For use as a four function calculator.
  10. To show kids what phones used to be like back in the day.

Tuesday, January 24, 2023

Review: A Woman of No Importance

One sign that I've been completely sucked into a story is when the plot starts to influence my IRL mood. This is where I found myself while listening to A Woman of No Importance: The Untold Story of the American Spy Who Helped Win World War II.

A double agent had infiltrate Virginia Hall's spy ring and now the authorities knew her identity and likeness. They were closing in and capture, torture and death were imminent. I was not pleased about this turn of events one bit and had a grumpy and sour mood to match. The fact that these events took place nearly 80 years ago as part of WWII was cold comfort.

These feelings are evidence of just how good the writing of A Woman of No Importance is, and how amazing Hall's life and accomplishments are.

My simplistic understanding of World War II is as American-centric as one would expect. To me, WWII is punctuated by Pearl Harbor, D-Day, the horrors of the Nazi Concentration Camps, and Hiroshima and Nagasaki. Hall's story greatly expanded this view by showing me the perspective of the war through a French person's eyes. From the toppling of France by Germany, to planting the seeds of a fragile resistance to--spoiler alert--an ultimate victory by the resistance and allied troops. Hall not just witnessed it all, but helped make it happen. This would be an impressive feat for anyone, but is even more remarkable when you consider her gender and status as an amputee would normally be an absolute deal-breakers for serving in an active war zone.

Hall's story is exactly why diversity is a feature to be nurtured, and not a bug to be endured. She demonstrated this not only by taking what appeared to be shortcomings, her gender and disability, and using them as strengths. But just as importantly, by showing that these were not her defining characteristics. Her street smarts, generosity of spirit, ability to be a quick study, thirst for making a difference and a boundless degree of fearlessness, all made her an ideal operator in the hazard filled world of spying and resistance building.

One difficult but poignant aspect of Virginia's story is how she keeps needing to prove herself over and over again. Her quality work for the State Department is consistently overlooked in favor of assuming her gender or disability should define and limit her. Perhaps the ultimate irony was when her request for promotion was brought to none other than FDR himself, a fellow citizen who had to overcome disability on a daily basis, and even he couldn't see her petition as worthy.

Perhaps that what makes Hall's story so compelling. She had to not only outwit and overcome the Vichy Police and Gestapo; she had to outwit and overcome her own employers and societal-expectations just to get into the fight. We should learn Virginia's story not just because she's a courageous and selfless role model, but because there are other 'Virginia Halls' out there that will thrive if just given the opportunity.

Wednesday, January 18, 2023

USGS Topo Maps from the Command Line

Below is a script that combines the USGS Product API and Google's Geolocation API to make a shell script to retrieve topo maps from the command line. 'Product' in this context refers to mapping products that the USGS makes freely available to the world.

This is a relatively easy script to write because the USGS API finds products given lat / long bounding box. Google Geolocation API will happily give you back just such a bounding box. This means you can seamlessly convert a place names into a searchable geographic location.

Here's the script in action. As you can see the USGS mapping products are treasure trove just waiting to be mined.

# Topo containing the White House

$ usgsassist -a topos -l "1600 Pennsylvania Ave, Washington DC" Washington West, DC,MD,VA 2019|2019-11-22|

# Topo maps for Chicago

$ usgsassist -a topos -l "Chicago, IL" Arlington Heights, IL 2021|2021-04-16| Berwyn, IL 2021|2021-04-23| Blue Island, IL 2021|2021-04-20| Chicago Loop OE E, IL 2021|2021-04-20| Chicago Loop, IL 2021|2021-04-20| Elmhurst, IL 2021|2021-04-20| Englewood, IL 2021|2021-04-20| Evanston, IL 2021|2021-04-16| Hinsdale, IL 2021|2021-04-20| Jackson Park, IL,IN 2021|2021-04-20| Lake Calumet, IL,IN 2021|2021-04-20| Palos Park, IL 2021|2021-04-20| Park Ridge, IL 2021|2021-04-16| River Forest, IL 2021|2021-04-20| Sag Bridge, IL 2021|2021-04-20|

# Topo maps for all of Virginia

# Peek at the start of list. $ usgsassist -a topos -l "Virginia, USA" | head -5 Abilene, VA|2022-09-14| Abingdon, VA|2022-09-16| Accomac, VA|2022-08-31| Achilles, VA|2022-09-09| Adams Grove, VA|2022-09-14| # Peek at the end $ usgsassist -a topos -l "Virginia, USA" | tail -5 Woodsboro, MD 2019|2019-11-22| Wye Mills, MD 2019|2019-12-02| Wyoming, DE 2019|2019-11-21| Yellow Spring, WV 2019|2019-12-12| Zaleski, OH 2019|2019-12-03| # the total number of maps $ usgsassist -a topos -l "Virginia, USA" | wc -l 1697

# Topo maps for a National Park

$ usgsassist -a topos -l "Rocky Mountain National Park, Colorado" Allenspark, CO|2022-03-25| Bowen Mountain, CO|2022-03-25| Chambers Lake, CO|2022-03-25| Clark Peak, CO|2022-03-25| Comanche Peak, CO|2022-03-25| Crystal Mountain, CO|2022-03-15| Estes Park, CO|2022-03-15| Fall River Pass, CO|2022-03-25| Glen Haven, CO|2022-03-25| Grand Lake, CO|2022-03-16| Isolation Peak, CO|2022-03-15| Longs Peak, CO|2022-03-25| McHenrys Peak, CO|2022-03-25| Mount Richthofen, CO|2022-03-25| Panorama Peak, CO|2022-03-25| Pingree Park, CO|2022-03-25| Raymond, CO|2022-03-25| Shadow Mountain, CO|2022-03-25| Trail Mountain, CO|2022-03-25| Trail Ridge, CO|2022-03-15|

# Historic maps of Boston published from 1910-39

$ usgsassist -a historic -l "Boston, MA" | grep '|19[123][0-9]' USGS 1:24000-scale Quadrangle for Blue Hills, MA 1936|1936-01-01| USGS 1:24000-scale Quadrangle for Cohasset, MA 1936|1936-01-01| USGS 1:24000-scale Quadrangle for Nantasket, MA 1936|1936-01-01| USGS 1:24000-scale Quadrangle for Norwood, MA 1936|1936-01-01| USGS 1:24000-scale Quadrangle for Scituate, MA 1935|1935-01-01| USGS 1:24000-scale Quadrangle for Weymouth, MA 1936|1936-01-01| USGS 1:62500-scale Quadrangle for Abington, MA 1920|1920-01-01| USGS 1:62500-scale Quadrangle for Dedham, MA 1919|1919-01-01| USGS 1:62500-scale Quadrangle for Duxbury, MA 1918|1918-01-01| USGS 1:62500-scale Quadrangle for Duxbury, MA 1918|1918-01-01|

# The 10 oldest maps of New York city

$ usgsassist -a historic -l "New York, NY" | sort -t'|' -k2 | head -20 USGS 1:62500-scale Quadrangle for Morristown, NJ 1888|1888-01-01| USGS 1:62500-scale Quadrangle for New Brunswick, NJ 1888|1888-01-01| USGS 1:62500-scale Quadrangle for Paterson, NJ 1888|1888-01-01| USGS 1:62500-scale Quadrangle for Plainfield, NJ 1888|1888-01-01| USGS 1:62500-scale Quadrangle for Sandy Hook, NJ 1888|1888-01-01| USGS 1:62500-scale Quadrangle for Brooklyn, NY 1889|1889-01-01| USGS 1:62500-scale Quadrangle for Harlem, NJ 1891|1891-01-01| USGS 1:62500-scale Quadrangle for Brooklyn, NY 1891|1891-01-01| USGS 1:62500-scale Quadrangle for Paterson, NJ 1892|1892-01-01| USGS 1:62500-scale Quadrangle for New Brunswick, NJ 1893|1893-01-01| USGS 1:62500-scale Quadrangle for New Brunswick, NJ 1893|1893-01-01| USGS 1:62500-scale Quadrangle for Plainfield, NJ 1893|1893-01-01| USGS 1:62500-scale Quadrangle for Plainfield, NJ 1893|1893-01-01| USGS 1:62500-scale Quadrangle for Sandy Hook, NJ 1893|1893-01-01| USGS 1:62500-scale Quadrangle for Sandy Hook, NJ 1893|1893-01-01| USGS 1:62500-scale Quadrangle for Morristown, NJ 1894|1894-01-01| USGS 1:62500-scale Quadrangle for Brooklyn, NY 1897|1897-01-01| USGS 1:62500-scale Quadrangle for Brooklyn, NY 1897|1897-01-01| USGS 1:62500-scale Quadrangle for Brooklyn, NY 1897|1897-01-01| USGS 1:62500-scale Quadrangle for Harlem, NY 1897|1897-01-01|

# Historic maps of Rochester from the 1800's

$ usgsassist -a historic -l "Rochester, NY" -v | jq -r '.items[] | select(.publicationDate | test("^18")) | .title' USGS 1:62500-scale Quadrangle for Ontario Beach, NY 1895 USGS 1:62500-scale Quadrangle for Ontario Beach, NY 1899 USGS 1:62500-scale Quadrangle for Ontario Beach, NY 1899 USGS 1:62500-scale Quadrangle for Ontario Beach, NY 1899 USGS 1:62500-scale Quadrangle for Rochester, NY 1895 USGS 1:62500-scale Quadrangle for Rochester, NY 1898 USGS 1:62500-scale Quadrangle for Rochester, NY 1898 USGS 1:62500-scale Quadrangle for Rochester, NY 1898 USGS 1:62500-scale Quadrangle for Rochester, NY 1898 USGS 1:62500-scale Quadrangle for Rochester, NY 1898

# An preview image of these maps

$ usgsassist -a historic -l "Rochester, NY" -v | jq -r '.items[] | select(.publicationDate | test("^18")) | .previewGraphicURL' # Now grab these images and make a contact sheet $ usgsassist -a historic -l "Rochester, NY" -v | \ jq -r '.items[] | select(.publicationDate | test("^18")) | .previewGraphicURL' | \ while read url ; do wget -q $url ; done $ montage *_tn.jpg contact.jpg

Fun, right? Here's the script. Happy Hacking!


## Tools to work with the USGS


if [ -f  $HOME/.config/usgsassist/config ] ; then
  . $HOME/.config/usgsassist/config
  echo "Missing: ~/.config/usgassist/config file"

usage() {
  cmd=$(basename $0)
  echo "Usage: $cmd -a topos -l location "
  echo "Usage: $cmd -a historic -l location "
  echo "Usage: $cmd -a geobox -l location"

  exit 1

summarize_json() {
  if [ "$verbose" = "yes" ] ; then
    jq '.items[]' | sed 's/^{/,{/' | sed '1s/^,//'
    jq -r '.items[] | .title + "|" + .publicationDate + "|" + .downloadURL' | sed -E  's/^(USGS )?US Topo 7.5-minute map for //'

get() {
  dataset="$1" ; shift
  bb="$1" ; shift
  offset="$1" ; shift

  curl -s -G \
       --data-urlencode datasets="$dataset" \
       --data-urlencode bbox="$bb" \
       --data-urlencode offset="$offset" \
       --data-urlencode max="$PER_PAGE" \ | tee $HOME/.usgs.last | summarize_json

all() {
  dataset="$1" ; shift
  bb="$1" ; shift

  if [ "$verbose" = "yes" ] ; then
    echo '{ "items" : [ '

  get "$dataset" "$bb" 0

  total=$(cat $HOME/.usgs.last | jq .total)
  remaining=$((total - PER_PAGE))

  if [ $remaining -gt 0 ] ; then
    pages=$(((remaining / PER_PAGE) + 1))
    for page in $(seq 1 $pages) ; do
      offset=$((page * PER_PAGE))

      if [ "$verbose" = "yes" ] ; then
        echo ","

      get "$dataset" "$bb" "$offset"

  if [ "$verbose" = "yes" ] ; then
    echo '] }'

while getopts ":a:l:hv" o; do
  case "$o" in
    a) action=$OPTARG ;;
    l) loc=$OPTARG ;;
    v) verbose=yes ;;
    *|h) usage ;;

case "$action" in

    if [ -z "$loc" ] ; then
      echo "Missing -l location"
    bb=$(usgsassist -a geobox -l "$loc")

    all "US Topo Current" "$bb"
    if [ -z "$loc" ] ; then
      echo "Missing -l location"
    bb=$(usgsassist -a geobox -l "$loc")

    all "Historical Topographic Maps" "$bb"

    if [ -z "$loc" ] ; then
      echo "Missing -l address"

    curl -s -G "" \
         --data-urlencode address="$loc" \
         -d key=$GOOGLE_MAPS_API_KEY  > $bounds
    if [ "$verbose" = "yes" ] ; then
      cat $bounds
      ne_lat=$(cat $bounds | jq '.results[0]')
      ne_lng=$(cat $bounds | jq '.results[0].geometry.viewport.northeast.lng')
      sw_lat=$(cat $bounds | jq '.results[0]')
      sw_lng=$(cat $bounds | jq '.results[0].geometry.viewport.southwest.lng')
      echo "$ne_lng,$ne_lat,$sw_lng,$sw_lat"

  *) usage ;;

Friday, January 13, 2023

Digital Redneck Engineering: a Trip Timer for the Mazda CX-5

We recently leased a Mazda CX-5. While it surprised us with it's comprehensive feature set (heads up display, baby!), it was missing one important aspect: a simple trip timer. This is nothing more than a stopwatch that starts when the car is turned on. The idea is that you can glance at it to see just how long you've been stuck in traffic, or just how quickly you got home from a daycare run.

Fortunately of all the features offered by a car, this is an easy one to add on. I know this because our Acura RDX had the same issue. Having built my own timer in the past, I had a good idea as to how to do this again.

First off, I visited and did a bit of shopping. I was looking for the components I'd need to create a simple programmable display. It didn't take long before I found the ESP32-S2 TFT Feather. This combined an ESP32 chip with a tiny, but sharp, display. It looked it would be perfect for a timer. I purchased one and patiently waited for it arrive.

When the board arrived I put Circuit Python 8 on it by following these instructions. The installation process went smoothly and I was able access the top level web page offered by the web workflow. Unfortunately, my luck stopped there and I wasn't able to access the Web REPL or file browser. While I wanted to try these features, I realized I didn't actually need them for this project so I gave up on trying to use the Web Workflow.

Starting with a simple Hello World example, I was able to create a file that implemented my timer. Here it is in all its glory:

## Shows how long the car has been turned on for

import time
import board
import terminalio
from adafruit_display_text import bitmap_label

scale = 5

started = time.monotonic();
text_area = bitmap_label.Label(terminalio.FONT, text="---", scale=scale)
text_area.x = 0
text_area.y = 60

def digit(value):
    return str(value) if value > 10 else "0" + str(value)

def time_string(started):
    now = time.monotonic()
    t = round(now - started)
    ss = t % 60
    mm = (t // 60) % 60
    hh = (t // (60*60)) % 60
    return digit(hh) + ":" + digit(mm) + ":" + digit(ss)

while True:
    text_area.text = time_string(started)

Deploying this code to the board is easy: I connected the ESP32 to my laptop via a USB-C cable, mounted it as a drive, and copied into the root directory. Copying the file into places causes the board to reboot and execute It's all very slick and coder friendly.

The trickiest part of the process was installing the various library dependencies that imported. I did this through trial and error. The board would run, and display an error message as to which library was missing. I'd copy the relevant library from the bundle I'd downloaded, and repeat the process.

Eventually my code ran without issue. I grabbed a cigarette lighter* to USB-A adapter, USB-C cable and some mounting putty and brought it out our car. I then quickly rigged up this demo:


I'm in awe of the Adafruit ESP32-S2-TFT. For $24, I've now own tiny computer that includes a brilliant color screen, a built in neopixel and WiFi support. Trivially, I'm able to program it over the air using Python, and with effort I could almost certainly turn it into a forth or scheme computer. My timer scratches only the surface of these capabilities. I'm not sure whether I'll build a v2.0 of my timer, or use the board in another project altogether. But I almost certainly need to do something with this device, it's amazing.

*Kids today must be baffled as to why the strange shaped power plug on a vehicle is a called a cigarette lighter. Have no fear, YouTube can explain. I have a vague memory of being a kid figuring out that this oddly shaped knob could pressed in and would eventually get red hot. Perfect for lighting up your cigarette. Crazy, right?

Wednesday, January 11, 2023

Accessing an SD Card on Windows from WSL2

Every so often I find that I want to access a Micro SD card from within Linux on my Windows WSL2 setup. I put the card in my laptop and am pleased when Windows announces the new D: drive is available. And then the confusion begins: how do I access this drive from a WSL2 Ubuntu session?

The C: drive is automatically available to me in /mnt/c; not so with D: and /mnt/d.

A Google search inevitably takes me this page which talks about using wsl --mount. While promising, this never works for me.

Often, by now, I've decided I didn't really need to access the SD card *that* badly and give up.

But, it turns out there's a trivial solution! The following mount command is all I need:

  sudo mount -t drvfs 'D:' /mnt/d

Once that command is issued, the SD card files are available under /mnt/d, and are accessible like any other files on Linux. This works for any drive letter, not just 'D'.

Notably, this isn't SD card specific. I just connected an Adafruit ESP32-S2 TFT Feather micro-controller to my laptop and Windows recognized it as a drive.

I issued the same mount command to make this visible to Linux:

$ ls /mnt/d
ls: cannot access '/mnt/d': No such device
$  sudo mount -t drvfs 'D:' /mnt/d
$ ls /mnt/d
'System Volume Information'   boot_out.txt   lib   settings.toml

And success! Now, if I can just remember to check the blog for this solution before I give up...

Thursday, January 05, 2023

Starting 2023 Off Right: Exploring Huntley Meadows

If there's a better way to start a year than with a New Years Day Hike, I surely haven't found it.

The weather cooperated and we had an awesome hike at Huntley Meadows with G, his Mom, Shira and our little one. Apparently enough time has passed since my last hike through Huntley Meadows that I'd forgotten just how delightful a location it is.

The well packed trails and boardwalk meant that Shira and my Sister-in-Law could bring strollers for the little ones, though G spent most of the hike walking.

We had some delightful flora and fauna sightings, including an impressive cluster (or was it a troop?) of what appeared to be turkey tail mushrooms. We spotted a number of Northern Pintail ducks. At first glance, these looked like 'normal' ducks, but thanks to my kid friendly binoculars I could see that these were far more handsome. My photos don't do these guys justice.

We caught sight of a bale of turtle lined up sunning themselves. While this may be common, it always makes me smile whenever I see it.

Finally, we came across a patch of red berries, which Google tells me were rose hips. While they were fun to photograph, I wish I'd collected some for drying and tea making. I've since learned that rose hips are typically collected after the first frost, so it's wild edible that would be perfect to search out this time of year.

Here's to a 2023 filled with family, adventure and discovery!