Tuesday, December 01, 2020

Auto Packaging of Videos | ffmpeg To The Rescue

The challenge: find a way to programmatically add a title screen, watermark, and credit screen to a video. My first thought was to turn to ffmpeg; it's gotten me out of every other video processing jam before, surely it wouldn't fail me now.

My initial plan was to generate content using ImageMagick and then use ffmpeg to combine the various images and source video into a finished product. However, the more I learned about ffmpeg's filters the more I realized that I could do the content generation directly in ffmpeg.

To do this, I needed to grok two ffmpeg concepts. First, ffmpeg's complex_filter option allows chaining filters together to create interesting effects using a technique reminiscent of Unix pipes. One source of confusion: ffmpeg allows for writing the same expression in various ways, from verbose to cryptically terse. Consider these examples:

ffmpeg -i input -vf [in]yadif=0:0:0[middle];[middle]scale=iw/2:-1[out] output # 2 chains form, one filter per chain, chains linked by the [middle] pad
ffmpeg -i input -vf [in]yadif=0:0:0,scale=iw/2:-1[out] output                 # 1 chain form, with 2 filters in the chain, linking implied
ffmpeg -i input -vf yadif=0:0:0,scale=iw/2:-1  output                         # the input and output are implied without ambiguity

These all do the same thing, combining the yadif and scale filter. The first one explicitly names each stream ([in], [middle] and [out]) while the last example relies on implicit naming and behavior. This tersification extends to the filter definitions as well. These are equivalent expressions:


This ability to compose expressions with varying degrees of verbosity means that many one-liners I found on the web were initially hard to understand and even harder to imagine how they could be combined. Once I started thinking of filters and their arguments as chains of streams, and working with them using verbose notation, the problem was vastly simplified.

The other concept I needed to wrap my head around was that before I could filter a stream, I needed to have a corresponding input. In some cases, the input was obvious. For example, adding a watermark image and text to a video stream was relatively straightforward. One solution has the following shape:

  ffmpeg -i source.mp4 -i logo.png  \           [1] These are my inputs, 0 and 1 respectively
         -filter_complex "\
           [0:v][1:v]  overlay=.... [main]; \   [2] Overlay the first two streams, write the output to [main]
           [main] drawtext=... [main] \         [3] Add text to the main stream
         \" output.mp4

But what about adding a title screen before the main video? drawtext and overlay would let me add text and images to the stream, but what's the source of the stream in the first place? One solution: generate a stream using the oddly named virtual device lavfi. With this in mind, adding a title screen looks roughly like so:

  ffmpeg -i source.mp4 \        [1] Main video
         -f lavfi \
         -i "color=color=0xf2e4f2:\ [2] Generated input source
            duration=5:\                for our title screen
            rate=30" \
         -filter_complex "\
          [0:]v ... [main]             [3] Do something with the source video and send it to [main]
          [1:v] drawtext=...[pre]\     [4] Draw on the virtual screen, send it to 'pre'
          [pre][main] concat [final]\  [5] Combined our [pre] and [main] streams for a final result
         \" output.mp4

With a solid'ish grasp of filters and lavfi generated streams, I was ready to tackle the original problem. I created a shell script for packaging video and the final result was looking acceptable. But then I ran into another issue: what's the best way to parameterize the script?

If I passed in all the arguments on the command line, about 40 in total, using the script would be a pain. If I stored all the options in a config file, then scripting the command would become tricky. With a bit of reflection, I realized there was a low effort, high value solution to this problem. At the top of the script, I define sane defaults for all the variables. I then process command line arguments like so:

cp /dev/null $config

while [ -n "$1" ] ; do
  arg=$1 ; shift
  if [ -f "$arg" ] ; then                [1]
    echo "# $arg" >> $config
    cat $arg >> $config
  name=$(echo $arg | cut -d= -f1)
  if [ "$name" != "$arg" ]; then         [2]
     echo $arg >> $config

  echo >> $config

. $config  [3]

For each command line argument, I first check if the argument is an existing file. If it is, I add the contents of that file to the end of ~/.pkgvid.last. I then check if the argument has the shape variable=value, if it does, then I add this expression to the end of ~/.pkgvid.last. Once I've processed all the command line arguments, magic happens at [3] by sourcing ~/.pkgvid.last. This reads the variable definitions that were defined in config files and on the command line, and has them override defaults.

This sounds confusing, but in practice, it's delightful to use. Consider blog.config which has been setup to override defaults for blog related videos:

main_caption_text="Ben Simon"

pre_title_text="A BlogByBen Video"




I can then override these settings by using command line options. For example:

 for part in $(seq 1 10); do
   pkg.sh blog.config pre_title_text="Blogging Secrets. Part $part of 10"

This strategy lets me organize variables into a config file and judiciously overwrite them on the command line.

OK, that's more than enough theory. Let's see this in action. Below is a simple test video, followed by this same test video packaged up with the command: pkg.sh blog.config.

Below is the script that created this video. Hopefully this gives you a fresh appreciation for ffmpeg and an interesting example to play with.


## This script is used for packing up a video

# Default Variables
base_dir=$(dirname $0)



main_caption_text="Thanks for watching"

pre_title_text="Title Text"

post_title_text="Thanks for watching"

cp /dev/null $config

while [ -n "$1" ] ; do
  arg=$1 ; shift
  if [ -f "$arg" ] ; then
    echo "# $arg" >> $config
    cat $arg >> $config
  name=$(echo $arg | cut -d= -f1)
  if [ "$name" != "$arg" ]; then
     echo $arg >> $config

  echo >> $config

. $config

main_video_width=$($ffprobe -show_streams $main_video 2> /dev/null |grep ^width= | cut -d = -f 2)
main_video_height=$($ffprobe -show_streams $main_video 2> /dev/null |grep ^height= | cut -d = -f 2)

if [ "$debug" = "on" ]  ; then
   ffmpeg="echo $ffmpeg"

$ffmpeg -y  \
 -i $main_video \
  -i $logo  \
  -f lavfi -i color=color=$pre_bg_color:${main_video_width}x${main_video_height}:d=$pre_duration \
  -f lavfi -i color=color=$post_bg_color:${main_video_width}x${main_video_height}:d=$post_duration \
  -filter_complex "\
    [1:v] split [logo_a][logo_b] ; \
    [logo_a]scale=w=$main_image_w:h=$main_image_h[logo_a] ; \
    [logo_b]scale=w=$pre_image_w:h=$pre_image_h[logo_b] ; \
    [0:v][logo_a] overlay=${main_image_x}:${main_image_y} [main] ; \
    [main]drawtext=fontfile=$font_dir/$main_caption_font: \
          text='$main_caption_text': \
          x=$main_caption_x: y=$main_caption_y: \
          fontsize=$main_caption_font_size: \
          fontcolor=$main_caption_color: \
          shadowcolor=black@.8: shadowx=4: shadowy=4: \
          enable='between(t,$main_caption_start,$main_caption_end)'[main]; \
    [2:v]drawtext=fontfile=$font_dir/$pre_title_font: \
         text='$pre_title_text': \
         x='$pre_title_x': y='$pre_title_y': \
         fontsize=$pre_title_font_size: \
         fontcolor=$pre_title_color [pre]; \
    [pre][logo_b] overlay=x='$pre_image_x':y='$pre_image_y' [pre] ; \
    [3:v]drawtext=fontfile=$font_dir/$post_title_font: \
         x='$post_title_x': y='$post_title_y': \
        fontsize=$post_title_font_size: \
        fontcolor=$post_title_color [post]; \
    [pre][main][post]concat=n=3 \
  " \
  -vsync 2 \

Monday, November 30, 2020

Mason Neck State Park - Enjoying Kid Friendly Hiking with Fun Kids

Last week we hiked Mason Neck State Park with friends. It had been years since we'd been to Mason Neck and I forgot what a little gem it is. We tackled the Wilson Spring and Bayview Trail, making for about 2 miles of hiking. This was the perfect distance for the kids. The trail winds through forest and swamps and there's no big ups to tire out little legs.

Shortly after the hike began we came across a geocache. The kids searched for, and Aurora quickly found, it. It was nice to start our hike with a win!

Midway thorugh our hike we stopped at the beach on the bay to properly explore the area. We marveled at the massive snail shells, and watched what appeared to be a duck-hunter doing his thing further down the beach. Shia even built a little sand castle!

I saw a number of hikers carrying big lenses, which is promising. My guess is that the glass was to be used for bird watching. We didn't see any interesting specimens this trip, but I'm sure they are there.

At about 40 minutes away from DC, Mason Neck is the perfect combo of backwoods adventure and family friendly convenience. I'm psyched to go back.

Monday, November 23, 2020

Countering News Bias - A Simple Solution

Getting your news from a single source is a great way to lock yourself into an echo chamber. Clearly, the smart news diet is the diverse news diet. And yet, there's no denying that I nearly always turn to CNN when I want to quickly check the news.

To try to kick this habit, I've added a new 'News' bookmark to my browser's toolbar. Rather than directing me to one site, it runs this code:

  window.location = U[Math.floor(Math.random()*U.length)]})()

This bookmarklet, as hopefully any first year CS student can tell you, creates an array of sites and picks one at random to visit. As long as I hit the News button on my toolbar, I won't be sucked into one news source.

I figured a random URL bookmarklet could come in handy in other contexts, so I created a tool for generating them. Check it out at http://code.benjisimon.com/bookmarklets/random-url.php. Here's an example of me creating a goof-off bookmarklet that sends me to a random site for a quick laugh:

Are my worries about news bias put to bed? Not by a long shot. But I feel like I'm at least headed in the right direction.

Check out the code behind the bookmarklet generator on github.

Thursday, November 19, 2020

Killer Artwork

Weeks ago, I was running at nearby Theodore Roosevelt Island and came across these cool abstract sketches:

And here's another example from our recent Blue Ridge Mountains adventure:

Is this ancient indigenous art? A coded message from aliens? Teens messing around? Nope, they're bug trails. Specifically, the side effect of beetle activity. The phenomenon even has an appropriate name: Beetle Galleries. While beautiful, they're art with consequences: the beetles can (often? always?) kill the tree.

That's some high stakes art right there.

Wednesday, November 18, 2020

Building a Garmin Watch Widget | Part 3: The UI

Once I had my Sun Compass Widget calculating the correct azimuth for the Sun, I was left with one more challenge: rendering this information graphically. My plan was to generate a simplified compass dial and then plot the current location of the Sun on it. The first hurdle was to realize I had to stop using XML based layouts and program the UI directly using the Dc object. The second challenge was learning how to convert polar to rectangular coordinates. With these obstacles surmounted, the UI came together with ease:

The above UI is drawn using a this strategy:

 for(degrees = 0; degrees < 365; degrees++) {
   var color = i % 90 == 0 ? "red" : "gray";
   drawDot(degrees, color);
 var start = pol2rect(watchFaceWidth / 2, azimuth);
 var end   = pol2rect((watchFaceWidth / 2) - 5, azimuth);
 drawLine(start, end, "yellow");

In short: once you can think of a round watch dial as a polar coordinate space, working with it becomes a breeze.

Looking at the above UI, I realized that along with creating a compass I'd also started down the path of creating a solar clock. For example, when the Sun is due South, it's solar noon. To build on this idea, I added markings for sunrise and sunset.

In the above UI, the red dots represent cardinal directions, the yellow line represents the position of the sun and the green lines represent sunrise and sunset. You can tell this screenshot was taken about an hour after sunrise. You can also tell this was taken in the winter when the day is relatively short and the Sun will never be due East or West.

I built the above app for my watch, loaded it on my Vivoactive 4 and spent a week field testing it in the Blue Ridge Mountains.

Generally, the widget performed well. It loaded quickly and always displayed accurate information. The biggest shortcoming: the effort needed to derive my current direction. To figure this out, I had to either stop and rotate my body until the yellow mark lined up with the Sun, take off my watch and line the yellow mark up up with the Sun, or attempt to do this translation mentally. While in motion, none of those options were ideal. I wanted the watch to do the work, not me. So I added the following feature.

By default, the widget shows a North-is-up view:

Clicking the top watch button rotates through a series of variations. It shows the compass with the sun ahead, to the right, behind and to the left of me:

The idea is that I can re-orient the display to roughly match where the Sun is relative to me at that moment. For example if the Sun is roughly behind me, I can click through until the compass dial shows this, and then use the cardinal markings to learn my direction.  This view is sticky, so re-checking the Sun Compass a few minutes later doesn't require me clicking through the options again.

I've now got this version running on my watch and I psyched to field test it. Once I'm satisfied with the UI, I'll finish off this project by uploading it to the Garmin IQ Store so others can give the widget a try.

As always, check out the code for this project on github.

Monday, November 16, 2020

Blue Ridge Mountains Adventure - Day 4

I wanted to squeeze in one more hike before we called it a trip. In the name of efficiency, we opted to string together trails located within the Wintergreen Resort itself. We connected up the Old Appalachian Trail, Upper Shamokin Gorge Trail and Chestnut Springs Trail. Given the density of housing within the area and the range of clientele who visit the resort, I had my expectations for these trails set low. If we got out in the woods for another hour or so, that was going to be considered a win.

The Old Appalachian Trail started off delightful, and as expected, quite tame. However, things took a turn for the interesting when we started down the Upper Shamokin Gorge Trail. This was a fun trail with plenty of rock scrambling. I finished that trail quite impressed. The Chestnut Trail brought us back to reality as we made up way up from the gorge to civilization and our car.

We stopped at one of the overlooks for one more picture and I continued to be blown away by the scene in front of me. This place is truly gorgeous.

With our last trail logged, we officially called it a trip and made our way back to DC.

I couldn't have been happier with our choice of vacation spots. The weekday off-season rate we paid for our condo was incredibly low. And the range of hikes, from those just a few minute drive away, to an hour away, were all impressive. And I could get used to drinking hard cider every night.

With all that said, we can't really speak to how nice a resort Wintergreen is. We didn't engage with any of the amenities, activities or restaurants. Still, there's no arguing they've got an ideal location. I'd go back in a minute.

Friday, November 13, 2020

Blue Ridge Mountains Adventure - Day 3

Today we took a break from accumulating trail miles and played tourist in our home state of Virginia. The first stop: we headed an hour away to Natural Bridge State Park.

I'd heard about Natural Bridge a few years ago, but could never justify the 3+ hour drive to see the famous rock formation. Staying at Wintergreen Resort put us close enough that it was a no brainer.

So, did the bridge live up to the hype? It did. First off, the rock formation is much larger than the pictures I'd seen suggested. There's also a mile-length trail that runs under the bridge which made for a pleasant walk. Because we visisted off season and early in the day we had no crowds to contend with, which surely furthered the quality of the experience.

I'm still not entirely convinced that it's worth taking the 3+ hour drive on a weekend from Northern Virginia, but if you're anywhere in the area, it's a must see.

After Natural Bridge, we made our way to another natural wonder: the Devil's Marbleyard. While far more esoteric, this site definitely deserves your attention if you are in the Natural Bridge area. The Marbleyard is an epic boulder field crying out for your exploration. I'd seen pictures before, and yet, I was impressed as the rock yard came into view after hiking through serene forest. One can't help but wonder how the rocks formed. One hypothesis, which I'm rooting for, is that worms did it.

I hopped around on the boulders like a 6 year taking in a new playground. It was awesome. Before I knew it, it was time to descend the trail back to our car and head back to Wintergreen.

Back in the Wintergreen area, we squeezed in one more sight for the day: Crabtree Falls. I assumed the falls were going to be pretty and all, but after Natural Bridge and Devil's Marbleyard, I figured they would pale in comparison. Apparently I missed the fine print: the falls are the highest vertical-drop cascading waterfall east of the Mississippi River. In short, they were quite impressive and fit in perfectly with the day's theme of seeing the Wonders of Virginia. I can't believe we were in the area and almost missed seeing them.

After the falls, we made our way to the secluded Bryant's Cidery, where we had the place nearly to ourselves. Bryant specializes in dry ciders, which have the (to me) unintended consequence of containing notably more alcohol than their sweet brethren. The flight of ciders I drank were both tasty and quite disinhibiting.


Related Posts with Thumbnails