Gotcha: Emacs on Mac OS: Too Many Files Open

Generally, the MacPorts version of Emacs works great on my Mac Mini. But every so often, I'd hit a Too many open files error.

The *Messages* buffer was little help as it just repeated what I already knew:

insert-directory: Opening process input file: Too many open files, /dev/null [2 times]

I'd attempt to close buffers or shut down projectile projects, but there was nothing I could reliably do to recover from this error. Ultimately, I had to do the unthinkable and restart emacs.

I tried the obvious fix: telling my Mac to allow processes to open more files. I followed this recipe:

$ sudo cat /Library/LaunchDaemons/limit.maxfiles.plist
<?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
          "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>Label</key>
      <string>limit.maxfiles</string>
      <key>ProgramArguments</key>
      <array>
        <string>launchctl</string>
        <string>limit</string>
        <string>maxfiles</string>
        <string>64000</string>
        <string>20480</string>
      </array>
      <key>RunAtLoad</key>
      <true/>
      <key>ServiceIPC</key>
      <false/>
    </dict>
  </plist>

$ sysctl kern.maxfiles
kern.maxfiles: 20480
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 64000

It didn't help. Perhaps I was setting the limit incorrectly for my particular version of the OS. Or maybe I was setting the value too high, and the system was reverting it to something smaller. Or maybe all was good at the OS level, and it was a bash ulimit problem.

I considered all of these scenarios, but no matter how I set the file limit or what I set the file limit too, emacs kept hitting a 1024 open files limit. I could easily confirm this with lsof:

$ ps auxww|grep Emacs
ben              15944   0.0  0.7 412688032 111280   ??  S    Mon06AM  71:07.20 /Applications/MacPorts/Emacs.app/Contents/MacOS/Emacs
$ lsof -p 15944 | wc -l
    1618

After much frustration and searching, I finally stumbled on this reddit thread where a fellow emacs user complained about the 1024 max file limit:

Been trying to figure out a way to get out of this trap… I got a new Mac as a work laptop and I can’t seem to update the file descriptors. I updated it for the system but whenever eMacs is opened, ulimit is still at 1024.

Thankfully, there was a helpful reply:

I hit this all the time, as I work on a large monorepo with lsp-mode (and sometimes treemacs, which also watches stuff).

Whenever it happens I run M-x file-notify-rm-all-watches and things go back to normal for a while.

Last time I looked into this, you could not work around it by adjusting ulimits or literally anything. It's a core limitation of a low-level API used internally and is not configurable. I will try to find the previous discussion.

Aha! This made sense, as I've added lsp-mode to my workflow recently.

I was delighted to find I wasn't the only one having this problem, and more importantly, there was an easy work around.

Alas, when I tried to execute M-x file-notify-rm-all-watches I found that my version of emacs didn't have this function.

A quick Google search turned up this source file for filenotify.el. It does have file-notify-rm-all-watches defined as follows:

(defun file-notify-rm-all-watches ()
  "Remove all existing file notification watches from Emacs."
  (interactive)
  (maphash
   (lambda (key _value)
     (file-notify-rm-watch key))
   file-notify-descriptors))

I copied the code, untouched, into my emacs configuration. Next time I got the dreaded Too many open files error, I ran M-x file-notify-rm-all-watches and just like that, emacs was happy again. And so was I.