Tuesday, October 30, 2007

Morning Linux Challange

What is this, Monday? I come and find that one of our dev wiki is down. I poke around and learn that the box is pingable but not sshable. Something is hosed.

I wander back to the machine room, where Phil and I find the physical hardware. The machine was running, but there was no video and the keyboard wasn't working. I hit the old reset button and watched the machine reboot. Just as it was showing me the prompt to log in, the screen went dark and the keyboard stopped responding. Crap.

Our first move was to swap out the keyboard, and reboot. That would have been too easy, and of course didn't fix the problem.

Here's what we did to actually debug the problem:

  • Boot into single user mode by following the instructions here
  • Once at a shell prompt, I changed into the startup scripts directory by doing: /etc/rc.d/rc3.d
  • We then listed every file that should be started in that directory by using command: ls S*
  • One at a time, we started up each service on the box by doing: ./S13portmap start.
  • As soon as we ran ./S56gpm start the console went dark the keyboard stopped responding. Aha - found it!
  • We ran: rm S56gpm and rebooted. Problem solved!

Apparently, a recent update of the Linux OS apparently installed a version of GPM which is incompatible with something. Either way, we don't use the console on this box, so ditching GPM was an easy decision.

Whew, back in business. OK Linux, what do you have for me next?!


  1. you got to love it when the problem revolves a package that allows you to use a mouse in a terminal. You know I thought the idea of using the terminal/console was to use the keyboard, not the mouse.

  2. I know!

    GPM was really handy and impressive...in 1999. Why it's still in the standard Linux distribution is kinda strange.

    Certainly the basic console is quite useful, so maybe still having mouse support on it is a good idea.