I've been trying to understand why one of my Linux servers keeps getting out of memory errors. While Googling around, I found this useful article which talks about measuring Linux memory usage.
It sums up the problem like so:
The moral of this story is that process memory usage on Linux is a complex matter; you can't just run ps and know what is going on. This is especially true when you deal with programs that create a lot of identical children processes, like Apache. ps might report that each Apache process uses 10 megabytes of memory, when the reality might be that the marginal cost of each Apache process is 1 megabyte of memory.
Although the problem of measuring memory on Linux is tricky, the article does give quite a bit of explanation and even introduced me to a new tool - pmap. It's a good place to start for understanding what makes your system go.
Update: Want to see the current top 10 processes that have the largest resident set size? Just run:
ps -e -o user,pid,%cpu,%mem,rss,cmd --sort=-rss | head