Keep in mind, I'm just brainstorming here -- if you have suggestions, insights and links to other resources, I'd love to hear them. Perhaps this is all wrong?
- Don't store anything on the local server itself. Consider using S3 for image/file uploads, and memcached for PHP sessions.
- Use a database such as CouchDB or SimpleDB that has replication and multiple instance support as a key part of its architecture.
- Leverage a queueing system like SQS and SNS to insure that any potentially long running process -- from sending e-mail to encoding a video -- happens outside the scope of the application itself.
With the above functionality in place, it seems like you could spin up arbitrary numbers of web servers and database servers, and the infrastructure would actually scale.
What am I missing?