Got a really detailed reason why Friendster is so damned slow today from one of our loyal readers:

To: Jason McCabe Calacanis
Sent: Friday, October 24, 2003 4:09 PM
Subject: Re: Social Software feature list

Friendster has a server farm (I believe 50 servers at Friendster.com) running Tomcats+MySQL that are load balanced through a traffic director.  Obviously if you have 50 servers being hit by 2+ millions users for query on a single MySQL DB server, it will kill MySQL (or for that matter any DBMS unless you're talking about a multi-million cluster system).  So (this is an educated guess) folks at Friendster are using the mater/slave MySQL configuration which was designed for high-availability. But they're using it to scaling performance.  So what that means is that every one of their server has an exact duplicate DB for ALL searchable member information.

So, if a user updates his/her profile, the update is made to a server and it's local instance of MySQL DB.  MySQL has a frequency of DB updates that gets propagate to other 49 servers.  Depending on the configuration and the load, you can get a fairly lengthy delay in another server getting the update.  And if the other member is let say logged in server #49 and your logged in server #0, and MySQL decides to update things sequentially, then you will have to wait a while to see the change (especially given that they're busy servicing queries).  And also if the Java coder decided to use some in-memory caching, then until the cache is invalidated, you might be waiting quite a while to get the update.

Bottom line, if you have a few $million for a 16 CPU SMP server from SUN or IBM, running Oracle or MS SQL, then you can get good performance within 30 days.  But if you want to support in the future 100+ million users, then you will need to spend much more $millions, so that doesn't scale well unless you have unlimited budget. 

Now if you have some really smart programmers who KNOW distributed transactional systems, then you can in maybe 3-6 months, redo their entire system with in-memory messaging between servers and extensive use of caching of user data.  So all you have to do is keep adding more server to support more users (maybe up to 100 million). Supporting a Web app that has 100 million hits a day (read only) is relatively easy, but supporting a Web app where you have 10 million updates that need to be readable by 100 million hits is REALLY hard.  Note, a site like Amazon updates their inventory far less frequently than what a site like Friendster does, and if you were to change even a fraction of Amazon's inventory every min, then you will have some serious load problems there too.  But of course, Amazon spent $100+ million on their IT setup.  Also, if you look at a site like TicketMaster.com, you will see a similar problem, where in a period of 15-30 mins, you get 10's of millions of request for 5,000 to 20,000 tix. Once again, it puts a massive load on a single DB.  I can tell you more about their problem some other time.