I finally got some photos of the Pork Room, which is strictly off-limits to Muslims and found in only a few select grocery stores here. While they had the typical pork products like pulverized meat stuffed into synthetic intestines and legs of pigs soaked in salt water, it was a bit unsettling to find products like Pop Tarts, Chewy Granola Bars, and Jiffy’s Corn Bread mix in there too.

Two weeks ago I went climbing in Tawayan. If you squint you can see me in the upper left corner being lame. The route was named, appropriately, Stone Pussy (6c/5.11b? Yeah, right). Thanks for the photo, Dylan.

No, it’s not the antithesis of the lower education system in much of the Arabic world. After writing my previous post, I was thinking about how powerful each “micro” instance actually is… I mean, what happens when my newly minted pages of beauty get linked from lifehacker or arstechnica or TUAW? Will the magical server in the cloud be able to handle the rush to download the woefully outdated ones and zeros?? And so I set forth to find out how my tiny little instance (costing me US $0.03, thus far) can handle the traffic.

I found a number of cool sites that automate the testing process for you. Yottaa is super fun (and it’s hosted in AWS itself!); it provides longitudinal monitoring, tells you which parts of your page may be slowing the load process, and compares several more benchmarks to other sites around the world. Great for pondering about all the geeky ways you can spend your time optimizing your site instead of struggling through that poorly worded research paper (*cough*). LoadImpact was were I did most of my stress testing; their free trial lets you launch 50 clients to attack your website, or about 49 more than typically visit mine. Amazon’s own CloudWatch service also lets you retrieve metrics for your server by typing incantations like the one below.

mon-get-stats CPUUtilization --start-time 2011-01-30T16:20:00.00Z 
--end-time 2011-01-30T19:20:00.00Z --period 300 
--statistics "Average,Minimum,Maximum" --namespace "AWS/EC2"
--dimensions "InstanceId=i-0a123456"

After an epic battle with the still buggy UI of Excel 2011 (yes, okay, I’ll use MATLAB next time), I finally managed to produce the following graphs:

Initial results were disastrous. With only 13 clients on the most complex page, load times were already above 10 seconds. In fact, with only 7 clients my server’s logs indicated out of memory errors, forcing instances of Apache to close. WordPress provides some great advantages with its dynamic production of webpages, but PHP + Apache can be a memory hog. This turns out to be a real problem on a server with only 613 MB of RAM. After 7 clients, we see memory swapping causing the disk op/s (IOPS) rapidly rise into the thousands (!), the CPU utilization follows due data starvation. As indicated by the network bandwidth, the whole operation maxes out at 10 clients when the CPU also hits 100% utilization. Amazon’s storage backend struggles mightily to keep up with all the swapping, but even with IOPS about 20 times higher than a 7200 RPM SATA drive, the latency really destroys the CPU. (In fact, I didn’t saturate the storage backend… the average queue length was only around 12 IOPS.) At 32 clients, load times exceeded 50 seconds and LoadImpact aborted the test. =P

The solution, of course, is caching! Fortunately, WordPress has no shortage of caching plugins available. I chose W3 Total Cache because the documentation was well written. With page, database, and object caching enabled, load times with just a single client were approximately halved. The test ran all the way to 50 clients without the server breaking a sweat. In fact, the limiting resource was the # of the Apache instance spawned; CPU utilization never exceeded 30%, and disk IOPS didn’t even register! Great success! Now if I do some tweaking to apache2.conf to increase # of Apache instances I can avoid reading more research papers…

Curiously the CloudWatch stats returned are for 1 minute intervals… I didn’t see this documented anywhere. For example, the NetworkOut metric returned the average number of Bytes in 1 minute, even though the measurement period was 5 minutes. Also, the measured bandwidth on the AWS side was a little higher than what LoadImpact returned.

apache2.conf:
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers       10
    MaxClients            50
    MaxRequestsPerChild   3000
</IfModule>

I think I’ve finally finished porting my old site over. There were several motivations behind this, but I was primarily attempting to clean up the layout and visual appearance a bit, and improve usability a lot. The previous site, which I built in iWeb, had extremely long and complicated URLs with hidden frames strewn everywhere. That resulted in problems with crawler indexing and slow load times. And, while the old site was visually flashy, I tried to make the content here much easier to actually read and use. I’ll flashy this up with Spry later, like I did with my research group’s site.

A bit about the architecture of this new site: it all started with discovering the Amazon Web Services free tier. I thought, “cool! I can get free hosting for a year!” Yeah sure, I’ve already prepaid for three years with HostGator, and they’re pretty cool people. But there’s something way cooler about having a virtual server in the cloud; and it’s a lot more practical, since you get root access, incredible I/O performance exceeding most RAIDs, and online snapshots of the entire filesystem. Computational performance has also far exceeded that on HostGator, which sometimes struggled just to load cPanel (at the time, I didn’t really care because I was just serving static pages).

Technical details: I built my own Ubuntu image to fit within the AWS 10GB storage limit using instructions from Scott Moser, an employee of Ubuntu tasked to make Ubuntu work on Amazon’s cloud (read his blog sometime, it’s fun!). The image is loaded on a “micro” virtual server somewhere in N. Virginia. On top of that, we’ve got our standard LAMP stack, with WordPress and a slew of plugins on the very top (check out this site on your mobile phone; one of the plugins makes it super pretty!). It’s amazing to see how far we’ve come with these publication systems… I still have private WordPress archives going back to 2002, and I should really dig out my MoveableType installation from my original Bondi Blue iMac and import those entries. Oh, and nearly every page on this site validates HTML5 (well, it used to until I installed a whole slew of plugins…).

Logitech Bluetooth MX 5500 Mouse & Keyboard
UAE: $198.40
USA: $139.99

Ikea Jansjo LED work lamp
UAE: $25.86
USA: $9.99

Apple Wireless Keyboard
UAE: $108.63
USA: $59.99

Invigilating the crap out of students?