Last month, I attended O’Reilly’s Velocity Web Performance and Operations Conference and, without fail, the information I came away with reinforced existing VMIX initiatives and strategies, and inspired new ones.

Attending the Velocity Conference, I had three goals:

• To listen to relevant industry-leading speakers, regarding how to improve performance and operational efficiency, taking this information and disseminating it to my colleagues who couldn’t attend.
• To meet and discuss with peers from companies facing the same challenges and hurdles when it comes to performance and operations. Understanding their approaches and pitfalls while offering my own experiences.
• And, finally, being away from interruptions at work, to simply be inspired by the incredibly smart people around me to draft new strategies for increasing scale, efficiency and performance to bring back and implement.

It’s All About the Data

At Velocity this year, as well as the previous two, I became aware of underlying themes that went beyond specific technologies or tools.

The most important, which in some cases may seem obvious, is to make decisions based on data.

A lot of times, decisions are made based on theory, instinct, wishful thinking and other motivating factors.

Validating these factors with data collected from your applications, infrastructure and clients will allow you to make the most informed decision. The more data points you collect, the higher the odds are that your decisions will result in the kind of outcomes you think they should.

One important method of data collection that is overlooked at times is the human factor. No matter how much our hardware computing evolves, humans still decide which methods or strategies to implement in order to provide a complete solution. And the human brain is still the highest performing computer on the market. Going to a conference and interacting with peers and colleagues on operations and web performance allows us to complete the system.

Learning and Implementing

As a result of attending the Velocity Conferences over the past three years, we have been able to implement the following technologies and optimizations, which have greatly impacted our efficiency at VMIX.

Velocity 2008

I met Brian Moon and Alan Kasindorf (dormando) and, after discussing file-system optimization, proceeded to implement some file-system tuning which, in turn, helped increase our Apache Web Servers running mod_disk_cache efficiency by 3 times.

Velocity 2009

After several BOF (Birds of a Feather) sessions, and speaking with various attendees, I was convinced to give Puppet (Configuration Management Framework) a try, along with Cobbler for rapid Linux installs. We’d previously used CFengine and Standalone Kickstart and felt they were not necessarily the right tools for our operation. After proof of concept (POC), we deployed Puppet and Cobbler and are now able to build and control systems at a quicker rate, with more management and less complication.

Velocity 2010

I attended a talk by Leif Hedstrom on Apache Traffic Server and realized it was open sourced from Yahoo to the Apache Organization. I had known about the Traffic Server back in the Inktomi days (1990’s), as well as from a company I worked for that licensed the code. However, I hadn’t realized it was openly available. After hearing that Yahoo uses the Traffic Server for their own CDN network, I decided to give it a shot and plan on using it to possibly replacing our Squid servers at VMIX.

Currently, we are running a bit of a bake-off between Squid and Apache Traffic Server, and preliminary results for Traffic Server are so favorable that, by next month, Squid might be calamari.

In quick closing, hopefully O’Reilly and the Velocity Program Committee keeps up the good work and my list of things learned and implemented grows!

Phil Chen is Director of Systems Engineering at VMIX.

And each year that goes by, our online video platform usage grows at a pace that could give anyone an ulcer, especially if you don’t come up with a creative solution to scale.

One of the solutions VMIX implemented a little over a year ago was incorporating cloud-based storage for a segment of our infrastructure. After much R&D, we chose to partner with the Amazon S3 service. When we started incorporating S3 to help meet some of our storage needs, we were slightly concerned about the service’s reliability and performance. So our approach was to design a system that would leverage our classic storage model as a buffer or cache for short-term newer content, and utilize S3 for long-term older content.

One year later, I thought I would share our findings with actual data collected from Amazon S3, as well as from the VMIX platform.

The graph below shows three data points (data stored, data transferred in, and data transferred out) on a month-to-month basis. As you will see, we did not experience any limitations on the amount of data stored and the amount of data transferred in and out.

The graph below tracks how many individual objects VMIX stored on a month-to-month basis. Given S3’s object-store model, it was clear that having two buckets in S3 was more than enough to handle what we needed.

The graph below shows the number of transactions on a month-to-month basis. We found that every time we stored, deleted or retrieved an object the performance was very dependable.

In conclusion, with the right architecture, Amazon S3 is an excellent option to help you scale storage. Though performance isn’t perfect (in the past year we’ve seen three to four S3 outages lasting an average of 15 minutes each) we feel you can architect around that as we have.

During this initial period VMIX used Amazon for a segment of its storage needs only—we now feel confident increasing our use of S3 moving forward.

Virginia – ami-b945acd0
California – ami-498cdd0c

United Kingdom:

Ireland – ami-993d17ed

Asia Pacific:

Singapore – ami-45e29d17

Enjoy!

So while troubleshooting using Firebug, I realized that the clients server I had deployed on was using Apache 2.2.11 with Mod_Deflate enabled, a method commonly used to compress up to 70% of data transfered over the wire to speed page loads. I then had a flash back on reading an article about issues with compression of SWF’s, and upon further investigation of the clients Apache configuration file I saw SWF was not excluded from compression via Mod_Deflate.

Mod_Deflate was compressing SWF files sending them with chunked transfer encoding to the browser. It appears the last part of the chunk was being missed by the browser. By refreshing the browser cache it sometimes filled in the missing chunk, and displayed the application.

So I added SWF to the list of files not to compress via Mod_Deflate and the application worked perfectly everytime.

Below is the section I added the SWF exclusion:

BEFORE:

<Location />
# Insert filter
SetOutputFilter DEFLATE
 
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
 
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
 
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 
# Don't compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|mp?g)$ no-gzip dont-vary
 
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

AFTER:

 
<Location />
# Insert filter
SetOutputFilter DEFLATE
 
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
 
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
 
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 
# Don't compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|swf|mp?g)$ no-gzip dont-vary
 
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

*Remember Results May Vary

Methods of Transport:
SCP & CURL

Data File Size:
17 Megabytes

Data Center Bandwidth:
Gigabit (RoutesScience)

32 Bit CentOS

64 Bit CentOS

32 Bit CentOS

64 Bit CentOS

For my intents and purposes I have a base CentOS 5.3 AMI with only the bare essentials. I then use this snapshot method to create different class AMI’s for different server functions ie. (webserver, memcached server, etc)

First thing is first you are going to need to have an existing AMI created, you can read how to do so in my other blog article How to create an Amazon Elastic Compute Cloud EC2 Machine Image (AMI). Once you have this AMI created and you launch the instance the fun begins to modify it and then take a snapshot for a new AMI based on the original AMI.

So once you launch your instance off your base AMI, you simply connect to your Amazon EC2 instance and install all the software you need to serve your purpose as you would any other server. After you have completed all the software installs and customizations, you then can begin the snapshot process.

The first step is to create a directory and download the Amazon EC2 Tools

[root@ec2instance]$ mkdir /mnt/EC2TOOLS
[root@ec2instance]$ cd /mnt/EC2TOOLS
[root@ec2instance]$ wget http://www.philchen.com/wp-content/uploads/2009/05/ec2-ami-tools.zip
[root@ec2instance]$ unzip ec2-ami-tools.zip
[root@ec2instance]$ cd ec2-ami-tools-1.3-31780
[root@ec2instance]$ mv * ../
[root@ec2instance]$ cd ../
[root@ec2instance]$ rmdir ec2-ami-tools-1.3-31780/

Now scp over your AWS private and public certificate

/mnt/EC2TOOLS/pk-yourprivatekey.pem
/mnt/EC2TOOLS/cert-yourcertificate.pem

You want to setup your .bashrc to know where your tools are

[root@ec2instance]$ vi ~/.bashrc
export PATH=$PATH:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin
export EC2_HOME=/mnt/EC2TOOLS
 
[root@ec2instance]$ source ~/.bashrc

Now you will want to download the correct kernel modules:

If your creating a snapshot of a 32bit AMI and don’t have the proper xen kernel modules do the following:

[root@ec2instance]$ mkdir /mnt/MOD
[root@ec2instance]$ cd /mnt/MOD
[root@ec2instance]$ wget http://www.philchen.com/wp-content/uploads/2009/05/kernel-modules2616-xenu.tgz
[root@ec2instance]$ gunzip -c kernel-modules2616-xenu.tgz | tar -xvf -
[root@ec2instance]$ cd /mnt/MOD/lib/modules
[root@ec2instance]$ mv 2.6.16-xenU /lib/modules
[root@ec2instance]$ depmod -ae 2.6.16-xenU

If your creating a snapshot of a 64bit AMI and don’t have the proper xen kernel modules do the following:

[root@ec2instance]$ mkdir /mnt/MOD
[root@ec2instance]$ cd /mnt/MOD
[root@ec2instance]$ wget http://www.philchen.com/wp-content/uploads/2009/05/kernel-modules-261633-xenu.tgz
[root@ec2instance]$ gunzip -c kernel-modules-261633-xenu.tgz | tar -xvf -
[root@ec2instance]$ cd /mnt/MOD/
[root@ec2instance]$ mv 2.6.16.33-xenU /lib/modules
[root@ec2instance]$ depmod -ae 2.6.16.33-xenU

Now you will want to create a location to store the image files

[root@ec2instance]$ mkdir /mnt/EC2IMAGE

At this step you are ready to create the EC2 image files and upload them to S3

[root@ec2instance]$ cd /mnt/EC2TOOLS
[root@ec2instance]$ bin/ec2-bundle-vol -d /mnt/EC2IMAGE --privatekey pk-yourprivatekey.pem --cert cert-yourcertificate.pem -u amazonaccountnumber -r i386 or x86_64 -p imagename
[root@ec2instance]$ bin/ec2-upload-bundle -b yours3bucketname -m /mnt/EC2IMAGE/whatever.manifest.xml -a accesskeyid -s secretaccesskey

Now on your local machine that you start instances from register the new AWS AMI

[phil@client]$ /Users/phil/EC2/bin/ec2-register yourbucketname/thenameofthenewami.manifest.xml

At this point you are done, and ready to fire up your new AMI instance whenever you like with your changes. Note the below directories have been excluded during the bundle process so your certs and secret info will be excluded, but also note anything you installed in these directories will not be kept though as well.

Excluding:
/sys
/proc
/dev/pts
/proc/sys/fs/binfmt_misc
/dev
/media
/mnt
/proc
/sys
/mnt

Happy snapshotting your AMI