Last month, I attended O’Reilly’s Velocity Web Performance and Operations Conference and, without fail, the information I came away with reinforced existing VMIX initiatives and strategies, and inspired new ones.

Attending the Velocity Conference, I had three goals:

• To listen to relevant industry-leading speakers, regarding how to improve performance and operational efficiency, taking this information and disseminating it to my colleagues who couldn’t attend.
• To meet and discuss with peers from companies facing the same challenges and hurdles when it comes to performance and operations. Understanding their approaches and pitfalls while offering my own experiences.
• And, finally, being away from interruptions at work, to simply be inspired by the incredibly smart people around me to draft new strategies for increasing scale, efficiency and performance to bring back and implement.

It’s All About the Data

At Velocity this year, as well as the previous two, I became aware of underlying themes that went beyond specific technologies or tools.

The most important, which in some cases may seem obvious, is to make decisions based on data.

A lot of times, decisions are made based on theory, instinct, wishful thinking and other motivating factors.

Validating these factors with data collected from your applications, infrastructure and clients will allow you to make the most informed decision. The more data points you collect, the higher the odds are that your decisions will result in the kind of outcomes you think they should.

One important method of data collection that is overlooked at times is the human factor. No matter how much our hardware computing evolves, humans still decide which methods or strategies to implement in order to provide a complete solution. And the human brain is still the highest performing computer on the market. Going to a conference and interacting with peers and colleagues on operations and web performance allows us to complete the system.

Learning and Implementing

As a result of attending the Velocity Conferences over the past three years, we have been able to implement the following technologies and optimizations, which have greatly impacted our efficiency at VMIX.

Velocity 2008

I met Brian Moon and Alan Kasindorf (dormando) and, after discussing file-system optimization, proceeded to implement some file-system tuning which, in turn, helped increase our Apache Web Servers running mod_disk_cache efficiency by 3 times.

Velocity 2009

After several BOF (Birds of a Feather) sessions, and speaking with various attendees, I was convinced to give Puppet (Configuration Management Framework) a try, along with Cobbler for rapid Linux installs. We’d previously used CFengine and Standalone Kickstart and felt they were not necessarily the right tools for our operation. After proof of concept (POC), we deployed Puppet and Cobbler and are now able to build and control systems at a quicker rate, with more management and less complication.

Velocity 2010

I attended a talk by Leif Hedstrom on Apache Traffic Server and realized it was open sourced from Yahoo to the Apache Organization. I had known about the Traffic Server back in the Inktomi days (1990’s), as well as from a company I worked for that licensed the code. However, I hadn’t realized it was openly available. After hearing that Yahoo uses the Traffic Server for their own CDN network, I decided to give it a shot and plan on using it to possibly replacing our Squid servers at VMIX.

Currently, we are running a bit of a bake-off between Squid and Apache Traffic Server, and preliminary results for Traffic Server are so favorable that, by next month, Squid might be calamari.

In quick closing, hopefully O’Reilly and the Velocity Program Committee keeps up the good work and my list of things learned and implemented grows!

Phil Chen is Director of Systems Engineering at VMIX.

So while troubleshooting using Firebug, I realized that the clients server I had deployed on was using Apache 2.2.11 with Mod_Deflate enabled, a method commonly used to compress up to 70% of data transfered over the wire to speed page loads. I then had a flash back on reading an article about issues with compression of SWF’s, and upon further investigation of the clients Apache configuration file I saw SWF was not excluded from compression via Mod_Deflate.

Mod_Deflate was compressing SWF files sending them with chunked transfer encoding to the browser. It appears the last part of the chunk was being missed by the browser. By refreshing the browser cache it sometimes filled in the missing chunk, and displayed the application.

So I added SWF to the list of files not to compress via Mod_Deflate and the application worked perfectly everytime.

Below is the section I added the SWF exclusion:

BEFORE:

<Location />
# Insert filter
SetOutputFilter DEFLATE
 
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
 
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
 
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 
# Don't compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|mp?g)$ no-gzip dont-vary
 
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

AFTER:

 
<Location />
# Insert filter
SetOutputFilter DEFLATE
 
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
 
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
 
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 
# Don't compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|swf|mp?g)$ no-gzip dont-vary
 
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

The first script encrypts and pads data from a plaintext.txt file using Crypt::Blowfish into a ciphertext.txt file. The second script decrypts the cipher and writes it to a file.

blowfish.pl

#!/usr/bin/perl
use Crypt::Blowfish;
$key = '12345678123456781234567812345678';
$cipher = Crypt::Blowfish->new($key);
open (PLAINTEXT, "plaintext.txt");
while (read(PLAINTEXT, $block, 8)) {
$len  = length $block;
$size += $len;
# Add padding if necessary
$block .= "\000"x(8-$len) if $len < 8;
$ciphertext .= $cipher->encrypt($block);
}
# Record the size of the plaintext, so that the recipient
# knows how much padding to remove.
open (CIPHERTEXT, ">ciphertext.txt");
print CIPHERTEXT "$size\n";
print CIPHERTEXT $ciphertext;

decryptblowfish.pl

#!/usr/bin/perl
use Crypt::Blowfish;
$key = '12345678123456781234567812345678';
$cipher = Crypt::Blowfish->new($key);
open (CIPHERTEXT, "ciphertext.txt");
$size = <CIPHERTEXT>;
while (read(CIPHERTEXT, $ct, 8)) {
$pt .= $cipher->decrypt($ct);
}
# Write only $size bytes of the output; ignore padding.
open (PLAINTEXT, ">decrypted.txt");
print PLAINTEXT substr($pt, 0, $size);