Last month, I attended O’Reilly’s Velocity Web Performance and Operations Conference and, without fail, the information I came away with reinforced existing VMIX initiatives and strategies, and inspired new ones.

Attending the Velocity Conference, I had three goals:

• To listen to relevant industry-leading speakers, regarding how to improve performance and operational efficiency, taking this information and disseminating it to my colleagues who couldn’t attend.
• To meet and discuss with peers from companies facing the same challenges and hurdles when it comes to performance and operations. Understanding their approaches and pitfalls while offering my own experiences.
• And, finally, being away from interruptions at work, to simply be inspired by the incredibly smart people around me to draft new strategies for increasing scale, efficiency and performance to bring back and implement.

It’s All About the Data

At Velocity this year, as well as the previous two, I became aware of underlying themes that went beyond specific technologies or tools.

The most important, which in some cases may seem obvious, is to make decisions based on data.

A lot of times, decisions are made based on theory, instinct, wishful thinking and other motivating factors.

Validating these factors with data collected from your applications, infrastructure and clients will allow you to make the most informed decision. The more data points you collect, the higher the odds are that your decisions will result in the kind of outcomes you think they should.

One important method of data collection that is overlooked at times is the human factor. No matter how much our hardware computing evolves, humans still decide which methods or strategies to implement in order to provide a complete solution. And the human brain is still the highest performing computer on the market. Going to a conference and interacting with peers and colleagues on operations and web performance allows us to complete the system.

Learning and Implementing

As a result of attending the Velocity Conferences over the past three years, we have been able to implement the following technologies and optimizations, which have greatly impacted our efficiency at VMIX.

Velocity 2008

I met Brian Moon and Alan Kasindorf (dormando) and, after discussing file-system optimization, proceeded to implement some file-system tuning which, in turn, helped increase our Apache Web Servers running mod_disk_cache efficiency by 3 times.

Velocity 2009

After several BOF (Birds of a Feather) sessions, and speaking with various attendees, I was convinced to give Puppet (Configuration Management Framework) a try, along with Cobbler for rapid Linux installs. We’d previously used CFengine and Standalone Kickstart and felt they were not necessarily the right tools for our operation. After proof of concept (POC), we deployed Puppet and Cobbler and are now able to build and control systems at a quicker rate, with more management and less complication.

Velocity 2010

I attended a talk by Leif Hedstrom on Apache Traffic Server and realized it was open sourced from Yahoo to the Apache Organization. I had known about the Traffic Server back in the Inktomi days (1990’s), as well as from a company I worked for that licensed the code. However, I hadn’t realized it was openly available. After hearing that Yahoo uses the Traffic Server for their own CDN network, I decided to give it a shot and plan on using it to possibly replacing our Squid servers at VMIX.

Currently, we are running a bit of a bake-off between Squid and Apache Traffic Server, and preliminary results for Traffic Server are so favorable that, by next month, Squid might be calamari.

In quick closing, hopefully O’Reilly and the Velocity Program Committee keeps up the good work and my list of things learned and implemented grows!

Phil Chen is Director of Systems Engineering at VMIX.

So while troubleshooting using Firebug, I realized that the clients server I had deployed on was using Apache 2.2.11 with Mod_Deflate enabled, a method commonly used to compress up to 70% of data transfered over the wire to speed page loads. I then had a flash back on reading an article about issues with compression of SWF’s, and upon further investigation of the clients Apache configuration file I saw SWF was not excluded from compression via Mod_Deflate.

Mod_Deflate was compressing SWF files sending them with chunked transfer encoding to the browser. It appears the last part of the chunk was being missed by the browser. By refreshing the browser cache it sometimes filled in the missing chunk, and displayed the application.

So I added SWF to the list of files not to compress via Mod_Deflate and the application worked perfectly everytime.

Below is the section I added the SWF exclusion:

BEFORE:

<Location />
# Insert filter
SetOutputFilter DEFLATE
 
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
 
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
 
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 
# Don't compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|mp?g)$ no-gzip dont-vary
 
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

AFTER:

 
<Location />
# Insert filter
SetOutputFilter DEFLATE
 
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
 
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
 
# MSIE masquerades as Netscape, but it is fine
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
 
# Don't compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
 \.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|swf|mp?g)$ no-gzip dont-vary
 
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
</Location>

Dell 1950’s Dual Core Intel Xeon 2.0 GHz
8 Gigs of RAM
Running 64bit Centos 5

Server Application Type:

API Caching Server

Basic Installation:

Install Apache 2.2.8
gunzip -c httpd-2.2.8.tar.gz | tar -xvf -
cd httpd-2.2.8
./configure --disable-status --enable-status=shared --enable-rewrite --enable-so --enable-proxy --enable-cache --enable-disk-cache
make
make install

Installed PHP

gunzip -c php-5.2.8.tar.gz | tar -xvf -
cd php-5.2.8
./configure --with-apxs2=/usr/local/apache2/bin/apxs  --enable-module=so --blah --blah --blah
make
make install

Below is the mod_cache portion of my apache vhost:

<IfModule mod_cache.c>
<IfModule mod_disk_cache.c>
CacheDefaultExpire 3600
CacheEnable disk /
CacheRoot "/opt/apicache/"
CacheDirLevels 2
CacheDirLength 1
CacheMaxFileSize 1000000
CacheMinFileSize 1
CacheIgnoreCacheControl On
CacheIgnoreNoLastMod On
CacheIgnoreQueryString Off
CacheIgnoreHeaders None
CacheLastModifiedFactor 0.1
CacheDefaultExpire 3600
CacheMaxExpire 86400
CacheStoreNoStore On
CacheStorePrivate On
</IfModule>
</IfModule>

The first performance decision I made was to use –enable-disk-cache after doing some research I found that contrary to what you would think, disk cache is faster then memory cache when it comes to Apache mod_cache and OS interaction. The reason why is when you use mod_mem_cache the process of reading a file into memory, basically copying its data into RAM and thus kernel buffer in order to deliver it is not optimal. When using mod_disk_cache Linux uses the sendfile API, which does not require the server to read the file before delivering it. The server identifies the file to deliver and the destination via the API, the OS then reads and delivers the file, so no read API or memory for the payload is required, and the OS can just use the file system cache. So the kernel acts as a buffer, increasing cache speed.

The second performance issues I saw was when I set my CacheDirLevels and CacheDirLength to high, load was skyrocketing. I found that CacheDirLevels 2 and CacheDirLength 1 was the optimal setting for about 50 Gigs of cache, lowering the amount of traversing needed on reads and writes.

The third performance issue I saw was when I met Brian Moon at the Velocity Conference in 2008 a very bright guy! I asked Brian how I could optimize the filesystem for Apache mod_disk_cache he instructed me to make some fstab changes only if I was using EXT3. Specifically to reflect the one entry below for my cache partition:

/dev/md1                /opt                    ext3   defaults,noatime,nodiratime,data=writeback 1 2

Setting the noatime effects removing a write for every read. Typically when a file is read the system updates the inode for the file with an access time so that the last access time is recorded, which basically entails a write to the file system. Unless you are running some sort of mirror you probably do not need the access time written.

Setting the nodiratime is the same as the noatime but for directories.
*Note 08/09/2009 it has been pointed out to me that noatime is a superset of nodiratime which is a subset. So if you use noatime you don’t need the entry for nodiratime

Setting data=writeback causes the non preserving of data ordering, the data to be written into the file system after its metadata has been committed to the journal which offers a higher throughput. Warning this setting could allow recently modified files to become corrupted in the event of an unexpected reboot or system crash.

If you look at the below graph you will see the sharp gain from doing the outlined tuning.

Happy tuning, fun stuff!

Recently I ran into a situation where an object (db conf) that was being called repeatedly via a web application ~100x more then any other cached item was being stored on one of four memcached servers. This item was stored as a flat file originally but had been implemented into a database which is how memcached came into play.

The excitement starts here, the four memcached servers were Dell 1950’s Dual Core Intel Xeon 2.0 GHz with 8 Gigs of RAM on each running Centos. Version of Memcached was 1.2.1. The particular object being cached ended up on a server as designed, however was being called by the web application a ~100x more then any other cached object and had a 5 minute expire. The particular memcached server with the object ended up taking a whopping 800+megabit/second in traffic, see the below graph:

*The reads and writes are reversed was the way cacti was set up.

The other server vitals at the time were:
CPU = 60%
Memcache Misses = 681 k/s
Memcache Hits = 1.16 M/s
Memcache Requests/sec (get&set) = 1.66 million
Server Load = 1.29

I haven’t ever had a single server reach passed 300 megabit in traffic first off so that was a new lesson of capacity and threshold. Also when memcached states it is not CPU intensive they really mean it. The server started crapping out meaning not serving requests properly at 780+megabit, so amazingly enough memcached can handle quite a lot of traffic/requests. Our switch port was saturated prior to memcached failing.

So lesson learned BEWARE what you cache when you are using memcached you don’t want to see the stats I did… In this case we removed that cached object which was a large DB config file and localized it. Hope this helps with giving the community a case study of memcached out in the wild.

Let’s outline the benefits first regarding the mod_status module and what it allows you to see (Documented on Apache’s Site):

1) Number of children serving requests
2) Number of idle children
3) Status of each child, the number of requests that child has performed and the total number of bytes served by child
4) Total number of accesses and byte count served
5) Time the server was started/restarted and the time it has been running for
6) Averages giving the number of requests per second, the number of bytes served per second and the average number of bytes per request
7) Current percentage CPU used by each child and in total by Apache
8 ) Current hosts and requests being processed

*All which can be beautifully graphed using Cacti btw for historical analysis.

I find mod_status most useful for tuning settings when using a prefork configuration, balancing the server resources and application memory usage.

So the downfall of using mod_status is upon every request Apache will perform two calls to gettimeofday. These calls are done so that the status report contains timing information. The theory is that this information is taxing in retrieval and calculation.

So as a limited but real world test using a production server I administer, I discovered only a small impact with or without mod_status enabled.

The test was on a Dell 1950 Dual Core Intel Xeon 2.0 GHz 8 Gigs of RAM Linux with Apache 2.0

I considered server load decrease as the marker for success, in that I compiled Apache with and without mod_status and looked for differences in cpu usage, memory usage, and load stats. The Apache server in question received on average 18 requests per second, mostly API application calls through PHP. And the test lasted a week for each configuration.

I saw no differences even in the slightest in server memory, nor server cpu utilization. I did notice server load was on average .29 with mod_status in extended mode compiled, and average load was .20 without mod_status compiled. So a savings of .09 on average load for a server over the course of 1 week of testing with an average 18 requests per second was seen. Regarding actual http requests I saw no speed increase or decrease.

So my conclusion, is every little bit counts however if you find that you need to tune your Apache servers often given sporadic changes in usage, you may want to enable mod_status with extended status on and reap the benefits of graphing that data using a tool like Cacti for the greater good of tuning other aspects of Apache for a more significant gain.

mkdir /var/log/logins
chown youruser:youruser /var/log/logins

Create the below script and place it some where permissions 755:

#!/bin/sh
#
#The Below Directory Path is where the script will keep track of logins 
BASE=/var/log/logins
#
# The two files below checked for a delta against each other 
HISTORY=${BASE}/history
CURRENT=${BASE}/current
#
# Failure Function
fail()
{
echo "Failed: $*"
exit 1
}
#
# Function to clean output from the last command
clean_last()
{
/usr/bin/last | sed '{
 /^reboot /d
/^$/d
/^wtmp begins /d
}'
}
MYGROUP=`id -gn`
MYIDENT=`id -un`
#
# Checking the env or error
[ -d ${BASE} ] || mkdir -p ${BASE}
[ -d ${BASE} ] || fail could not create ${BASE}
[ -G ${BASE} ] || fail ${BASE} not owned by ${MYGROUP}
[ -O ${BASE} ] || fail ${BASE} not owned by ${MYIDENT}
#
# Store current info
clean_last >${CURRENT}
# Is there a history file?
if [ -f ${HISTORY} ]
then
#
if ! `cmp --silent $CURRENT $HISTORY`
then
# Yes mail someone
#
diff $HISTORY $CURRENT |mail youremail@whatever.com -s "Login report"
fi
fi
#
# Make current history
#
mv ${CURRENT} ${HISTORY}
[ $? -eq 0 ] || fail mv ${CURRENT} ${HISTORY}
exit 0
 
#END OF SCRIPT

Create a crontab for your user to run the script:

*/5 * * * * /path/to/my/script/checklogin.sh

This should do it, gives a little more comfort, but I still recommend your typical safe guards IPTABLES, SNORT, etc… best practices.

vi ~/.bashrc

then activate your changes by sourcing it

source ~/.bashrc

1) To display the username, short version of hostname, and the current working directory (light blue)

export PS1='\[\033[1;34m\][\u@\h:\w]\$\[\033[0m\]'

Output:

[phil@server1:/usr/local/bin]$

2) To display username, short version of hostname, date, time, and the current working directory

export PS1='[\u@\h \d \A] \w \$ '

Output:

[phil@server1 Sat Jul 28 15:44] /usr/local/lib $

3) To display new line, username, short version hostname, base PTY, shell level, history number, new line, and the current working directory

export PS1='\n[\u@\h \l:$SHLVL:\!]\n$PWD\$ '

Output:

[phil@server1 0:1:123]
/usr/local/bin$

4) To display, username, short version hostname, number of jobs in the background a new line and working path

export PS1='\n[\u@\h jobs:\j]\n$PWD\$ '

Output:

[phil@server1 jobs:0]
/usr/local/bin$

server1# mkdir ~/.ssh

Step 2:

server1# cd ~/.ssh

Step 3:

server1# ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (“your_local_home”/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
18:6a:e3:78:ab:2d:0c:8e:f9:67:f7:30:32:44:77:34 phil@server1

Step 4:

server1# scp ~/.ssh/id_rsa.pub phil@server2.philchen.com:/home/phil/id_rsa.server1.pub

Step 5:

server1# ssh phil@server2.philchen.com
Password:

Step 6:

server2# mkdir .ssh

Step 7:

server2# chmod 700 .ssh

Step 8:

server2# cat id_rsa.server1.pub >> .ssh/authorized_keys

Step 9:

server2# chmod 644 .ssh/authorized_keys

Step 10:

server2# exit
server1# ssh phil@server2.philchen.com

*Note repeat steps 4-10 for all target servers you would like passwordless access from server1

* FYI Ensure your /home/user directory has the permission 755 also!

You should be all set!

Configuring MCrypt support for Mod_PHP can be a pain so hopefully this article will help!

The below was tested on a Centos 4.4 64 Bit server.

Package Dependencies:

libmcrypt-2.5.8.tar.gz
mhash-0.9.9
mcrypt-2.6.4
php-5.2.1.tar.gz

Installation Notes:

libmcrypt:

gunzip -c libmcrypt-2.5.8.tar.gz | tar xf -
cd libmcrypt-xxx
./configure --disable-posix-threads
make
make check  (note:  'make check' is optional)
make install
added /usr/local/lib to ld.so.conf
ldconfig or LD_LIBRARY_PATH=/usr/local/lib
(Below is important)
cd libmcrypt-2.5.8/libltdl
./configure --enable-ltdl-install
make
make install

mhash:

ldconfig
gunzip -c mhash-0.9.9.tar.gz | tar xf -
cd mhash-xxx
./configure
make
make check   (note:  'make check' is optional)
make install

mcrypt:

ldconfig
gunzip -c mcrypt-2.6.4.tar.gz | tar xf -
cd mcrypt
./configure
make
make check   (note:  'make check' is optional)
make install

PHP:

gunzip -c php-5.2.1.tar.gz | tar xf -
./configure --with-apxs2=/usr/sbin/apxs  --with-jpeg-dir=lib64
--BLAH --BLAH --BLAH --with-mcrypt=/path/to/mcrypt-2.6.4
(*NOTE the path of --with-mcrypt= install directory source code)
make
make install

This should get you up and running to use mcrypt!

#!/bin/sh
# SAMPLE BASIC INIT SCRIPT
#
# Below is the chkconfig syntax for auto startup at different run levels
# Note runlevel 1 2 and 3, 69 is the Start order and 68 is the Stop order
# Make sure these are unique by looking into /etc/rc.d/*
# Also below is the description which is necessary.
#
# chkconfig: 123 69 68
# description: Description of the Service
#
# Below is the source function library
#
. /etc/init.d/functions
#
if [ -f /etc/sysconfig/BLAH ]; then
. /etc/sysconfig/BLAH
fi
#
# Below is the Script Goodness controlling the service
#
case "$1" in
start)
echo -n "Start service BLAH"
/usr/sbin/BLAH start
;;
stop)
echo -n "Stop service BLAH"
/usr/sbin/BLAH stop
;;
restart)
echo -n "Restart service BLAH"
/usr/sbin/BLAH restart
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
;;
esac

Now to run chkconfig commands:

bash-3.00$ chkconfig BLAH --add

bash-3.00$ chkconfig BLAH on

The above will read the Init script and add and enable the script at the proper run level
specified in the script. Appending the start command on boot.

You could also do the below to be more specific in which run level to enable:

bash-3.00$  chkconfig --level 123 BLAH on (to turn on at run level 1 2 3)

or

bash-3.00$ chkconfig --level 123 BLAH off (to turn off at run level 1 2 3)

To list all run:

bash-3.00$ chkconfig --list

This should be a quick and dirty to get you going.