On the Path

JQuery scrollTo + HTML5 not working in IE

I had an issue today using the scrollTo JQuery library with html5. Everything worked perfectly on Firefox, Safari and Chrome, but wouldn't work at all on Internet Explorer (7 and 8). IE reported this error message:

"Object doesn't support this property or method"

After spending far too much time, I finally figured out the html5.js I was using to give support for html5 in IE was causing the conflict. Moving this to load after the JQuery and scrollTo libraries resolved the problems and I'm happy to report I'm back in business with scrollTo and html5 on IE. If you have the same problem, hopefully this will save you some time troubleshooting it.

Have a Relationship with your Customers

I have been a customer of Tivo for a long time; I first saw one at a friend's house in the fall of 1999, and bought one for myself that Christmas.  Over the years I have owned somewhere around 6 tivos... Upgrading when the series2, series3, and most recently the Tivo Premiere came out.

Of course at its heart, the Tivo is a computer with a hard drive, and in the past 12 years I've had a few technical issues.  This story starts last month when my Tivo Premiere died a horrible hard drive death and needed to be replaced, and ends with a lesson on the importance of having a relationship with your customers.

I wasn't looking forward to replacing the Tivo - besides the call to Comcast (which is an entirely separate blog post), I know from experience that every Tivo is a blank slate...  I have to 'teach' it everything about me again... the shows I record, the stuff it should suggest for me, and so on.

Once I got the box set up, I was greeted with a "Welcome to Tivo" message in the Tivo's "messages and settings" that politely told me what I needed to know to use my Tivo, as if I was a first-time user.  I was slightly annoyed at this...  I thought to myself "Tivo *knows* who I am - this Tivo is linked to a lifetime account.  Why not have a welcome message that says 'Here's your replacement Tivo - we hope Rudy at the service center was able to resolve your issue.'?

Of course, Tivo could easily do a lot better than that suggestion.  My Tivo is linked to an account where I can schedule shows on the Tivo website; Why not back up my preferences there, and automatically download them to the new tivo?  Why do I have to set up, yet again, things like "Record episodes of Phineas and Ferb off of DisneyHD"?  I'm not asking Tivo to back up tons of data - not the actual shows, just the 'metadata' about my shows, my thumbs up and down ratings, the premium channels I subscribe to, etc.  This can't be more than a few hundred kilobytes. (I would think they would be interested in that data too - wouldn't TV executives PAY to know how many people gave a TV show a thumbs up vs. a thumbs down rating?).

A week after I set up my replacement Tivo, I received a new message that says "You may have noticed extra recordings that you didn't request - these are Tivo Suggestions".  This reminded me how annoyed I was...  "Yes, I know you suggest shows to me - you've been doing it for ELEVEN YEARS...  DON'T YOU KNOW WHO I AM?!?"  I began to get the feeling that Tivo, as a company, doesn't care that I'm a loyal customer.  They should be sending me a message that says "Its been a week and we see you're recording your preferred shows again.  We're happy everything was set up properly with your cable company".

Another week went by and I got another message "Have you ever missed an episode of your favorite program?  Learn how to set up a Season Pass".  Sigh.  Perhaps you didn't notice, but I've already set up, like, 70 of them?  Remember me?  I was mildly annoyed at having to set everything up - but weeks later, Tivo's naive attempts at customer contact are simply reinforcing the negativity of the whole experience.

Now, I'm not expecting Tivo to greet me by name when I call their service department, or send me cookies at Christmas time, but they can do a lot better than this. As a software engineer, I know the things I'm suggesting are next to trivial to implement.  I'm also quite sure the Tivo engineers have a list of user stories like these sitting in an idea tracker somewhere...  The only reason Tivo, as a company, would not do these things is because they haven't been made a priority.

Thats right, Tivo, as a company, has not made it a priority to have a relationship with their customers.

As much as I love my Tivo, I think they survive because their biggest competition is from cable-proprietary DVRs... and we all know the track record of customer relationship management cable companies typically have.  Tivo should be thankful Apple didn't make the AppleTV a DVR... I don't think they could survive the competition.  If Apple made a DVR, I'm sure it would be sending me tweets: "Hey Dave - Neal DeGrasse Tyson is going to be on Letterman tonight - don't worry, I'll record it for you."

You know what would have been the ultimate experience?  A message on my tivo 2 months ago that said "We've noticed your Tivo is beginning to have issues with its hard drive.  We've backed your schedule data up and scheduled a hardware replacement under warranty.  call 1-800-get-tivo to schedule your return and replacement".  Upon plugging my new Tivo in, a message like "We've downloaded and rescheduled your shows and noticed you've inserted your cable card.  You now need to call 1-800-comcast, and give them this MAC ID and Card Pairing Code to have them reactivate your cable service".  Every piece of hardware and data is available to make that possible; all thats missing is software.


Is there a lesson here?  Yes!  We can trivially use the data we have on our customers to improve their experience, and make it seem like they have a more personal relationship with a big faceless company.  How can you improve your relationship with your customers?

Small Issue with the Asset Pipeline

This is a small complaint about Rails 3.1, but its one I'm surprised I'm able to make at all.

If you have been following the evolution of Rails, you know that years ago the framework adopted REST as one of its core principals (well, restful concepts - I'll leave it up to others to debate/defend the true purity of the implementation). One of the core ideas behind REST is the idea that a url represents a unique resource, and that the mimetype/filename extension is used to negotiate the format that gets delivered. for instance:

http://www.davebock.com/resume.html

and

http://www.davebock.com/resume.pdf

should be the same underlying *thing*; what gets transferred is a representation of the thing in the format requested... in this case, my resume, as either html or pdf.

Enter the new Rails Asset Pipeline, which is a fantastic piece of work in general. In diving in and playing with it for the first time, I was trying to wrap my head around some of the conventions. I generated a new rails project with v3.1rc4, and I hadn't added any of my own content yet. I was looking at how everything under assets directory seems to be served up under the /assets/ url without much respect for the directory structure underneath it. Out of the box, rails gives us several files that have default contents. For instance:

http://localhost:3000/assets/application.css

and

http://localhost:3000/assets/application.js

Compare that to my first example above. These are the same url, but with different extensions - different "content negotiators"... but they aren't serving up the same thing at all. One could argue that they are two things designed to work together, but I don't think thats enough - the principal is violated - these are not the same thing, just in different formats, these are really much different things - one being code in the Javascript language meant to be used by your web application, and the other being a css file that contains all of the view/layout of your app. the browser isn't saying, "Give me that same information, but in the .js format", the browser really is requesting a different asset.

I like the asset pipeline, and I like the convention of treating all of these things as "first-class citizens" in their own assets directory under /app instead of as "a bunch of stuff" that happens to live under the /public directory - but I don't think this first-class citizenry has gone far enough - why aren't these things further scoped by the /images, /javascripts, and /stylesheets directory when referenced via url?

Power of the Command Line

I was just doing some routine maintenance on our servers when I had to use some command-line judo so solve a small issue and thought I'd share it.  Not for the particular solution, but for the path of thought that led to the solution; that is, demonstrating the power of the unix/linux command shell.

On a server running CentOS 5.6 (a linux distribution), I ran the command to perform routine maintenance and got an error upgrading a package named "shadow-utils-4.0.17-18.el5_6.1.x86_64".  To make a long story short, I realized it was failing because a file it wanted to update was set to be 'immutable'...  that is, a security tool I ran modified a bunch of files that are routinely hijacked by viruses and made them unmodifiable.  This makes the machine a little more secure, but this also means the updater can't modify them either.

I found the unmodifiable file and using the 'chattr' command, made the file modifiable (with the intent to set it back to be unmodifiable later).  I ran the updater again, and it failed on yet another file.  I did the same to that file, tried to update, and ran into a third.

I stopped and thought to myself "there could be dozens of files I'll have to manually set and restore for this process... and it'll be prone to error... there has to be a better way".

I realized all three files I had to modify so far were in the /usr/sbin directory.  This makes sense - all the shadow utilities I was updating were tools for the system administrator to run.  So I made the assumption that all the files that were going to need to be set and reset were in this directory.

So I reset the files I had manually modified so far, and dumped the contents of the extended permissions to a file:

lsattr /usr/sbin > sbin-results.txt

That uses a feature of the shell called 'redirection'...  the '>' symbol says "take whatever this would normally dump to the screen, and dump it into a text file with this name instead".  So now I had a record of the settings I was going to have to restore after my update.

Next, I ran this command to reset all the permissions to a modifiable state:

sudo chattr -i /usr/sbin/*

'chattr' is the tool to 'change attributes', and '-i' means 'remove the immutable setting'

I was then able to run 'yum update' and have my update succeed.

At this point I had some work to do.  I could have manually gone through that text file, found every immutable file, and manually applied the immutable setting back...  but that might take a long time *and* be error-prone.  Is there a way I can have the computer do that for me?

First, lets use 'grep' to get a list of all the files that were originally set to be immutable:

grep '\-i\-' sbin-results.txt

and we get the output:

----i-------- /usr/sbin/irdadump
----i-------- /usr/sbin/pwunconv
----i-------- /usr/sbin/grpconv
----i-------- /usr/sbin/clockdiff
----i-------- /usr/sbin/useradd
----i-------- /usr/sbin/sys-unconfig
----i-------- /usr/sbin/tracepath6
----i-------- /usr/sbin/irdaping
----i-------- /usr/sbin/adduser
----i-------- /usr/sbin/grpck
----i-------- /usr/sbin/pwck
----i-------- /usr/sbin/usermod
----i-------- /usr/sbin/findchip
----i-------- /usr/sbin/irnetd
----i-------- /usr/sbin/groupdel
----i-------- /usr/sbin/irattach
----i-------- /usr/sbin/ping6
----i-------- /usr/sbin/chpasswd
----i-------- /usr/sbin/arping
----i-------- /usr/sbin/usernetctl
----i-------- /usr/sbin/pwconv
----i-------- /usr/sbin/dongle_attach
----i-------- /usr/sbin/tracepath
----i-------- /usr/sbin/groupmod
----i-------- /usr/sbin/grpunconv
----i-------- /usr/sbin/newusers
----i-------- /usr/sbin/userdel
----i-------- /usr/sbin/groupadd

looks good... thats not a huge list as I feared, but that is still going to be a lot of copy and pasting, and error prone to do by hand.  In order to get closer to automating this, I need to extract just the filenames from that list.  Here a tool called 'awk' is invaluable:

grep '\-i\-' sbin-results.txt | awk '{ print $2 }'

gives me the output:

/usr/sbin/irdadump
/usr/sbin/pwunconv
/usr/sbin/grpconv
/usr/sbin/clockdiff
/usr/sbin/useradd
/usr/sbin/sys-unconfig
/usr/sbin/tracepath6
/usr/sbin/irdaping
/usr/sbin/adduser
/usr/sbin/grpck
/usr/sbin/pwck
/usr/sbin/usermod
/usr/sbin/findchip
/usr/sbin/irnetd
/usr/sbin/groupdel
/usr/sbin/irattach
/usr/sbin/ping6
/usr/sbin/chpasswd
/usr/sbin/arping
/usr/sbin/usernetctl
/usr/sbin/pwconv
/usr/sbin/dongle_attach
/usr/sbin/tracepath
/usr/sbin/groupmod
/usr/sbin/grpunconv
/usr/sbin/newusers
/usr/sbin/userdel
/usr/sbin/groupadd


looking good...  notice the "|" symbol in the command - this is called the 'pipe' symbol, and with it, I can take the output of one command and feed it to the input of another.  the command above now means "take the file with the previous directory permissions in it, find me every line that contains a file set to be immutable, and print out just the filename".

We are getting very close now...  for every one of those files, we now need to run the command:

sudo chattr +i <filename>

so we can restore the immutable setting.  We can use the command "xargs" to do that, again, with the pipe command:

grep '\-i\-' sbin-results.txt | awk '{ print $2 }' | xargs -l sudo chattr +i


and voila!  All directory settings are restored.  One line, no manual mistakes, and it took less than a second to run.  that saved me enough time to write this blog entry.


This is such a useful mechanism I'm now going to turn it into some command line utilities.  Imagine running into this situation again and being able to say:

sudo unlock-immutable /usr/sbin
yum update
sudo relock-immutable /usr/sbin

and having that just work.  By passing in the directory, I could choose to reuse this on directories like 'usr/bin' should that be the cause of my pain next time.  Maybe thats my next blog entry, or maybe I'll just leave that as an exercise for the reader.

I'm hardly the first person to use grep, awk, and xargs in a command line - I do things like this a lot, and so do many sysadmins...  but I see far too many developers who don't think this way - they look for a button in their IDE, and when there isn't one, they spend far too much time and effort on manual, error-prone attempts.  Spend the time to learn a half dozen command line tools that can be chained this way... make the command line your IDE!

Programmers Who Launch

My dad is a Vietnam Vet. He doesn't talk much about his war experiences, but one thing he did tell me is that there are two types of soldiers. Those who can pull the trigger and those who can't. This was brought to mind today after meeting with a fellow business owner and hearing the situation he's found himself in with respect to his software project. The business owner is working with an intelligent, experienced developer. But, after several years and many, many $$, he still doesn't have an app that he can use. His business has seen $0 ROI on his extensive software investment, even after all this time.

This got me thinking that there are two kinds of programmers. Those who launch, and those who don't. As a business owner or software project sponsor, my advice to you is to always look for evidence that you have a software developer/team who launches. If possible, start looking for this evidence before you start the project. Does this person or team have a track record of launched/live projects. Do they have working software out in the field that you can look at in it's native environment?

If it's a new person/team, or if you've already started, look for this type of evidence with your project. Is working software being delivered to you on a regular basis (weekly/biweekly/monthly)? Or do you get an occasional dog an pony show, displaying the software to you on a screen, but not available for you and your staff to use, experiment with on your own, interact with in between demos.

You should know in fairly short order if you have the kind who launches or not. If it seems to be the kind who does not, don't wait years. Cut your losses and start looking for the kind who launch.

An Open Letter to Restaurant Owners

Dear Restaurant owner,

I want to eat at your restaurant, but I need to find you first. You need a website. And not just any website. It has to work on a mobile device. If not, you might as well not exist anymore.

In the past few months, I have been out with my friends several times, spontaneously decided to get a lunch, dinner, etc. and had a bad experience with a restaurant's website.

Today's typical dining experience starts with a bit of research on a phone, looking for a menu. I start with the maps app on my iPhone, search for something like "Indian" or "Italian" around my current location, find a restaurant, go to the website, and I'd greeted with a message like "you must install Flash version 9 or higher" if I'm lucky enough to find your website at all.

You know what the cool kids call that? EPIC FAIL.

Even if I can see the website, I'm more likely to find a picture of the owner's family as if they are slaving over a hot stove, the history of the restaurant starting with the owner's mother immigrating in 1977, or a stock photo of a slice of pizza. If there is a menu at all, its an out of date pdf file I'm struggling to view on my tiny screen.

If you care about your restaurant, you should want your customers to find you.

What information should be at their fingertips?

When visiting a restaurants website from my phone, I want:

1. location (including phone number)
2. hours
3. menu

Anything else is just noise on a mobile phone... If I'm planning a catered event, I'll do that from my computer.

But I bought a package deal...

Yes, I'm aware that ground-up website design can be expensive, and restaurants often fall for the "Website for $29.95 a month" marketing ploy, where you get a "billboard-at-a-url" and a scanned menu. You are losing out. You need a real website.

Here's why:

Customers can find what they are looking for

You wouldn't imagine not having a sign outside your restaurant - a website is the same thing to the person actively looking for you. Restaurants think nothing of printing thousands of folded menus and sticking them on every car in a parking lot or every door in a heighborhood. Your website can have a far greater reach than that, and with immediate impact.

you can have a CMS to keep the menu up to date

For most restaurants, the menu on the website is like a time capsule. Dishes come and go... prices change... why shouldn't your website reflect that? With a content management system behind your menu, you can log in to update prices, add specials, etc. Why shouldn't your website be able to tell people what the soup of the day is?

You can tailor your SEO strategy

Try this: Put the name of your city and the type of food your restaurant serves into Google. (If you are in a big city, try your zip code). Is your restaurant's website on the first page? It should be. If you put in the name of your restaurant, are you the first result? If you're not, the first result is probably something like a restaurant review - and you have no control over that! Your worst customer could be the first thing people learn about your restaurant.

With your own website, you can easily influence how search engines view you. That is a topic larger than this blog entry though - its called 'Search Engine Optimization' or 'Search Engine Marketing'.

Further, by having your website as a real website (as opposed to flash or pdf), all your dishes become keywords Google can index. Maybe I'll find you because I searched on "Vindaloo" or "Pad Thai". All those words are pure SEO gold.

You can target your market more effectively

Visitors to your website can be tracked more closely than cattle with tags in their ears. Are most of your visitors coming from a particular neighborhood? Maybe thats the place to target with your door hangers. Is there a neighborhood thats not visiting? Send them a coupon for a free cheesy bread.

Ok, so how do I do it?

I'm not trying to sell you anything here - in fact, our typical projects are much larger than restaurant websites. I just want a better dining experience, so I'm offering you some technical advice:

Use 'Open Technology' standards

Make sure your website firm designs with html/css/javascript. Anything that requires a browser plugin to use (flash, Java, and most likely pdf), are non-starters for your marketing purposes.

Make sure your website looks good on small devices

I'll spare you most of the technical details, but with the kinds of tool we have today, a good website design scales down to work on phones, and scales up to work on big screens. On big screens, sure - show me your family in the kitchen. On my cell phone, echo the design with logos, fonts, and colors, but just give me the menu, hours, and location.

Search Engines are your best marketing tool

You don't need to hire an expensive SEO consultant - in fact, I think that most SEO consultants are the equivalent of Internet Snake Oil. Instead, just make sure your design firm eats and breathes the advice in Google's SEO starter guide. From your business' perspective, you should read a book like Search Engine Optimization (SEO): An Hour a Day or Internet Marketing: An Hour a Day to figure out how you can best market your restaurant.

Thats it! I hope you find this advice useful. Remember... I want to eat your food, I just need to be able to find you when you're just down the street.

What is Crowdsourcing?

We’ve all see the online CAPTCHAs that make you type in a pair of garbled words to prove you’re not a spam-bot. What you might not know is that by doing this, you are participating in crowd sourcing, a way to gather small amounts of data from large numbers of people. In this case, many of those words are used to digitize old books from the Library of Congress. When a computer can't read the scanned text, it "crowd sources" the unknown word. Once two or more people enter the same word for the garbled image, the computer now knows what that scanned word is. It’s pretty amazing to think a few seconds of time from millions of people can create public works of real value. This is the value of “crowd sourcing”.

Can your business use crowd sourcing? It works well for things that are “hard” for a computer to figure out, but easy for humans. For example, “which of these pictures contain pasta dishes?”. Amazon has a service called 'Mechanical Turk' which farms out work like this to interested people in exchange for micro-payments. Say you're willing to pay a penny a picture to have your images categorized or tagged. Amazon’s service will match up your job with interested parties. Amazon gives you back the results just as if a computer had processed the images, but instead the work was done with human intellegence. The results can be integrated into your data, allowing your systems to act on the new information. In fact, we’ve used services like this to verify that profile photos on a social network were all 'G-rated', and to generate transcripts for video files.

Can your business make use of crowd sourcing?

Rails3 and FullCalendar

Fullcalendar is an impressive calendar widget for JQuery, showing you events on a month, week, or day view, and letting you drag them around to reschedule them.

It needed a good sample app showing how to use it in Rails 3, so I wrote one.

Rails3_fullcalendar is a rails application that tries to live up to the rails3 notion of restful ideals.  It does this thanks to the power of Adam Lassek's jquery.rest plugin, which handles all of the wrapping of the jquery.ajax method into create, read, update, and delete methods, and also deals with the complexity of the token's rails uses to avoid cross-site scripting attacks.

Download it from github, run the migrations, and fire up the app.  You'll see an empty calendar, and scaffolding for adding events.  When looking at the calendar view, you can drag events around to change their date and time.  Dig into the code to see how it works.

I'm going to need to prepare a longer rant regarding timezones, but fullcalendar does something nice - it respects timezones (as long as you send it the right time format in the json), and normalizes everything to the local timezone on display...  Now if I schedule something for 12:00 noon in Washington DC during Eartern Daylight Time, it'll end up on the server as 16:00:00 UTC on the server, and if viewed by someone in California will show up on their calendar at 9:00am.

I have some future plans for this demo - including having multiple schedules, and then showing how to overlap them on the same calendar, with the events from each schedule showing up in their own color.  With a view like this, someone can have a calendar 'command center' view, dragging around events from different people's schedules until conflicts are resolved.

RESTful Search

Last night we capped off a great weekend at RubyNation 2011 with a wrap-up dinner (thanks Dr. Nic!). I'm proud that so many CodeSherpas had a hand in putting together this conference over the years. Anyways, during the dinner a fellow attendee, Robert, asked me for my thoughts on RESTful search. "It's just the query string on the index, right?" That is a very interesting question, and as it turns out there's more than one way to do it...

We are all familiar with search and use Google every day. Perform a Google search and it gives you results. But "search" itself is a verb - an activity (bear with me if this seems a bit pedantic).

When using Google, we are really interested in web pages or sites related to our search term: a set of resources related to that term:

  term ("fast cars") -> search (submit to Google) 
    -> resource (a list of web pages related to fast cars)

We will use this as our guide to develop a RESTful search service. I will make use of some Rails conventions in my examples below, but the concepts here apply to any RESTful ideal.

Let's say I'm building a web site for a fantastic Mini dealership, and my site includes search for all cars on the lot. I will need to answer questions like:

Do you have any red & white Mini Coopers with superchargers?

Option 1: The query string!

  Yes this is very obvious.
    GET => /mini_cooper_inventory
  Returns my entire inventory
    GET => /mini_cooper_inventory?color="red"&color="white"
        &supercharger="true"
  Returns the portion of my inventory matching the given
  query parameters

Option 2: A filter resource.

  Aha! I have played a trick on you in Option 1. We were
  already using a filtered resource to see only those
  Coopers in our inventory. We excluded cars that have
  been already been sold, or those on order or in transit.

  To think of filters RESTfully:
    GET => /mini_coopers
  Returns all Mini Coopers that have anything to do with
  my dealership: ones that I have sold, ones that are in
  inventory, and ones on order.

  Because these are such commonly used resources, instead
  of doing a query every time I wanted to look at my
  inventory:
    GET => /mini_coopers?status="inventory"

  I have a dedicated another resource:
    GET => /mini_cooper_inventory

  In Rails you would perform the search in the #index
  method of this controller, and then narrow by any
  other query parameters. A note of interest, the
  mini_cooper_inventory is not backed by a model of
  it's own. REST is not CRUD, but that is a topic for
  another day.

Option 3: A search-results resource

  This is probably useful when you have a really
  complex search that spans several resources. Let's
  say the IRS (yikes!) is askng me for all details on
  all sales made by Duzzy Cheetham or his family in
  the past two years. RESTfully speaking, I'm potentially
  dealing with any of a number of resources in my system:
    mini_coopers
    customers
    sales

  I could use one of the previous 2 options, and search
  for sales matching certain critera:
    GET => /sales?customer_last_name=Cheetham
          &customer_last_name=Swindell
          &date_range=2009-04to2011-04
          &detailed_report=true

  But at this point they query string (Option 1) does not
  seem to be a good fit. I could create a search object:
    POST => /search
  But again, this is not really what I am after, unless
  I am giving users the ability to save searches. What I
  really want is some sort of customer report:
    GET => /customers/:duzzy_cheetham_id/customers_report
  I want to see all purchases made by Duzzy Cheetham and
  his family:
    GET => /all_purchases_made_by_duzzy_cheetham_and_his_family

  Neither of those requests are really reusable or useful
  outside of this one search. Let's try again.
    GET => /search_results/new
  This is the form for doing a complex search (familiar to
  those of you using Rails). Not bad. What's next?
    POST => /search_results
  This creates a new search results report that matches my
  search terms, including date range, customers, and
  whatever else I want to throw in. And if I want 2 types
  of searches, one that returns a list of results, and
  another that summarizes results in some type of report:
    POST => /search_results
    POST => /search_results_reports

  The name of the actual resource is up to you. The search
  submitted can even be given a name that allows users to
  save and re-run searches:
    POST => /search_results/:saved_search_id

Anyways, there are many options when it comes to organizing your applications to work RESTfully - for the way the web was designed. I hope this example gives some food for thought.

Beautiful Visualization

Last night we performed some upgrades and restarted a server with nearly a year of uptime.  This morning I was checking on the status of the server, and saw this graph.  It is a beautiful example of the kinds of things you can see with Munin.

 

You are looking at a graph from munin, showing the usage of memcache of a large Ruby on Rails application.

After the server restarted, the cache was empty.  As the server traffic picked up this morning, more and more database calls and page fragments were generated by users hitting the site.  They were tucked away in memcache so they could be served up faster to the next visitor.  As we get more traffic, this cache fills up, and the website actually gets faster.

The performance gains aren't the subject of this post though - I thought this was a beautiful representation of a day in the life of a server process.  This represents one day period, with time going from left to right, and the three lines representing the amount of data in bytes (green), the number of objects (whether that be objects from the database or pre-rendered pieces of html - red), and the number of connections our rails processes are making to the memcached server (blue).

What does this show me?  I can see that memcache is doing its job - that the rails app is putting data in and getting it out.  I can also see that the cache is filling up... based on the 'hockey stick' curve, I can see we have allocated enough memory to hold the data we are putting in there.  If we hadn't allocated enough memory, I would visually be able to see it 'hit the ceiling'.

I can also tell things about the system overall by comparing to other graphs from munin and other tools.  Comparing this to graphs of mysql queries, I can see the impact of using memcached on mysql (by the number of reduced database queries) By comparing it to processor utilization, I can see the amount that pre-rendering reduces processor load (since we don't spend as much time rendering).