loopback0 – Douglas Gourlay's Blog Data Centers, Virtualization, and Cloud Computing

Request for improvements to RFC 2544

If you don't like synthetic astroturf, you'd plant real grass, right?

In March 1999 Scott Bradner from Harvard University and Jim McQuaid of NetScout got together and published RFC 2544 - "Benchmarking Methodology."  In the subsequent eleven years this informational RFC has been used to provide a baseline for testing many networking devices.  It is designed to provide consistency between vendors so an end-customer can make a more informed buying decision and have some idea of the performance and scalability characteristics of the products they are considering.

For many years this RFC was applied by testing companies to provide comparisons and contrasts between different networking vendors.  Recently though, a company who usually takes an 'elder statesman' role in the networking industry and takes pride in its public brand image wrote that this was 'synthetic testing' and was not in any way indicative of 'real world' performance results customers were likely to see.  This was published on their blogs, and then on comments made on NetworkWorld's web site by their employees renouncing the testing and trying to invalidate the good work of David Newman.

I have a simple question...

"In the last eleven years why didn't you write a better and more 'real world' benchmarking methodology if the one you blast as synthetic is really that deficient?"

I mean, let's be serious, you are a huge company, and have the resources.  You have lots of people who go to the IETF meetings and try to steer standards.  You have lots of customers and have no problem telling us that, so it can't be a lack of revenue.   Why not just help us all by writing a better test plan rather than proverbially taking your ball and going home?

As I close this little diatribe let me remind everyone of two fun little stories...

In 2006 Kanye West was up for 'Best Video Award' at the European MTV Music Awards.  He won in a smaller category, 'Best Hip Hop Artist' but failed to win the prestigious 'Best Video Award' losing to a  smaller production.  He stormed the stage and "lashed out in a tirade filled with expletives," West said he should have won the prize for his video "Touch the Sky," because it "cost a million dollars, Pam Anderson was in it, and I was jumping across canyons."

Apparently to the judges it didn't matter how much Kanye spent, or that he looked cool flying across canyons, they judged on value.

By contrast at the 2009 Academy Awards 'Slumdog Millionaire' won Best Picture, Best Direction, and six other Oscars.  As Danny Boyle and then Christian Colson took the stage to thank their teams and supporters their competitors stood up and cheered for their victory.  You never saw Ron Howard, Gus van Sant, or Sydney Pollack trash-talking the Academy for how they voted.

These guys are smart enough to know two things - One, you are measured by how you well you lose as much as by how you win.  Two, if you bad mouth the Academy how will they treat you next year?

Do you want your primary networking vendor to be more of a Kanye West or more of a Ron Howard?