America’s Data Center Top 40 for QoS Implementations
a rainbow of fruit flavors for the network
Do you really use network quality of service within the glass house data center? This is a question I have been pondering pretty much all day- I get that on the costly WAN links we almost all use it. I also completely acknowledge that most engineers plan to use some of the QoS features within the data center, but what do we really use? Which features are necessary and which are just... well... extraneous?
Here's my thought- I think there are some QoS 'hit songs' implemented in the data center, and some that just are not that appropriate for this iMix.
Bad Boy Bad Boy, whatcha gonna queue....
No matter how many security products we surround ourselves with we all have the 'time out queue' for traffic that just isn't, well, good. It's for those pesky control plane DoS attacks, the misbehaving broadcast storm NIC, or that one application that just won't shut up.
Voices Carry
For some reason I have the 80's Til Tuesday song in my head by the same name- but most engineers I have talked with seem to set up, by default, a strict priority queue for voice traffic, then never,ever touch it again because it just works.
Marky Mark and the Funky Bunch
We all use markers, especially at the edge where we can have maximum visibility to what comes from a single host, and mark it up appropriately so that we don't have to continuously re-inspect and re-apply policy everywhere. Tag at the edge, queue in the core, shape on the WAN or point of massive congestion/cost.
(Don't) Drop it like it's Hot
No matter what religious SCSI encapsulation you love- FibreChannel, FibreChannel over Ethernet, iSCSI or even file based alternatives such as NFS or CIFS storage just performs better when is is not dropped. As much as I love technologies like Aspera for moving large files from A-to-B, until it is integrated in my host stack (which would monumentally rock for copying my iTunes library to S3 btw, hint...) the world will be dependent on TCP-guaranteed or PFC/BCN guaranteed storage delivery for a while - so we need a "lossless don't-drop-me-please" queue.
Shape ya Tailfeather
As we hit congestion and contention for bandwidth in a well designed data center this happens at the cost-center link, i.e. the edge router: this is where I apply large buffers, shapers, and policers to ensure that traffic can en-queue long enough to get onto the link, we can also adhere to Peter Lothberg's long-time rule of providing enough buffer to handle a round-trip-response of around the world or ~300ms. Shaping the previously marked-up traffic onto the link while trying to get all the right traffic on the link is the job of the core routers, usually not fixed switches.
That's it for now, any songs I should also think about including into the Data Center QoS Playlist?
dg
Additional comments powered by BackType
interaction
- Carrie on Interop 2010
- Chalon on Interop 2010
- nohype on Interop 2010
- SFoskett on Interop 2010
- Danny on Validating Some Power/Cooling Cost Assertions
texthibition
- @cxi let me know how Chicago goes, I have some new VM stuff would love to present up there if they have a good group! in reply to cxi 1 day ago
- pulling together another cool product/sw launch - been working on this one for last year. Excited... 6 days ago
- @vTrooper @vTrooper I need a few linecards in there too :) in reply to vTrooper 1 week ago
- @theosison Yes, Arista Networks :) in reply to theosison 1 week ago
- RT @yogeshmalik: SUN, CISCO and GOOGLE guys company Arista Networks show chassis switch, running 10,000 servers for $140k http://ht.ly/2em1s 1 week ago
- More updates...
Posting tweet...
coffee fund
default gateway
- A bet on BBCs and VM Mobility July 26, 2010
- Innovations - something I love in networking... July 21, 2010
- Why Data Centers Must Fundamentally Change -- Part 2.2 May 18, 2010
- Why Data Centers Must Fundamentally Change -- Part 2.1 May 17, 2010
- Why Data Center Networks Must Fundamentally Change May 10, 2010
August 1st, 2009 - 19:54
What about Justin Timberlakes…Bringing Sexy Back?
August 1st, 2009 - 21:23
My friend Phil Alexander, one of the most fun channel marketing personalities I know suggested – “You drop a BOM on me”
August 2nd, 2009 - 03:54
The key thought process to remember in the Data Center is that there is no “one size fits all” or in this case no one song that will always top the chart… The overall decision of what traffic is more important will always be debated, however a multitude of solutions would need to be implement at different levels to get the job done right the first time. If you already have the technology paid for then it might be time to bring someone in to help plan which songs should be on your personal data center play list, as well as how to implement them properly for best results.
August 2nd, 2009 - 10:35
You asked me to comment on QoS technology in the data center. Here is my viewpoint.
In a data center I would think you would overprovision like mad, because doing so is effectively free. I would recommend that real time traffic be separated from elastic traffic, either literally or by putting real time traffic in a priority queue, but I wouldn’t put any more effort into it than that. See
http://www.ietf.org/rfc/rfc5127.txt
5127 Aggregation of DiffServ Service Classes. K. Chan, J. Babiarz, F.
Baker. February 2008. (Format: TXT=43751 bytes) (Status:
INFORMATIONAL)
August 2nd, 2009 - 21:05
I would put even less effort into it than Fred. Watch like a hawk for dropped packets, and then decide what to do about it — either provide a bigger pipe or selectively apply QoS at the point of congestion. Otherwise QoS is going to make a choke point less visible, even though there may be a performance impact. On WAN links QoS is necessary because it’s not economical to provide as much bandwidth as everyone would like, but in the data center you want to discover the problem, not work around it.
August 2nd, 2009 - 21:31
For one customer I was playing with a switching QoS mode where I would mark packets with a particular DSCP value when link utilization went above 90% and then start a packet capture to a ring-buffer. If the link had drops (i.e. ran out of buffer) then I wanted to inject a packet into the mirrored stream to the ring-buffer and snapshot the buffer so you can get immediate forensics on what traffic type caused the drop.
I still think this would be useful some places…
August 2nd, 2009 - 21:50
I think there is value in that.
August 2nd, 2009 - 22:00
I guess my problem is that I think real time applications are fundamentally different than elastic – elastic will adapt while real time traffic just gets messed up. So we agree on the elastic traffic, but I would take some pains with voice, video, and sensor traffic.
On WAN links, loss is your friend. As the “Davenport” example in the attached web site shows, a WAN service that tries to prevent loss can wind up with multiple instances of the same packet in queue, with disastrous effects on both goodput and RTT. It would be better to drop a packet and bring TCP back into line than let that happen.
In the data center, virtually infinite bandwidth is virtually free, and your customer is gratified by millisecond-scale response time. To me, this has two implications. As you say, let’s not accept loss as a parameter – I would rather see a delay-based TCP like CalTech FAST than Reno/NewReno/Cubic to keep RTTs low and goodput high. I would also suggest that TCP’s initial burst be set to put the average file in flight in the first RTT. Usually this is set to 2*MSS, which means that a loss on the first transmission results in a timeout (bad in a data center); it has to be at least four to get fast-retransmit into play in the first RTT. But I think you want to use 8K segments (9K datagrams) if you can, and put the first five or six segments in flight to fully utilize that first burst. 6*8192=49K, which should serve the average file nicely.
August 2nd, 2009 - 22:01
It didn’t display the attached website
. ftp://ftpeng.cisco.com/fred/RTT/index.html
August 2nd, 2009 - 22:04
fixed it I think – embedded the link into ‘Davenport as well’
August 3rd, 2009 - 21:43
Still missing an “l” at the end of the URL: index.html, not index.htm
August 3rd, 2009 - 21:46
think I edited it up right now
check me on this one…