Feb/104
Validating Some Power/Cooling Cost Assertions
What is the easiest way to account for space/power consumption of a network element?
Am making a spreadsheet comparing different products and looking at longer term costs, maintenance, power, cooling, etc. I felt that rather than scrubbing the DOE sites and trying to get power costs by state I would just use the national average, but then fell flat on that because I found negotiated rates could be much less than published tariff rates.
Then I stumbled upon what may be an easier solution to my quandary and one inline with what I see a lot of enterprises doing - call a hosting company. I haven't talked to too many enterprise customers that are not at least considering if not seriously considering using a hosting environment, or event a full-blown cloud deployment for some portion of their enterprise data center workloads. Why? - the main reason I keep hearing is that most enterprise customers cannot build big enough to achieve the same economy of scale as a Google, Microsoft, Facebook, etc. So they may as well lease space from a provider who can achieve a higher density, lower PUE, better delta-T, and handle the compliance tasks like SAS 70 Type-II (Switch, Equinix, Corelink, etc) and not to mention the IT assets put within the data center grow at a power/performance curve that usually breaks the facility they are housed within in 5-7 years, so who wants that on their books - better to let the provider manage/operate it.
In asking around I got to an average number of ~$155 per month per kilowatt consumed when in a hosted environment (ping, power, pipe). Does this seem inline to you or too high/low based on what you are seeing?
With this data you can then extrapolate Watts/10Gb port across several systems and you get variability from $92/year per 10GbE port up to $372/year per 10GbE port assuming $155/month per kilowatt. (I am eliminating my own companies products from this so as to avoid being a blatant advertorial...) Annualized hosting/power cost comes to $9,400 to $25,800.
I will be the first to admit there are HUGE differences in features, programmability, buffering, network segmentation, encapsulation methods, and Quality of Service Granularity between many of these platforms. Those that performed the best were usally more 'switch like' with smaller buffers, less features, and fixed function ASICs for the data path. Those at the top end of the spectrum were almost always products like Juniper's T640/T1600 and Cisco's CRS - extremely high function core routers with huge performance, buffers, shapers, policers, and probably most importantly a software upgradeable packet processing engine that allows incremental feature additions that execute in the data plane.
It's clearly not an apples-to-apples and don't want it to come across that way, my real question is - is using an average of US hosting pricing per kilowatt an effective way to get a model for opex cost/10Gb port or are there other models people would recommend? Am pretty open to anything right now provided it is accurate and neutrally intentioned.
dg
Feb/108
Request for improvements to RFC 2544
If you don't like synthetic astroturf, you'd plant real grass, right?
In March 1999 Scott Bradner from Harvard University and Jim McQuaid of NetScout got together and published RFC 2544 - "Benchmarking Methodology." In the subsequent eleven years this informational RFC has been used to provide a baseline for testing many networking devices. It is designed to provide consistency between vendors so an end-customer can make a more informed buying decision and have some idea of the performance and scalability characteristics of the products they are considering.
For many years this RFC was applied by testing companies to provide comparisons and contrasts between different networking vendors. Recently though, a company who usually takes an 'elder statesman' role in the networking industry and takes pride in its public brand image wrote that this was 'synthetic testing' and was not in any way indicative of 'real world' performance results customers were likely to see. This was published on their blogs, and then on comments made on NetworkWorld's web site by their employees renouncing the testing and trying to invalidate the good work of David Newman.
I have a simple question...
"In the last eleven years why didn't you write a better and more 'real world' benchmarking methodology if the one you blast as synthetic is really that deficient?"
I mean, let's be serious, you are a huge company, and have the resources. You have lots of people who go to the IETF meetings and try to steer standards. You have lots of customers and have no problem telling us that, so it can't be a lack of revenue. Why not just help us all by writing a better test plan rather than proverbially taking your ball and going home?
As I close this little diatribe let me remind everyone of two fun little stories...
In 2006 Kanye West was up for 'Best Video Award' at the European MTV Music Awards. He won in a smaller category, 'Best Hip Hop Artist' but failed to win the prestigious 'Best Video Award' losing to a smaller production. He stormed the stage and "lashed out in a tirade filled with expletives," West said he should have won the prize for his video "Touch the Sky," because it "cost a million dollars, Pam Anderson was in it, and I was jumping across canyons."
Apparently to the judges it didn't matter how much Kanye spent, or that he looked cool flying across canyons, they judged on value.
By contrast at the 2009 Academy Awards 'Slumdog Millionaire' won Best Picture, Best Direction, and six other Oscars. As Danny Boyle and then Christian Colson took the stage to thank their teams and supporters their competitors stood up and cheered for their victory. You never saw Ron Howard, Gus van Sant, or Sydney Pollack trash-talking the Academy for how they voted.
These guys are smart enough to know two things - One, you are measured by how you well you lose as much as by how you win. Two, if you bad mouth the Academy how will they treat you next year?
Do you want your primary networking vendor to be more of a Kanye West or more of a Ron Howard?
Feb/100
The Peril of Earn-Out based Mergers and Acquisitions
Sometimes the 'golden handcuffs' tarnish the other hands in the company...
Thirteen years ago I worked at a small technology consulting firm, headquartered out of Columbia South Carolina, named The Computer Group. TCG was acquired by IKON Office Solutions, the copier company.
IKON then went on a spending spree over the next year acquiring many technology companies, largely small to mid-size systems integrators in a fairly classic channel roll-up strategy to build footprint. This was quite smart given the complexity of digital copier and printing systems coming to market in the mid 1990's as well as the upside and opportunity to move into adjacent markets aside from their core paper distribution and copier sales and service divisions.
The problem came not in the vision, although I can say that some of the engineers did ask the question, "Ummm, we work for a copier company? Huh?"
When these companies were acquired almost every company was given an interesting choice - take some payment up front or take a bit smaller amount up front but if the company hits specific P&L targets there is a significant upside (between a 50% to 200% multiplier) that is paid if the targets are hit or exceeded.
Now, we all learned in Sales 101 that leveraged compensation plans tend to be highly motivational to the principals involved, and this case was no different. Each company was so singularly motivated to achieve their individual targets that the following happened:
1) Each group tried to keep their own identity
Each group almost always referred to themselves by their 'old' name, and never as the identity of their new employer. They never integrated their identities.
2) Cross-Charging became commonplace
Each group started cross-charging the others for any internal resource sharing. This made it more costly to use internal 'big company' resources than it often did to hire your 'own' resources because the cost-basis was higher because the other groups wanted to make a profit on each other.
3) No tools integration
Their was no mandate to centralize IT resources, standardize on toolsets, etc. Each company did things their way, and used any mandate to integrate as an opportunity to complain how 'corporate was slowing them down' and was an excuse for why any earn-out targets were not achieved.
4) Internal Competition
Since there was significant overlap in product/services from each group, and no common positioning or strong hand at the helm the groups would compete with each other for customer opportunities. There was no strategy or consistency in the pricing models between the divisions, and we looked like 20 companies to the end-customer, not one.
5) Haves/Have-Nots
Their was pretty significant disparity as well between the equity distributions between those acquired and those hired. This led to a haves/have-nots schism in the organization as well. Many of the new hires would often ask a question I have heard repeated in other organizations where earn-out M&A was tried, "Why am I working so hard to make them so rich?"
The net result? IKON is still a copier company, does a good job at it. But they closed the doors on the experiment of being a technology services company. It is not regarded as a 'win'.
We can find many reasons why it did not work and I am certain there are other very valid opinions and reasons on why this experiment/investment did not yield the expected business results. The reason I home in on the most though is the earn-out M&A structure created a culture clash, especially when combined with product/services overlap, and lack of strong leadership.
I have seen recently other companies follow a similar model, and have seen the undercurrents of similar challenges. Know any?
Feb/100
High Frequency Trading Webinar
microseconds: the difference between market makers and spectators
Am hosting a webinar on reducing the latency in high frequency trading environments this coming Wednesday. (which means, if you know me, that I am working furiously on PowerPoint slides, although I have been using Keynote more lately...)) HFT is pretty interesting to me as it is one of the markets I spend a lot of time focusing on at work and with folks from Solace Systems and Intel joining we will be chatting about how to reduce the latency, specifically on the back-end connections between feed handlers and trade logic and order execution systems. These are usually TCP based and there is a lot of room for improvement from the base stacks and generic NICs and messaging systems people may use.
We'll do a follow-up webinar later focusing on the market data feed handlers and scalable multicast as that is very important between the exchange and the feed handlers. If there are any other topics we should think about, let me know...
If you are interested in registering and joining a healthy chat please click here to register. If you can't make the live webcast, no worries - we will be archiving it and enabling the VOD to be watched in arrears.
Feb/105
Synthetic Testing of Automatic Transmissions
Imagine a car that was fixed speed and couldn't adapt from highway to city to parking lot. Just sayin'...
I am still a very proud alumnus of cisco Systems, but am also not bashful about areas I think the networking behemoth can improve. My main recommendation would be to get the business units working together to consistently solve customer problems - be a big company, but act like one company, not 20 or 50 or however many initiatives, boards, councils, or work streams there are. As a former commissioned combat arms officer I will state that some things are better run in a command and control environment if you want consistency and necessary if your customers want a consistent experience.
The recent 'data center' announcement of 10GBASE-T products really served to illustrate this better than I could ever explain.
According to Cisco's Frequently Asked Questions about their 10GBASE-T products for the Catalyst 6500 and 4500 they state the following:
Q. Is the 10GBASE-T line card on the Catalyst 4900M compatible with Gigabit Ethernet?
A. The 10GBASE-T line-card module on the Cisco Catalyst 4900M supports Gigabit Ethernet or 10 Gigabit Ethernet mode for each port group. The eight ports are divided into four port groups, and each port group can be configured to operate in either Gigabit Ethernet or 10 Gigabit Ethernet mode. All ports within the same port group must have the same mode. This allows customers an easy migration path from Gigabit Ethernet to 10 Gigabit Ethernet network connectivity.
Q. Can the 10GBASE-T line card on the Cisco Catalyst 6500 Series be connected to Gigabit Ethernet network adapters using auto-negotiation?
A. No, the 10GBASE-T line-card module will not support Gigabit Ethernet. It will support 10 Gigabit Ethernet network adapters only.
Can someone explain to me how the board, or council or whatever new-age org model is in charge decided that customers want a 10GBASE-T port that would support your existing cable plant but not interconnect your existing GbE attached servers on one switch (the Catalyst 6500 - hard coded into the PHY so it is not a field or software upgrade) and wanted a completely different behavior on the other (Catalyst 4900 that did the rather obvious feature of speed autonegotiation)?
Quick car analogy, since some people who have issues with 'synthetic tests' (apparently auto-negotiation tests are synthetic now too....)
Why build a network equivalent of a Bentley Continental GT that goes either 12mph or 120mph yet requires a mechanic to switch between the two speeds, and then have the audacity to claim that this offers an easy migration path between the city street and the autobahn? (analogy credit to Ed, you know who you are!)
This makes no sense to me.
dg
Jan/100
Homebrew Render-Farm. Frankly, ‘just cause’
Wire Frame of an Icon created in Rhinocerous and VRay
Ok, putting this in context- I wanted some new network icons. Somehow all the ones I used in the past were made by an art department I strangely do not have access to anymore, and I really don't want to have to pay an agency to make them for me. I could probably outsource somewhere, but don't want to have to explain what I want, so sometimes it is just easier to do it yourself. (and learn a few new things while you are at it.) Plus it was a good way to spend an evening...
So off I went. Using a 3d design program known as Rhinocerous, which is an insanely cool name (I wish I could name my products things like that... a new switch is the Raven 98000, and over here we have the Magpie 5600 connected to the Corpus Corax 11000. Ok, you get the drift, cooler names should be used in networking products, not just secret-decoder ring needing acronyms and SKUs limited to 17 characters.... (and who picked 17!)) Ok, off soapbox, don't expect this to change anywhere anytime soon.
So we have Rhino running, doing a bit of drawing, getting the shape right and such. Then you couple that with a ray-trace rendering app, in this case VRAY, for Rhino. You get a lot of choices about textures, lighting, etc frankly too many for a neophyte, but these are pretty powerful programs. This is where it gets fun though - there is an option in VRAY for 'distributed rendering'. Nerd alerts went off throughout my office as I madly scrambled around loading a VM with the VRay distributed rendering client onto every machine I could get my hands on. Old Mac laptops, an 8-Core MacPro, a 4-Core MacPro, even a 2-Core MacMini fell victim to loafing this intimidating piece of software. I then realized that I had some network issues to quickly patched through a few more Cat6 ports from the office to the wiring closet, locked the ports down at 1000-FULL and moved my IP Phone to a PoE port while I was at it...
Probably the coolest part was watching the MacPro spawn multiple execution threads which you cold see rendering in real time. Render times were cut down by about 70% from using just one machine.
Here you can see the active raytrace threads modeling different surface segments. I was playing with lines on the front to show hot-aisle cold-aisle airflow. FAIL. 
Lessons Learned
It wasn't all roses. A few things I learned and a few things I think the SW developers should focus on in future versions.
1) VRay and Rhinocerous both do not have native Mac versions yet. This is frustrating but you can work around it with VMware Fusion 3. They both worked pretty well through a VM on Windows XP. I am still not up to 7 being happy to have skipped Vista.
2) Since you are running it in a VM note this- On the station with Rhinocerous be sure to tweak your setting to as many CPU cores as you can. I set it up for 4 cores and 3-4Gb of DRAM on the VM. I need more RAM for this machine, it could easily be happy with 16Gb on the VM. I am looking forward to the native version.
3) On the distributed render farms you don't need a whole lot of memory as it seems mostly CPU intensive, at least for the way I was using it. I set mine at 512Mb of Dram and let the other machines continue their happy servitude streaming iTunes, serving photos, keeping my Drobo happy, and generally performing well. Even the Tweetdeck machine. On these and the master you will have to move the Network Interface Card settings from NAT (default) to Bridged. You will probably have to at least go to the console and do a 'ipconfig /release, ipconfig /renew' to ensure the adapter comes up and you are ont eh same LANs egment as teh physical hosts. I was not able to get it working with NAT. Also be sure to let the sockets through any host-stack firewalls, McAfee goofed me for a bit on this.
4) Room for improvement- a native MacOS client for Rhinocerous and for VRay would really help. But the way the developers have you add distributed render nodes is archaic. First on the node themselves it spawns a text window and doesn't provide any diagnostics, just a scrolling log when it gets a job.
b) VRay requires you put the IP address, hard coded, into the master machine of each client. Don't you think this would work much better integrated with Bonjour or something that enables auto-discovery of potential render-nodes.
c) Even smarter would be have the render nodes run as a reduced priority process in the taskbar. Then each machine in a studio could be helping any rendering via processor reclamation when not being dominated by the user.
d) I like the real-time display of the ray tracing going on, but put a report in their showing what system did what percentage of the work. This way I would now which ones to upgrade, find the bottlenecks, etc. A little diags would go a long way here.
e) Also when showing the list of the servers check availability and let me know BEFORE I start a render job.. novel.
Here you can see the finished object, ready for export to a PNG to plague PPT users everywhere... Not sure about my air inlets though...
In the end, it was fun, I will continue to use them, but there is some room to improve that would be really useful for someone like me and I imagine the IT staff at any design studio. Here's some shots of the finished products...
Here is a final of a 1RU switch, no air inlets or anything on the front. I like the raised blue arrow look a lot...
Nov/092
HP takes out 3Com- what is the next consolidation step?
HP-3Com - ushering a wave of tech consolidation?
People have been asking me for a while what the next 'shot' would be in the tech titan border-clash. Cisco entered the server market with UCS, and everyone was wondering what the response would be.
I didn't think 3com would be taken off the market this quickly, I figured everyone would wait a year or so to see if 3Com could be successful in breaking back into the global marketplace, outside of China, with their current and new product lines. HP, taking some risk in that department, made an aggressive move knowing that HP has the global footprint and 3com has strong roots in China that HP can leverage. I have to say it's an impressive bit of M&A.
But what is next?
The real question is how will others respond to this move. What will IBM do? What will Dell do? I have postulated for some time that we are in a phase of consolidation where the tech-titans, in order to have competitive portfolios, will acquire or build these capabilities.
Neither IBM, nor Dell have data center networking presence, both have partnerships with Juniper and with Brocade. A lot of people were betting on an HP-Brocade acquisition, as evidenced by the share price impact on Brocade today. And who can count out Oracle/Sun? They also do not have a networking footprint.
I think the major players will wait for a quarter or so, through the holiday season - evaluating their options and also seeing how HP rolls up the 3Com acquisition. If it creates competitive advantage for HP then IBM, Dell, and Oracle will follow in HP's footsteps. I can't say who will acquire who, but there is only a small universe of potential acquisition targets as well.
Does this spell the end of independent networking?
dg
Oct/096
Cautious Optimism, Irrational Exuberance, Full-Circle Come-a-bouts, and Economic Recovery
Everything seems to come full circle in IT...
Cautious optimism is a term I have been having many discussions lately with friends and analysts about - whether we are seeing true economic recovery or a bit of a 'W' and whether to make serious investments in planned growth or not. Candidly, in IT we have compressed capital spending for a while, so it could just be a bit of elasticity - although one major thing strikes me as different.
In the current world order many of the IT investments seem to be directly proportional to short-mid term ROI, sure everyone wants to build for 5-10 years, but they also want to see real business results, right now.
Mostly this means that new project types are getting priority and IT is finding creative and innovative ways of delivering near-term business value without, hopefully, taking their eye off the architectural ball. Ideally we can do both- deliver short-term value creation, while building towards a longer-term vision that enables IT to reinvent itself and infrastructure to transcend generational shifts. Sadly this is not always the case, some companies and people seem to want to either over-rotate on short-term. Sadder, others refuse to admit the world is changing. Even worse are those who keep their head in the sand and cannot move at all. Denying change happens is dooming almost any business to failure, embracing a fickle trend too quickly can be just as painful, and relying on past formulas from previous successes doesn't always work.
You may wonder where I am going with this. Over the past thirteen years I have seen a lot of things change and come full circle- Cut-Through Switching, Lossless L2 Networks, Ring Topologies, Hosting/Cloud/Insource/Outsource. Universal truth - things change and open and experienced minds that can capture this change tend to prevail.
Architectures have to change with the trend, the old way of doing things is not always the best- althought there are always viable lessons to be learned and due respect should be paid to past success.
Looking at networking, especially in the data center there are a lot of architectural changes in play. Obviously, the changes being driven to effect convergence between Ethernet and FibreChannel is a big one. The other is the collapsing of layers and efforts to simplify the topologies while increasing the scale of operations - I think in my next post or two I will have to explore these some more, what are your thoughts on other architectural changes in the data center network?
dg
Oct/094
ISR G2 – what I wish it was…
Cisco ISR G2 - the best a branch can get?
Cisco announced the new ISR line recently, a 3x performance improvement for the high-end moving up to ~150Mbps. But the question I have that has been lingering with me for a while is, "Why not use an x86 processor and a decent hypervisor with that?"
Crazy, I know, right? But with the current set of Intel Nehalem cores I can get several Gb/s of sustained throughput at varying packet sizes. So it's not like I have a data plane performance issue. You can even schedule the cores to provide additional protection for mission-critical control plane processes.
Regrettably, to me, this was not the direction taken with this line. Why do I think it would be cool? For several main reasons:
One thing you could do is run several VMs for integrating neat things like Call Managers and Network Analysis. Who needs a separate co-processor when you can cost-effectively get a CPU with more than enough horsepower and DRAM to run a variety of concurrent branch office workloads.
Control Plane Performance would be through the roof - so you can actually support the market that fiber to the home is creating for Gigabit Ethernet handoffs to the home and business. This is rapidly expanding and becoming a more and more popular handoff in dense urban environments.
Killer integration - run branch office apps, rin your own apps, run the routing protocol stacks, and have enough process and VM separation to guarantee performance and stability. But also you wouldn't have to do special versions of IP-PBX call management for the router- you could run the full-blown image right on it. Want WAN Optimization, load a VM. Same with Network Analysis, etc. At some point performance will peter out, but not too soon.
Sadly for me, this feels like an opportunity lost. But who knows - maybe they will pull a rabbit out of the hat with something like this someday.
dg
Oct/097
On Merchant Silicon and Lawnmowing
Custom or Merchant Silicon?
Hello there!
I've been on a bit of a vacation the past couple of weeks, so sorry for the slow-down in posting frequency. But now, I am back at my desk, with a decent speed Internet connection, and too many ideas on what to write about.
---------
Chance had it that today a friend of mine sent me an email reminding me of a blog post I wrote about a year and a half ago titled - 'On Merchant Silicon and Mowing my Yard.' It was a piece, purposefully a bit inflammatory, designed to have a bit of a go at the guys who were drawing some interesting comparisons to product lines I worked on.
As often happens, times change.
Reading this email which spoon-fed my prior writings back to me, I had to reflect a bit. Does merchant silicon matter? Is it a sell out? What creates customer value in a networking product? Is it the silicon, the software, the system as a whole, how it operates with other adjacent products, etc...
Realistically, to me, what most people seem to care about in a switch is that it works, at the performance rates necessary, and that the software is stable and has the features they need to accomplish the networking task at hand.
Inside this system there is a collection of silicon, some of which may be custom, some which comes from vendors other than the systems manufacturer. The PHYs are almost always 3rd party, the CPUs as well, most every part is always 'merchant silicon' except the switch fabric and the packet processor. These are custom in some switches, or merchant in others depending on that specific vendors goals for that particular product.
Ethernet continues to evolves and offers lots of opportunities for innovation and improvement - some of these will require new silicon, some will require new software. Using merchant silicon sometimes will affect a vendors ability to control the pace and order they bring features to market that require new silicon. However, with the breadth of silicon available in the market today there are several viable silicon choices.
Custom chips create their own set of challenges - most vendors doing their own chips actually do their own logic and rely on a third party to actually build the chips and fab them out, deal with the process challenges, etc. The vendors doing full custom take on a high cost burden up front and hope they get enough market traction to enable future R&D in their ASIC teams. In this period of compressed budgets these programs tend to be one of the first things to get financial focus. What do you think- is one better than the other, or is a healthy mix appropriate for our maturing industry?
P.S. Oh, and I outsourced yard mowing... too time consuming and others did a better job than I ever could.
default gateway
- Vendor Math September 18, 2009
- Implications of Virtualization and Cloud August 19, 2009
- Virtualization's Impact on Networks August 6, 2009
- Brad Reese, PR's Worst Nightmare August 5, 2009
- Hello World, and Random Career Thoughts August 4, 2009