Visit Your Local PBS Station PBS Home PBS Home Programs A-Z TV Schedules Watch Video Donate Shop PBS Search PBS
I, Cringely - The Survival of the Nerdiest with Robert X. Cringely
Search I,Cringely:

The Pulpit
The Pulpit

<< [ Oh Brother Where Art Thou? ]   |  WYSIWYB  |   [ DRM Catcher ] >>

Weekly Column

WYSIWYB: Honesty (and a little TCP Mustang) is the solution to tomorrow's Internet video problem.

Status: [CLOSED] comments (63)
By Robert X. Cringely

For the past two weeks I have covered what I consider to be a rapidly impending bandwidth crisis created by the rise of Internet television. Google understands this better than most and believes it has a solution that will enable it to gain strategic advantages and new business opportunities (that was my column two weeks ago). There are clever ways to put in place a solution similar to Google's though not owned by Google (that was my column last week). This week we look at the same problem from a deeper level and consider yet another potential solution that might work, if it ever really gets a chance. That solution is from a company called NuMetra, Inc., which of course I own no stock in so stop bugging me about that.

NuMetra, based in Pleasanton, California and founded by ex-particle physicists from the Stanford Linear Accelerator Center -- your typical bunch of atom smashers -- is a company dedicated to improving Internet video by removing kinks from the last mile of distribution. Their work is novel and innovative and clever as all get-out. I have nothing but admiration for their technology, though I think it requires so much buy-in from parties with diverging interests that adoption in the U.S. is problematical. That will put the U.S. at a further international disadvantage because NuMetra is doing much better in places like Japan and Korea where Web 3.0 will probably be born.

We agree on many things, NuMetra and I. We agree that Internet video traffic is growing at such a rate that it is going to lead to severe network congestion problems almost immediately -- problems that will hurt the transition to television over the Internet if not promptly addressed. If progress is not going to be impeded too much, something will have to change.

At the basis of NuMetra's view of the marketplace is a paper by William B. Norton of Equinix that is included as one of this week's links. Norton presents an analysis of probable video distribution costs using different loads and network technologies, including: 1) continuing to use the current transit model; 2) using content distribution networks (CDNs) like Akamai; 3) inventing new CDNs, and; 4) using peer-to-peer (P2P) distribution technologies like BitTorrent. Norton's analysis, which appears to me to be well thought-out, concludes that P2P is vastly cheaper than any of the other approaches. He concludes that distributing a 1.5 gigabyte movie over the Internet in high volume will cost $0.20 using the current transit model (a single huge distribution server), cost $0.24 using an edge-caching CDN like Akamai, cost $0.17 with a homemade CDN like I used last season to distribute NerdTV, or cost $0.0018 to distribute using P2P. That makes P2P 35 times cheaper than any of the alternate approaches.

But of course it is much more complex than that. Norton makes the perfectly correct point that the p2p cost savings isn't a true savings but is rather a hidden cost borne by the Internet Service Provider running that last mile connection. It isn't that P2P is really so much cheaper, it's just that a lot of the costs are being borne by the ISP without specific compensation.

This is the best explanation I have yet seen for why broadband ISPs hate the concept of network neutrality. P2P, which constitutes a huge percentage of network traffic right now (about half), uses ISP bandwidth without providing any specific ISP compensation. While many writers, including me, have called ISPs greedy for wanting to bust net neutrality, Norton's work puts some numbers to the ISPs argument, and you can see why the ISPs are so upset.

Norton further makes the point that none of these distribution models does anything to soften the blow on the ISP. CDNs in particular cost more -- that more being revenue to the CDN -- yet do nothing for the ISP.

One fairly obvious solution to this impending crisis is to throw bandwidth at it, but that isn't as simple as it sounds. At the heart of this video distribution problem is the lie that ISPs tell about how much bandwidth we are really buying. While you may think your 1.5-megabit-per-second DSL service or your 3-megabit-per-second cable modem service is actually backed by 1.5 megabits or 3 megabits of Internet bandwidth, they really aren't. ISPs provision backbone access based on the expectation that people usually aren't on the Internet, and even when they are on the Internet most of their time is spent reading the screen, not actively sending or receiving packets. As such, ISPs have been able to get away with buying 20-30 KILOBITS per second of Internet backbone capacity for every MEGABIT per second of Internet service they are selling at retail. This 20-to-1 provisioning ratio of what's sold to what is promised (and believe me, 20-to-1 is me being generous to the ISPs since it is probably much higher than that) is what creates the burgeoning Internet video problem.

So if ISPs would simply provision the amount of bandwidth they are selling us there wouldn't be a problem, right?

Alas, that's not the complete story.

What would appear to be an obvious solution to this impending problem is increasing effective backbone bandwidth for every broadband user. There is no LOCAL bandwidth crisis, just an INTERNET bandwidth crisis. As I've explained over the last two weeks, Google is going to solve this for us in exchange for taking the majority of global advertising revenue. Maybe that's the answer, in which case we don't have to do anything. But if Google doesn't step in appropriately or you'd rather not rely on Google for some reason, there is the alternative of going with a different ISP.

If you want a beautiful Internet video experience, one answer might be to buy service from a broadband ISP that will allocate more backbone bandwidth per account. If you are a DSL user, you have a choice of ISPs in most markets. They all go through the local phone company, of course, but that phone company has no problem allocating the rated value of the DSL line because to the telco that last mile bandwidth is free. My DSL ISP is MegaPath and they buy a 1.5-megabit-per-second circuit from BellSouth (now AT&T I guess, but the trucks don't say that yet) and it can really carry 1.5 megabits per second 24/7, no problem. The problem is that MegaPath then makes me share a DS3 (45-megabit-per-second) connection to the Internet with every other customer of theirs in my LATA or service area, so in practice I get a LOT less than my 1.5 megabits.

One would think a national DSL ISP would have a huge business advantage if they could sell me a 1.5 megabit service and actually PROVISION a 1.5 megabit service. The wholesale cost of Internet bandwidth is right now around $15 per megabit per month. So delivering a REAL 1.5 megabits vs. the 30 kilobits I am probably getting right now will cost my ISP 75 times as much, but that's still only $22.50 vs. $0.45. So the question is whether I would pay $22.05 more than I am paying right now to be able to effortlessly watch video over the net?

Of course I would, only it's not that simple.

Going to a 1:1 provisioning ratio WOULD improve my Internet service, but it would still be far from perfect. Blame Bob Kahn and Vint Cerf for that wacky TCP/IP, which does some very non-deterministic schtick that creates problems for networks of nearly any size.

You know how you can be driving along at the speed limit in your car when suddenly everyone slows down nearly to a stop? There must be an accident ahead, yet when you finally get to the head of the queue nothing is there; traffic stopped for no reason at all? Well TCP/IP is like that, too, with unexplained traffic glitches.

The total solution involves increasing bandwidth, sure, but the way that bandwidth is USED also has to be changed, and toward that end NuMetra is pushing its own network protocol, the InterStream Transport Protocol (ISTP). When supported from server to router to client device, ISTP creates a faster network using the same pipes and a really fast network with somewhat bigger pipes. It is essentially an Internet fast lane -- a communication channel optimized for speed over the last mile through priority routing.

There are three major components to NuMetra's technology. There is the ISTP protocol, which is compatible with TCP/IP but must be supported by devices end to end; there is NuMetra's TCP Mustang code, which appears to be an alternative TCP stack on steroids (or from Hell if you happen to be an ISP); and finally NuMetra's Stateless Mediation Controller.

Nobody really understands the math of TCP/IP well enough to predict traffic glitches or know how to avoid them, but the folks at NuMetra know some about how to exploit them. NuMetra's TCP Mustang code can be used to effectively game TCP, creating a clear path for packets to follow. The problem is that TCP Mustang's success comes at the expense of all other traffic, which can be severely detained as a result. Slow things down enough and packets have to be retransmitted, grinding the net to a halt yet again.

To hear NuMetra explain it, a company like Yahoo or MSN could adopt TCP Mustang (it is open source) and use it as a weapon to gain quicker access to client machines. TCP Mustang is the threat NuMetra holds over the ISPs. TCP Mustang's power is controlled by the system's two other components -- NuMetra's Stateless Mediation Controller and its InterStream Transport Protocol.

NuMetra's total solution, while technically complex, is easy to understand. They want networks and users alike to adopt ISTP in all servers, routers, and browsers. An end-to-end ISTP network is called the InterStream Media Grid and ought to make supremely efficient use of current network bandwidth and hardware to speed video from server to client with minimal hassle and completely avoiding the burbles and eddies of TCP/IP. It is brilliant and ought to work fine, but requires the cooperation of ISPs.

The folks at NuMetra think ISPs will join their consortium and adopt ISTP out of self-interest since it will speed their network. But it will also REVEAL their network, showing for the first time exactly how fast or slow it really is. That's because ISTP comes with a whole system of metrics (NuMetra, get it?) that will show the true performance of an ISP's network. These metrics will be published and searchable and for the first time users will be able to compare ISPs based on performance, not claims or specifications.

NuMetra's business model relies on politically correct Open Source software (I feel better already, don't you?) and revenue sharing with the ISPs and transit or backbone providers. Their model has NuMetra charging a toll for using the InterStream Media Grid and sharing that toll revenue with backbone providers and ISPs alike, giving both some compensation for the higher video volumes to come and the inevitably higher costs associated with those higher traffic volumes. It makes sense in a weird way if you don't think too hard, though it isn't immediately clear who pays these tolls, whether it is the content provider or the viewer, though perhaps that doesn't really matter if the money actually comes, say, from advertising.

For all its elegance, there are still unanswered questions about NuMetra. Yet the premise described above is that video will constitute at least 80 percent of future Internet traffic. We're not building a new road, remember -- not adding any lanes -- just prioritizing traffic over the lanes we already have. So what happens when 80 percent of traffic tries to use the fast lane?

NuMetra can argue, of course, that additional toll revenue will allow ISPs to make capital improvements and add more lanes, or even that those same ISPs will be shamed into better performance by coming up short on NuMetra's comparison metrics, though my mother will never see those. And remember, this is with the expectation of a 1,000 times increase in traffic. If I double my capital budget or even increase it by 10X, what real effect will that have against a 1000X increase in demand?

NuMetra might argue, of course, that tolls would act to mitigate demand, but that's hardly going to help an industry develop. "Look, we're helping you grow by charging you money." Yeah, right.

In the end, what NuMetra's metrics will show is what we already know -- that there isn't enough ISP Internet bandwidth. The key is getting what we pay for and actually paying for it. The current system is built on marketing lies, and we're being told more lies about how to solve the resulting problems. The problem isn't transit costs or content distribution networks or peer to peer -- that's just BS.

I hope NuMetra is successful, but if it isn't here's an incredible business opportunity for a national broadband ISP with rational bandwidth provisioning that will simply acknowledge reality and fill a need.

What You Buy Is What You Get (WYBIWYG). It might work.

Comments from the Tribe

Status: [CLOSED] read all comments (63)

I'm shocked, I mean *shocked* that Bob has joined the 'ISPs need compensation' bandwagon. Why am I paying then $35 a month if I can't use the pipe any way I see fit? A couple of people have already commented on that, but it is worth repeating.

Abhijeet | Feb 17, 2007 | 9:20PM

I would love to see the papers on some of these technologies. I can't fine any references to TCP Mustang or ISTP at IETF, and NuMetra's site is shockingly sparse on specific information about how these technologies actually work.

Eugene | Feb 18, 2007 | 12:06PM

Great insight...

Although, if you look carefully, nuMetra isn't really that new an idea. The academic community has been focused on fixing TCP for the better part of the past several years (see TCP Fast (CalTech), and TCP Africa (Rice)). The real innovation here is in their economics. If you look at their association,, it looks like the real objective is to get the "old Big Media" to partner with the ISPs to only stream authorized content over their "diamond lane" service. Bill Smith, the CTO at BellSouth, proposed something similar about a year ago. nuMetra's approach appears to be one heck of a lot more elegant and scalable...
It looks like old Media needs to get a clue and partner with them or perish to the likes of Google.

Jack | Feb 18, 2007 | 10:35PM