Visit Your Local PBS Station PBS Home PBS Home Programs A-Z TV Schedules Watch Video Donate Shop PBS Search PBS
I, Cringely - The Survival of the Nerdiest with Robert X. Cringely
Search I,Cringely:

The Pulpit
The Pulpit

<< [ Declassified ]   |  In a Jam  |   [ It Takes a Monopoly ] >>

Weekly Column

In a Jam: Stories of ISP bad faith and can the government really listen in to your VoIP calls? Yes they can.

Status: [CLOSED] comments (77)
By Robert X. Cringely
bob@cringely.com

My son Channing, who is four years old, recently celebrated Pajama Day at his preschool when everyone -- even the teachers -- came to school in their pajamas. In this instance at least, Channing was a month ahead of Google, which will in December hold its own Pajama Day when the company's global workforce will be encouraged to come to work in their jammies. As usual for Google, though, this first Pajama Day will be treated as a beta release with specific exemptions for people from "cultures where pajamas are not worn." No word yet on the proposed etiquette for Googlers who sleep in the buff.

I am not making this up.

Here is something I wish I were making up. A good friend of mine noticed last June a sudden and precipitous decline in his volume of incoming e-mail with the numbers dropping by 80-90 percent. Was he less popular, less interesting than before? Or maybe some Bayesian filter had been imposed by his ISP (Earthlink) to suddenly spare him completely from spam. No such luck.

The trend continued so my friend, who has long been in the networking business, himself, started running experiments. He sent messages from other accounts to his Earthlink address, to his aliased Blackberry address, and to his Gmail account. For every 10 messages sent, 1-2 arrived in his Earthlink mailbox, 1-2 (not necessarily the SAME 1-2) on his Blackberry, and all 10 arrived with Gmail.

Swimming upstream through Earthlink customer support, my buddy finally found a technical contact who freely acknowledged the problem. Since June, he was told, Earthlink's mail system has been so overloaded that some users have been missing up to 90 percent of their incoming e-mail. It isn't bounced back to senders; it just disappears. And Earthlink hasn't mentioned the problem to these affected customers unless they complain. The two groups affected are those who get their mail with an Earthlink-hosted domain and those with aliased e-mail addresses like my friend's Blackberry.

Were they thinking these thousands of affected customers simply wouldn't notice? And what about those customers whose livelihood depends on e-mail communication? There are both ethical and business questions here and Earthlink doesn't look good on either scale. Fortunately the company says it is installing new software and hopes to have the problem resolved before the end of the year. Lucky us.

This sort of ISP dissembling happens more often than many of us might guess as companies play the odds and pray that their faults aren't noticed. These mistakes, by the way, typically aren't actionable thanks to our blindly clicking on those Terms of Service agreements that we never read. In Earthlink's case, if they don't deliver your e-mail, well that's just tough.

But there are instances where even if you think you have a guarantee, you often don't. I have a backup business DSL account from Megapath, a national broadband ISP which is the only DSL provider here in Charleston who offered static IP addresses when I was shopping around. Of course the DSL actually comes from BellSouth, which I think of simply as The Devil for its poor service and vindictive ways, from which Megapath presumably would protect me for only three times the price. My Megapath account even came with a Service Level Agreement that guaranteed 99+ percent uptime with money back for any interruptions that exceeded certain reassuringly slim limits.

Then my Megapath service went out for a whole week and I learned the bitter truth. Lucky for me this was my backup account used regularly on just a single notebook in the bedroom, but its failure did put a temporary crimp in my wife's eBay obsession.

Of course I filed a trouble ticket with Megapath, which they cheerfully acknowledged, then 24 hours later told me what was already obvious -- I had no Internet service. The problem was a bad DS3 leased line to Atlanta that was giving them trouble, but it would be back up in a matter of hours. It took days. And the problem (this was last year) continued to happen intermittently for months. One thing I learned from this experience was that Megapath, seeking good customer service stats, times out its trouble tickets WHETHER THE PROBLEM IS ACTUALLY FIXED OR NOT. If you want them to keep working on the problem you have to keep opening new trouble tickets. And one important measure for them of customer satisfaction is the percentage of trouble tickets that are closed which, of course, has to be nearly 100 percent.

But the worst part of this experience came when I tried to invoke my Service Level Agreement and get some money back. They should have owed me a free month of service. Nope. You see the service they were guaranteeing wasn't actual Internet connectivity, they explained, but my connection to the DSLAM half a mile across town at The Devil's office. As long as the log showed that I had a continuous connection to BellSouth (a connection that I am sure was guaranteed by BellSouth under THEIR Service Level Agreement with Megapath) then Megapath was off the hook. The ever-cheerful Megapath customer service agent explained that just because I couldn't get on the Internet for days at a time that wasn't their problem. They guaranteed presence, not utility. In fact, the whole Service Level Agreement scam wasn't even their problem since all they were really guaranteeing to me was something that they were, in turn, guaranteed by The Devil. And when all you have left is a guarantee from The Devil, well you know you are in trouble.

Then they closed my trouble ticket, chalking up another satisfied customer.

If you are still with me, here's the part of this week's column that I find the most interesting. It's yet another example of how the world doesn't really work the way we think it does. Your homework assignment in this case is to read all 269 pages of RFC 3261, which defines the Session Initiation Protocol (SIP) which is the basis of many Voice over IP (VoIP) telephone services as well as other fun things like various video, audio, and text chat services. You'll find the complete text of RFC 3261 in this week's links.

Finished? I'm especially interested in Section 26, which details threats to SIP security and how SIP defends itself. Not so well, it turns out. In fact, the very insecurity of VoIP (services like Vonage, for example, are based on SIP) is unnerving.

To understand this, we have to distinguish between end-to-end encryption, and hop-by-hop encryption, both of which are supported by SIP but are used under different conditions.

With end-to-end encryption, the packet header is open (naked) and the payload is encrypted. If the encryption is set up correctly, then the payload is safe. However, the packets can be subjected to traffic analysis of the header information as well as Denial of Service. So someone with the appropriate software can know who you are talking to and when even if they can't listen in. And if they dislike you enough, they can keep you from talking at all.

With hop-by-hop encryption, the packet header is open (naked) and the payload is encrypted. It's just like end-to-end encryption, except for the problem of proxy servers. In order for the proxy to properly route the packet forward, it must examine the payload to extract the destination information. The proxy will decrypt the payload, read the destination address, modify the already open header, re-encrypt the payload, and then forward the packet. At the instant that the packet payload is decrypted at the proxy, it can be copied. That's the attack point. There are many other problems, of course, about how the successive re-encryptions are done with what key, with what Certificate Authority, with what authentication? This is security?

But wait, there's more!

Session Border Controllers (SBCs) are hardware devices that make SIP work over the heterogeneous Internet. There are thousands of them at every ISP. SBCs require that the packet payload be made naked, period. This is absolutely necessary to do transcoding, perform NAT traversal, etc. That's the attack point.

Even more.

The Communications Assistance for Law Enforcement Act (CALEA -- I've written about this one before) requires "managed" VoIP operators to provide law enforcement agencies a point of interception so they can tap your VoIP calls. What's a "managed" VoIP service? Packet8, Vonage, Comcast, and AT&T all certainly qualify, but does Skype? Yes, if you think of billing as management, now that there is SkypeOut and SkypeIn. And given the current management at the U.S. Department of Justice, "managed" could mean pretty much anything.

VoIP interception is usually done at the SBC/proxy. The network operator's SBCs perform decryption/encryption on the "secure" packets as they go through the node. It is a matter of "trust," as they say in the industry. If you want to encrypt you must also be willing to trust an SBC/proxy in China, Russia, wherever. That's the attack point.

Remember this is the technology that will shortly be the basis for nearly all telephone service.

On second thought, I think I prefer writing about Pajama Day.

Comments from the Tribe

Status: [CLOSED] read all comments (77)

'You can have it done(or fixed) fast, right, or cheap- pick two.'


No. Those are not mutually exclusive.


Otherwise, no argument from me.

Wendell Cochran | Dec 11, 2006 | 1:12PM

I am curious - i have friends who have not been receiving email sent to their juno account, and have not been able to send email to friends (delivery not made, but it seems to "send"). Is this another example of the email overload? I think i will run my own tests....

Steve Graham | Dec 13, 2006 | 2:00AM

Yes, Earthlink seems to have lost some of my emails the past month or so. My email seems to break down and I am guided by a tech to try to repair it. Sometimes it takes an hour. Sometimes 2 or 3 days. Some email are not accounted for when the service comes back.
Strange. Earthlink (dial up) used to be so reliable.

fred ackerman | Dec 14, 2006 | 10:08PM