Comments on: Easynet outage

By: Buzz

Buzz — Mon, 18 Aug 2008 09:33:38 +0000

[15/08/2008 16:42:16]
We have identified an issue with our peering to Global Crossing. It appears that Global Crossing suffered a cable cut / failure this morning and have re-routed all traffic. This has resulted in congestion and latency across their network. We have been re-routing traffic away from the Global Crossing network which is why some customers will have seen some improvement already.

—

Or so they would have you believe …

A tracert to mysql.com for instance clearly shows 67.17.198.202 … (removed my ip’s obviously :p)

tracert mysql.com

Tracing route to mysql.com [213.136.52.29]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms xxx.xxx.xxx.xxx
2 1 ms 1 ms 1 ms xxx.xxx.xxx.xxx
3 9 ms 9 ms 15 ms cr0.wmhas.uk.easynet.net [195.189.159.122]
4 3 ms 3 ms 3 ms 80.238.54.161
5 12 ms 11 ms 11 ms 67.17.198.202
6 11 ms 11 ms 11 ms 67.17.198.201
7 35 ms 35 ms 35 ms 64.214.141.10
8 37 ms 38 ms 37 ms sto-cr2.upp-rr1.bahnhof.net [85.24.151.173]
9 40 ms 39 ms 40 ms upp-rr1.upp-dr4.bahnhof.net [85.24.151.159]
10 40 ms 39 ms 40 ms upp-dr4.c3750-upp3.bahnhof.net [85.24.151.171]
11 39 ms 39 ms 39 ms ue1-gw.eth0.bahnhof.net [195.178.188.117]
12 37 ms 38 ms 37 ms http://www.mysql.com [213.136.52.29]

Trace complete.

By: Buzz

Buzz — Fri, 15 Aug 2008 15:55:33 +0000

Thanks for the info M, that last part made me chuckle!

It is amazing how an ISP can warrant about 8 hours of outage …

—–

We currently have a fault that is effecting customers trying to browse and connect to some sites.

We are currently investigating the cause of this fault.

[15/08/2008 12:26:43]
Easynet engineers are working with our transit provider to try and resolve this fault.

[15/08/2008 13:25:27]
Our transit peering provider has confirmed that they currently have a fault on their network. 15/08/2008 08:15:00 0d 7h 34m 1 Major Outage

—–

A classic “shift the blame” to a third party peer … someone at easynet wants a slap to be honest, if your third part you PAY to peer with goes down, CHANGE YOUR ROUTING TO MAINTAIN YOUR SERVICE.

The last two lines I have quoted as just typical,

[15/08/2008 12:26:43]
Easynet engineers are working with our transit provider to try and resolve this fault.

[15/08/2008 13:25:27]
Our transit peering provider has confirmed that they currently have a fault on their network.

So by these accounts it has take easy net an HOUR to figure out there is actually a fault, as if the call volumes they must be getting are not evidence enough …

By: M

Fri, 15 Aug 2008 12:28:32 +0000

I’m stuck too, it’s been a fun morning of tracerting the places I need to go and finding everything stopping at 67.17.198.202. When I reported the misbehaving router to support, they helpfully asked me what kind of router I was using.

In desperation I’ve been pulling a few pages I needed out of Google’s cache, which takes forever because the adverts Google plasters all over them are hosted somewhere 67.17.198.202 doesn’t like.

67.17.198.202 belongs to Global Crossing, whose own websites are inaccessable thanks to 67.17.198.202. Slightly worryingly, their Wikipedia page says at the bottom “See also: List of notable business failures”.

It aggravates me, I’m sure there’s millions of pounds worth of shiny Cisco hardware along the way, with the most advanced routing protocols known to… Cisco, and they’ve obviously made a static route for half the net through a box that doesn’t even have a DNS entry and belongs to a company that used to be bankrupt.

By: Jim

Jim — Fri, 15 Aug 2008 11:51:56 +0000

so thats the problem.