All is well!

The problem with status pages is that they are invariably doom and gloom, especially numerous recent posts on external routing problems. We try and keep you all informed and not cover things up, which does not help this image. So I thought I would post something more positive for a change and a brief summary to save you reading all of my detailed ramblings.

In short: We believe external routing is now working!

Basically there have been routing problems or blips meaning loss of internet service (mostly for 20CN customers) for anything from a few seconds to a few minutes at a time. This have been very occasional but much worse over the last few days (which thankfully most people still consider a holiday period). These short blips have been a total bugger to track down and we have worked through a number of possibilities.

We have finally found the cause(s) of the problem which all stem from a link between two data centres in London. It is ironic that it is actually because we added extra redundancy to these links last month that there have been these problems. The main cause is a hardware problem, outside our control, in the link provider and they are contacting the hardware suppliers to resolve this long term. In the short term we have worked around the problem now we know what it is. All is well now!

There were things worse than a few seconds blip though. Friday we upgraded routers in case they were the cause, and found the latest software has a serious bug causing what would be a blip to be over an hour out of service at 10pm and then at 4am. We worked through the night and this has been resolved. We thought we had found the cause on Saturday lunch time, but it was probably just one factor. We finally narrowed the problem down to the link between data centres and last night the link providers confirmed they could see errors on the link. Late last night we co-ordinated a reconfiguration with them and everything has been fine since.

The only reason we have this link in use for the broadband customers is because of the delay in deploying the new 21st Century services with BT. Later in the year they will not be in use at all for broadband.

So, good news! Sorry for any inconvenience. We have learned a lot from this. And thank you all for your patience.

Lets all have a good 2009.

Adrian
Director