There appears to be a serious issue affecting our systems in Telehouse. We are investigating.
None of the systems are responding which means that all ADSL2+ customers are off line at present.
Thursday, September 18, 2008
Subscribe to:
Post Comments (Atom)
Posts
11 Follow-up Messages (Posted by AAISP Staff):
This appears to be a fault with a link between data centres in London which failed at 22:08. We are investigating.
We have tracked down to a faulty link in Telecity and trying o get this addressed now.
The good news is we know what happened, and it will be sorted in a few minutes.
We have also had a good chance to review the fallback arrangements and work out why they are not quite as they should be. That is something we will be going through tomorrow.
However, this really is something we could not have expected to happen.
The man from Telecity, who says "sorry that was my fault", and who is described as an "idiot" by the man from datahop, managed to knock out at least 5 datahop ports, so affecting a number of ISPs.
He then proceeded to plug them back in randomly.
So, we have an active switch port that is not going where it should be.
Datahop have their own engineer who will be on site any minute to unravel the mess as they have banned the telecity engineer from touching anything. Oh, he moved more cables after being banned.
The link has been restored, thanks to the Datahop engineer. ADSL2+ lines can now reconnect.
We will be reviewing our backup procedures over the next few days to try and better allow for situations like this.
And I have to say thanks to the helpful people at Datahop at 2am...
having restored services there was something of a routing blip affecting all customers for a minute or two. This seems to have been resolved.
We are somewhat concerned that L2TP links are not coming back up, and investigating further.
We are seeing an issue with BT links in THN now. This is unrelated as a different data centre. However we did get an engineer to check the connections when this fault first happened and he may have disturbed something.
Sorry, this means ADSL2+ lines still down. We are working on this now.
When trying to identify the initial problem we reset some switches in Telehouse. However, that had an unexpected side effect which is apparent now the actual problem has been resolved. We have had an engineer in Telehouse fix this for now and one of our engineers will be going in tomorrow to find out why that happened.
In the mean time ADSL2+ service is back and working.
Thank you and good night.
I am actually looking at an engineer going in over the weekend and replacing a faulty switch which will probably mean a few seconds downtime for ADSL2+ lines.
He will also installing some more of the equipment we need in Telehouse.
I'll post a planned work announcement when we have more details.
ADSL2+ customers also had around 6 seconds of outage, twice, just before 08:30 this morning while we checked the fallback link to BT. Something which was not working at 3am and is now fine.
Post a Comment