Thursday, October 09, 2008

21CN BRAS-RED2.CF-C fault

One of BT's 21CN RAS's appears to have a problem affecting a small number of 21CN lines. BRAS-RED2.CF-C appears to be unable to connect. We have reported to BT.

4 Follow-up Messages (Posted by AAISP Staff):

AAISP Status said...

We are still chasing BT but it seems engineers are currently working on the fault and hope to have the affected customers back up ASAP.

AAISP Status said...

There is a really strange network fault in BT causing this. We have managed to set up a work around for now while we get to the bottom of this.

AAISP Status said...

Sorry this has taken so long. Cardiff and Bristol 21CN lines were affected and we have managed to work around both.

The problem is some sort of micro black holes in BTs network. We can be pretty sure it is BT! The effect is specific BT RAS's (Cardiff and Bristol) cannot see specific LNS IP addresses our end (different ones for each RAS). There is no logic to this. The RASs can see other ISPs, and our LNSs can see other RASs.

We have ended up setting up alias IPs on the same LNSs our end and routing those RAS to the right aliases. I.e. no change to the box our end apart from giving it an extra IP address.

BT are investigating!

AAISP Status said...

BT tracked this to a CISCO router in Faraday with a corrupt cEF table (route cache). It appears the router must have been rebooted as lines were dropped briefly.

Note that this will make a mess of the graphs for affected customers.