[closed] issues in Birmingham

There were some blips on one of the Birmingham BRASs early this morning - no word from BT on this yet, but affected all ISPs not just us. We then saw Solihull exchange fail - no sync on all lines. Again no word from BT on this, but we have reported faults on the affected lines.

[info] Dialup using PAP

Dialup using PAP is not working at present. Please try CHAP instead. We should have this resolved in the next couple of days. If you have any issues please contact support or ask on irc.

[info] We like cats!

We just had a call claiming to be from the cat protection league concerned over "Max" the cat in this picture as shown on our web site. Apparently concerned that making him wear a bow tie was cruel...

It is not a bow tie, it is a red cat collar. A perfectly normal and not at all cruel cat collar... Max was not posed for the picture for the web site - we simply took pictures of him when he was messing about (trying to kill the camera) as a family cat, and later thought they would be nice on the web site.

No animals were harmed in the making of our web site!

[info] Up to 40Mb/s broadband

We are on the FTTC (Fibre to the cabinet) trial and placing first orders today for installation next week. This is the start of up to 40Mb/s broadband. It will be interesting to see how quickly this is rolled out to the rest of the country.

[info] bug hunting

We are still trying to sort a network issue affecting 21CN services. We have clear evidence of some sort of network storm on the LAN in Telehouse. We have been unable to identify the source. We suspect a switch issue, we are planning to replace switches with some shiny new Allied Telesis models. However, our equipment should not fail in light of this occasional nuisance. We have worked through a number of techniques for addressing the problem which is fundamentally the same as handling a denial of service attack. We keep pushing the problem to the next bit of code. In the past we have managed attacks well and we should be able to sort this.

However, we have the development team (including me) working over the weekend, and with the testers on the test LNS, to get to the bottom of this. We are putting in even more diagnostics and even more rate controls at different levels in the system to ensure we can handle the problem without a restart. We are determined to get to the bottom of the issue.

Having issued a "stability" guarantee from 1st July, we have "paid out" by applying two extra 1GB peak usage top-ups to affected 21CN logins. I am sure this will not seem much to some of you, but this is very costly for us as a surprising number of people pay top-up usage each month so this can equate to many thousands of pounds. We don't really need this incentive - the quality and stability of our service is paramount and we have a reputation to regain for reliability.

This is made even more important by the fact we hope to move all lines to the new 21CN links to us some time next month. A big task, and one that is over a year after we expected. It is essential we have 100% reliable LNSs for our service now and in the future. Do not worry - we will delay this if we do not have stable LNSs.

We do apologise, most sincerely, for the issues. We know that for many this is a matter of seconds of outage every few days. We know a few of you have had minutes of outage (an issue we are taking up with BT as it should recover instantly, our end does!). We know that even a few seconds is not acceptable.

Testers on the test LNS (simple prefix your login with test- on 21CN) are invaluable in confirming the beta releases of new LNS code are stable and working. You get all usage counting 50% for billing purposes. Feel free to join in if you wish. You can change login back at any time.

I'll post follow-ups to this with details of the progress we are making.

Please do bear with us. If you are fed up, please do call or irc or email the specially set up mailing address we posted. If you leave, please do look again in a few weeks. You may find that you should have stayed, and we'll welcome you back.

And thank you all of our customers. We appreciate your support in difficult times.

[close] 21CN BT Birmingham BRAS

We have seen a number of lines drop and reconnect on Birmingham BRAS for 21CN again. Seems like there are still some problems there. We'll let you know if we get more details from BT.