Reminder, maintenance in telehouse

Just a reminder that we are today doing some maintenance in Telehouse as per planned work.

We expect to be changing routing around while we do this and there should be no disruption while we do that.

There may also be a few seconds of outage on 21CN links while we replace a faulty switch and test a second fibre link.

This work is replacing some faulty equipment and adding more equipment to improve future resilience. We are working towards being able to do maintenance with no outages at all even of a few seconds, but this will depend on new equipment planned for later in the year.

We'll update on irc as we do the work.

MSO affecting 21CN customers

There is a major service outage affecting our 21CN customers, BT are aware of this and engineers are currently working on the fault.

Update to follow.

ADSL2+/21CN update

Yesterday we moved a lot of lines over to the new 21CN platform. The process was interesting. BT still have issues with orders not closing off properly and so these lines are on ADSL1 with faster uplink for now, and will change to ADSL2+ some time in the next few days.

For the most part the upgrade was as expected with only a couple of minutes disconnection, but a small number of lines have issues which are being resolved as quickly as possible. Also a small number did not upgrade until today but have gone through without problems.

There was an unrelated incident within BT affecting some lines (not just ours) just as they happened to be upgraded yesterday afternoon, which caused some confusion. BT have fixed this.

We are doing the final batch next Tuesday. Well, final for now. We'll do a batch each month now.

However, yesterday, BT also announced news on the IPStream connect product which has come as a surprise. As I have mentioned before, IPStream connect is the product that allows us to finally turn off our existing expensive and limited capacity "BT Central" links.

Like many small ISPs we were led to believe that IPStream connect was like "throwing a switch" to just connect all of the existing lines to the new network. It was launched 31st July! We expected that to be the date it would all happen.

Then we were told "launch" meant order on 3 months lead time. But OK, so November time for change over.

Then we were told "end of the year", so December time. However we confirmed they meant "financial year", so that meant March.

Yesterday we were told probably August 2009 if we are lucky. A big delay!

This has to be one of the worse communicated product launches I have ever seen from BT and we know many small ISPs that spent a lot time and money ordering new 21CN links (as we have) specifically to allow them to turn off the old "BT Central" links.

It means we are running two networks for nearly a year, at extra cost not reduced cost.
We are now upgrading previously temporary network links and changing plans as a result of this news.

However, we are continuing to regrade people as their exchange is upgraded and the target is around 40% of population by March. As such we do not expect growth in use on the existing "BT centrals" to exceed capacity. We could order new capacity on the old links, and BT have even made a nice special offer for that. but they are insisting on 3 month lead times even where existing fibre is in the ground (as we have), so by the time it arrives we will not need it!

Not good news. It is fair to say we are "disappointed" by this, and feel "misled" by BT. We can't say BT have broken any commitments as BT never formally committed to any of those dates.

Router reboot

We are rebooting the router called restless. This should not cause any disruption.

Good news for ADSL2+

Having explained some of the issues with BT here, I appreciate some customers may be apprehensive about their regrade to ADSL2+ on Tuesday. We have several hundred lines being regraded and even BT were apprehensive at some points this week!

However, I'd like to reassure those customers being regraded that we have carefully reviewed the issues that affected the first batches earlier in the week. Issues affected small number of customers but it was not a huge batch of upgrades. Most customers simply regraded cleaning with a few minutes off line and now have a better service.

My team (mostly Shaun) have been working with BT all week on this, and we have a handful lines that have been off all week following regrades. Contrary to our usually conservative terms these customers will receive a full month's credit for the inconvenience which BT have caused. BT have been woefully slow in sorting that issue and it is not really acceptable that customers should suffer from their incompetence.

Major issues now fixed:-
a) Docklands not linked to Docklands as reported. Affected upgrades have been cancelled but customers stay on the new cheaper tariff. Will be upgraded when BT connect up.
b) Some orders not closing due to a system fault within BT which BT are fixing the weekend.
c) Some lines where customers have ZyXEL routers not working on ADSL2+ and were upgraded and put on ADSL2+ by mistake. Changed so they stay on ADSL1 for now but go on the new network. There is also a work around on the ZyXEL router to fix to ADSL1

Minor issues:-
d) Jumpering errors, quickly fixed by advising BT of the fault when it happens
e) Other router problems, quickly fixed or worked around in some cases with new routers

So, we are confident that the upgrades on Tuesday will go fine.
We have put in place extra monitoring of the whole process so we have a clear overview of regrades as the happen during the day and can see any that fail and resolve promptly. BT are on alert (as much as they ever are) to help resolve issues promptly.

We plan to do several hundred more lines the following week.

These upgrades will take significant load off the existing BT Central lines and improve service for everyone.

We will then do upgrades each month as exchanges are upgraded - this will continue for years as the whole country moves to ADSL2+. We believe Bt are aiming for something like 50% by March next year, but who knows.

Thank you all once again for your patience.

One of the issues with ADSL2+ upgrades

A dozen of the customers notified that we would be upgrading to ADSL2+ next week will not be upgrading, sadly. They will be emailed individually.

This is due to a problem with BT's systems which we were assured could never happen (yes we checked in advance). This specific problem has had an impact on a few lines that were upgraded last week and it has taken days to get these customers lines back on line.

This time we have been able to identify the lines that would fail and ensure the order is cancelled in advance. So hopefully Tuesday should run much more smoothly.

This is the explanation to the best of my knowledge at present...

All broadband lines on 21CN connect back to a main interconnect point, one of around 20 in the country. There is then backhaul connecting these points around the country back to one place where we connect to BT. That one place depends where the ISP is and in our case that is a location in Docklands. There is then fibre links to the data centre where we have our equipment.

Sounds simple enough.

The problem has come about where there are exchanges that are upgraded to 21CN, and connect back to one specific interconnect point. This specific interconnect point is not yet connected to the backhaul back to Docklands. As such it is possible to upgrade customers on these few exchanges and they then cannot actually use the service. This is what BT assured us cannot happen. The order should not have gone through for these customers.

What we have just discovered, and needed to know to do checking of potentially affected lines, is which of the interconnect points is not yet "plugged in" to the point in Docklands.

We just found out...

It is "Docklands".

Yes, the problem is that the Docklands interconnect point does not yet have any backhaul to connect it to the Dockland interconnect point where the fibres to us are located.

And no, this is not just in a name, we are talking the same actual building here.

I should know better, having dealt with BT for a few decades, but what can I say?

Thankfully, now that we are aware of this, and now that we are ignoring the assurances BT gave us, we can stop it affecting customers on Tuesday.

P.S. Any customers notified of their upgrade to ADSL2+ and then not upgraded get to stay on the new cheaper tariff anyway. We expect these customers to be able to be moved to 21CN quite soon and there is no point messing with the billing twice.

Thank you all for your patience.

Routing problem to some services in Redbus

We are currently investigating a problem with routing to some hosted services in Redbus/Telecity from ADSL lines. Eg this is affecting customers with hosted servers and FireBricks in one particular rack.

Serious issue in Telehouse

There appears to be a serious issue affecting our systems in Telehouse. We are investigating.

None of the systems are responding which means that all ADSL2+ customers are off line at present.

Router reboot

We are rebooting the router called restless shortly. It should be possible with no disruption to service, but the fallback has not had a lot of testing. If there is any disruption it should only be a few seconds.

Trying to work with BT

We are trying hard to work with BT to resolve the handful of lines that have failed to regrade to ADSL2+ and stopped working. This is very few customers, but it is not really acceptable that these customers have been affected like this.

We are working with BT to try and ensure that there are no more like this when we regrade more customers next week. It is however a somewhat frustrating process.

We have found that 3 lines have been moved to ADSL2+ when BT don't actually have the necessary links in place to connect that exchange to us, so those customers are off line now. We did get specific assurances from BT before we started migrating customers that this very situation cannot ever happen. It seems those assurances are worthless.

Normally, if there is a fault with a line, and we do not fix it promptly, the main recourse a customer has is to leave. However, in the interests of good will, any customers off line due to an ADSL2+ upgrade for more than the day of the upgrade can ask for a credit of the broadband services charges for the days they are off line. Please wait until the fault is resolved and then contact sales to arrange the credit.

We are also looking at discount prices for new routers to take full advantage of ADSL2+ services where existing routers are out of warranty and cannot handle ADSL2+. Note that normally a non ADSL2+ router works in ADSL1 mode.

Customers we thing have ZyXEL routers are being upgraded to the new links but only ADSL1 mode. A small number have been regraded to ADSL2+ by mistake and this has been corrected now. There is also a work around to force a ZyXEL to stick to ADSL1 only.

Also, any customers where we are unable to regrade to ADSL2+ having advised that you will be, and hence remaining on the existing ADSL1 service for now - can stay on the new cheaper tariff anyway.

I would like to reassure customers that the majority of these regrades are working fine, and that customers are connecting to our new higher capacity links and getting faster speeds and better service. In most cases where the regrade is not as planned the existing service continues to work. It is very few lines that are having serious issues with the regrade process.

Thank you for your patience.

VoIP - Problem with International Calls

We are currently looking in to a problem with being unable to make international calls, instead an error message is given or a recorded message saying "you are not registered to use this service..."

ADSL2+/21CN update

We have done initial small batches of upgrades, and most have worked. There have been a small number which have failed. BT have identified the cause of this now, so we are going ahead with the first big batch of upgrades next Tuesday. You will get an email today confirming if you are one of the customers being upgraded. Assuming that goes well we will be doing more the following Tuesday.

"Home" now called "Basic"

We have decided to change the name of the new entry tariff from "Home" to "Basic". The change will be reflected on the web site shortly. This is only a change of name.

We think it better reflects that it is the "entry level no frills" tariff, and also avoids confusion with our very old "Home 500", "Home 1000", etc, tariffs.

VoIP 999 and 112 calls

Calls to emergency services now have to be possible via VoIP. We have been trying to arrange this for some time, and neither of the carriers we use are quite ready yet. We expect that this will be resolved any day now. OFCOM are aware of this. In the mean time we have added a clause to our terms to cover this.

Whilst we will be providing access to 999 and 112 this is not something to rely on. You should always ensure there are older means to call Emergency Services numbers. Correct operation of VoIP depends on your VoIP and internet equipment working (so power available). It also depends on an inherently unreliable service of the internet itself. We cannot guarantee 100% availability of our internet services and hence VoIP access to 999 and 112.

The change in regulations also mean it is quite likely that we will no longer be providing international outgoing calls. We are looking at ways to arrange it so a separate company provides such calls in a seamless way. We'll provide more specific advance notice if we do have to withdraw this service or if there is a change of configuration necessary.

Maidenhead hosting prices

Following a price rise by the data centre we are increasing our power and space prices for hosted servers in Maidenhead by 10% from next month.

21CN update

Just an update of how it is going with 21CN and ADSL2+...

BT are still facing some challenges and we are working with them to try and resolve these, but it is slow going. The key thing is that we can actually upgrade lines to the 21CN connection and ADSL2+ service. We have weekly calls to go through the long list of outstanding issues. BT are themselves having daily conference calls with all departments to discuss progress with senior management. BT do recognise that they have not really managed to "launch" the services to us, but the majority of our issues revolve around diagnostics and repair services which we can do by phone, so are not really stopping us go ahead with upgrades, thankfully.

We are taking it a step at a time. We have done several ad-hoc upgrades already, along with our ex-traillists lines. We have the first small batch of ADSL2+ upgrades going through tomorrow, one in each dialling code. One has even completed already this afternoon. We then have a small batch of 50 lines going through on Monday. So far it looks good.

If this all goes well we'll be able to put through the remainder of lines that can now be upgraded (around 6% of our lines). We aim to do them over a period of around one week which will start a week from when we start. Our aim is to have them all sorted by the end of this month.

Then, each month we will go through exchanges that have been upgraded that month and migrate those lines over.

Hopefully later in the year we can connect the rest of our customer to the new 21CN link.

As we move people over they move on to the new tariffs.

Thank you all for your patience.

Glitch

One of the 4 BT links we have for our ADSL1 services failed for no apparent reason. This means nearly 25% of customers just lost service and reconnected a few seconds later. We are now rebalancing lines which means there will be a few seconds outage on some lines so as to ensure usage is spread over all 4 lines cleanly.

High usage

OK, I have never heard so much hype over a physics experiment before, and it is not even as if they are colliding anything today.

Anyway, just to say, that over the last hour we have seen very high levels of internet traffic causing some packet loss - much like the Olympics. People streaming video feeds and checking news sites before the world ends!

Unlike the Olympics this should die down quite quickly, though I would expect traffic to be a little higher than normal today.

As for solving the issue with high load once and for all - we are making progress on the migration of customers to the 21st Century Network and have the first small batch of customers moving on Friday with another small batch on Monday. If this goes well we expect to move the rest by the end of the month. Sorry it is taking so long but BT are experiencing some "challenges" in making the whole process work.

Oh, and in case you are not sure, this site may help...
http://www.hasthelhcdestroyedtheearth.com/

VoIP Registration Problems (SIP)

We are investigating a problem with SIP VoIP phones being unable to register.

New web site, clarification of tariff implications

The new web site defines the new tariffs we are launching.
These are for new orders from now.

Existing customers are on the previous tariffs. These are still listed under "old tariffs" on the new web site, and are not changed.

We are in the process of moving the existing customers to the new link we have to BT and at that point changing those customers to the new tariffs.

For most existing customers, right now, the tariff has *NOT* changed.

There is a mapping from old to new Max tariffs.
http://aaisp.net.uk/broadband-transitional.html
The new tariff is cheaper and in some cases has more peak bandwidth allowed.

We are moving around 6% of customers to the new link which should be this month (i.e. before the next bill). We have the ADSL2 ex-triallists on the new link and are moving a small batch of customers next week to confirm it runs smoothly.

We are then moving customers as and when their exchange is upgraded. This means these customers also have ADSL2+ available which is faster for many lines. This should move reasonably quickly over the next few months,

Soon, but depending on BT, we expect to be able to connect all existing customers lines to the new link to BT and change tariff at that point. This will affect all Max customers. We are hoping some time around November, but this may not be for a few months after that depending on BT. BT's plans have slipped rather. We'll keep you all posted.

Customers on very old tariffs, not Max, are not changing tariff or service until they select a usage based tariff.

I hope this clarifies matters.

New web site

We have launched our new web site.
See the News section for details.

ADSL2+/21CN update

Just a quick update on progress.

We are still working with BT to help them resolve a number of issues. In theory we are in a position to actually migrate over 6% of existing lines to the new 21st Century links as orders are now able to be submitted to BT. However we are not keen to do so just yet as BT still do not have diagnostics, repair, or even migration code generation working yet. Until this is working we do not want hundreds of lines on the new service for obvious reasons.

This could change any day if BT fix the issues. We have a small number of test upgrades going through. If you would like to be one of the guinea pigs, please ask on irc. It takes a week!

Unfortunately the first such test orders have now past the contractual delivery date with nothing happening, so we cannot even be sure that ordinary upgrade orders are working properly yet.

We are very keen to get the 6% of customers moved before the end of September. We are doing all we can to help BT resolve the issues quickly.

However, we are hoping to launch our new web site this weekend.

Thank you all for your patience.

ADSL2+

There is an issue affecting ADSL2+ customers at present which we are investigating now.