Sunday, June 29, 2008
Power issue in Maidenhead
Our logs suggest there was a major power outage around 5am in Maidenhead data centre. This case caused issues with some machines starting cleanly including email servers which are being reset now.
Subscribe to:
Post Comments (Atom)
Posts
2 Follow-up Messages (Posted by AAISP Staff):
The data centre advise:-
At approximately 4:45am this morning BlueSquare 2 and 3 suffered a power outage affecting rack customers.
Power has been restored, and engineers are on site currently making sure all racks are back online.
Full details on power outage from blue square:-
Dear Customer,
At approximately 4:45am this morning BlueSquare 2 and 3 suffered a power outage affecting rack customers.
Power has been restored, and engineers are on site currently making sure all racks are back online.
An update with further information will be sent as soon as possible. Please do not hesitate to raise a support ticket if you continue to experience difficulties.
Regards
Dear customer,
Following on from the issues this morning, we are continuing to work with suppliers to find the cause of the outage.
Current investigations appear to have identified a phase detection relay which may have failed in the generator auto-changeover panel. This could have caused the panel to think there was a mains power failure when in fact there had not been.
The UPS systems worked as expected without fault, however due to the mains being forced off by the potentially failed relay, and power from the generators also locked out, the UPS batteries fully discharged after 10 minutes resulting in a critical power failure.
We have engineers collecting a new relay, along with other spares that maybe needed from the manufactures in North Yorkshire. We expect these to arrive in the early hours of tomorrow morning.
During this time we have manually disabled the mains auto changeover system in order to aviod any potential further mains cuts. In the event of a real power outage on site we will have staff perminatly onsite to operate the generator changeover procedure manually.
This should be treated as an 'at risk' period for customers in BlueSquare 2 & 3 until further notice.
Once we have further information we will update this ticket.
We apologise for any inconvenience caused.
Dear Customer,
The switch panel vendor has been on site this morning and has conducted several tests of the board. A decision has been taken to replace the Phase Failure Relay (PFR) for a new one.
Once this has been done the vendor will re-check all circuitry.
We then wish to complete a mains failure test, in order to fully test the systems that have been replaced to ensure a full switch over to generator takes place in the event of a real power cut. We will run on generator for approx 15 minutes, and then complete a switch back to mains to test the reverse operation.
We expect the generator test to start at approx 11.30am and be completed by midday.
Under normal circumstances there should be no effect to customers service, however this is an at risk period.
We will further update this ticket once this test has been performed.
Many Thanks for your continued support during this time.
Dear Customer,
The generator test has been completed as planned, however our vendors have requested one further test this afternoon to make sure all goes to plan a second time.
We expect to complete the second test at 1:30pm.
We remain in an at risk period at this time.
Further updates will be posted once the next test has been completed.
Dear Customer,
Tests on the board have been completed sooner than expected, we will now complete the generator test at 13:10 instead of 13:30
Updates to follow.
Post a Comment