RSS Feed
[ LOSS OF PROTECTION ] 0000-0500hrs Tuesday 27th September.
Posted by Anthony Hogbin on 23 September 2016 12:35 PM

Day: Morning of Tuesday 27th September;

Time: Midnight to 5am Europe/London;

Reason: Soft and firmware maintenance along redundant transit route;

Effect: Loss of protection to core fibre and resulting transit to DC8 / St Asaph.


DC8 is supplied by redundant geodiverse fibre feeds.

During the early hours of Tuesday 27th - our transit providers will be carrying out work on a section of our reundant leg.

They have allowed a 3 hour window of hard down while software and firmware updates are applied, services restarted and tested.

We do not envisage any effect to services what so ever - however we will lose the N+1 protection for this period.

As such no services or transit will be effected directly.


We will update on safe completion.

[UPDATE 1109]

This test had been carried out - but has not completed well.


Post testing a modular UPS/PDU dropped the load causing a core switch failure.

We will remain in loss of protection state until this has been remedied.

[UPDATE 1300]

This engineering has completed successfully, and is to be considered closed.

Read more »

[RESOLVED] Issues affecting Mysql01 server
Posted by Jonathan Hughes on 15 September 2016 09:53 AM

Engineers are working to resolve a fault with the mysql01 database server. 

Please accept our apologies for any difficulties caused by this issue.

[update 10am]

Having identified the underlying issue, our engineers have moved to resolve the problem.

Service was delivered as soon as possible - and this is currently delivering content without issue.

We are now working through the databases and tables on this host, checking for consistency, defragmenting, and optimizing.

Service should return at an improved speed shortly.

If you have any issue after this point - please do raise this as a ticket so we are able to look into as an individual and unrelated issue.

Thank you for your patience during this resolution.

Read more »

[ RESOLVED ] SMS messages
Posted by Anthony Hogbin on 19 August 2016 04:12 PM

An issue with software that was sending text messages has been resolved.

The issue was raised with the vendor and they have resolved this - regretfully flushing the queue.

As a result a number of historical texts have been sent in error.

If you believe there is an overdue accounts matter - then please check your control panel to clarify.

Apologies for the inconvenience and confusion this may have called we are following this up currently.

Read more »

[RESOLVED] Network Interuption
Posted by Anthony Hogbin on 17 August 2016 02:45 PM

We are aware of packet loss issue effecting one of our ranges.

Networks are currently looking into this.

We will update regarding this as soon as we are able.

However it is important to note this is only effecting a limited number of addresses and this is connectivity as opposed to hardware or service related.

[update 1510hrs]

Networks have mitigated a distributed denial of service attack against a customer.

Access and connectivity have been restored.

We consider this matter resolved.

Should you continue to have problems please do not hesitate to contact us with specifics of the matter to so we can look into this further for you.

Read more »

[COMPLETED] At Risk Notification - UPS Maintentance 23rd and 24th August
Posted by Anthony Hogbin on 10 August 2016 10:47 AM

As part of our continuous service improvement program we will be conducting work on our UPS systems to replace ageing batteries and near end of life components. This work is necessary in order to maintain high levels of redundancy on our power infrastructure.

This work will be completed with our partners and overseen by senior management on site. The work touches all of our UPS systems in both Reynolds House - one of our Manchester hosting locations, however only one (out of ten) devices will be worked on at a time to ensure minimal risk.

All components are modular and RAMS and Method Statements are in place to ensure that UPS redundancy and capacity are maintained throughout the exercise. We do not expect this work to be service affecting but as with any work on key infrastructure the Manchester facilities should be considered at low risk for the duration of the works.

This only effects services delivered from Reynolds House.

Read more »