Blog>> System Downtime Report: 1st to 3rd August 2015
Period of Downtime:
Saturday 1st August 2015 20:00 BST, to Monday 3rd August 2015 12:00 BST
Reason for Downtime:
An issue developed affecting a large data storage array which resulted in our primary server becoming unresponsive on the 1st August 2015 at approximately 20:00 BST. Our three independent monitoring systems failed to identify this downtime resulting in an extended period of downtime before the issue was investigated. Our primary server was brought back online on the 2nd August, and backup/restoration processes were restarted around 24 hours after the initial failure.
At this time the affected data storage array is offline whilst it is performing integrity checks, as a result some customer backups are unavailable.
UPDATE 12:00 BST: The affected data storage array is now back online and all customer backups are now available. We have no reason to believe any data has been disrupted, though please check your backups and notify us immediately if you notice any issues.
Future Prevention:
We have increased the sensitivity of our proactive monitoring system and introduced new monitoring criteria to improve the likelyhood that such issues will be detected in advance in the future. We will be reviewing all our monitoring systems to identify how to prevent such failures from re-occurring. We will also be releasing an update to the myRepono WordPress Backup Plugin to address connection issues reported by several customers.
We sincerely apologise for any inconvenience caused by this downtime!
"Update: Version 2.0.11 of the myRepono WordPress Backup Plugin has now been released which will automatically detect connection failures of this nature and then disable connection attempts for 3 minutes. This adjustment will minimise any potential disruption caused if the myRepono system becomes unavailable in the future."
On 3rd August 2015, myRepono Team posted:
"Simon, yes this issue is related. The myRepono WordPress Backup Plugin connects to the myRepono.com system to retrieve data, since the system was down the connections would have hung until they timed-out which should occur after 30 seconds, at which point the WordPress administration panel should load as normal. We sincerely apologise for the inconvenience caused by this! Please rest assured this is a very rare occurrence, this is the longest downtime our system has ever suffered, and is in fact the first time the system has been down for more than an hour or two in over five years. We will be reviewing how the plugin can be improved to address this issue in due course."
On 3rd August 2015, Simon posted:
"This weekend I had to deactivate the WP plugin on a large number of sites as it was making their back-ends completely unresponsive. Is this related? I have rather lost faith in your system as a result - being unable to backup is one thing; being unable to manage sites is in a different league!"
