Reason for unexpected outage - we've run out of disk space!

[ryanschristmaslights]

Did you try to access AusChristmasLighting earlier this morning but unexpectedly greeted with a strange single line error message? The answer is our storage drive filled up to 100%, and when there's no free space things tend to fall over. How embarrassing!

As a result, for the moment at least I've had to turn off new attachment uploads. They'll be renabled once the issue of too little free disk space issue is sorted. Existing attachments are still available to view/download.

Certain safe guards with the backend systems did not prevent us completely running out of space, annoyingly.

 

[djgra79]

Thanks for the update Ryan and for all your work in the background to keep things running.

 

[ryanschristmaslights]

Update: attachments uploads are permitted once again.

So what was determined? An automated scheduled site backup operation was occurring at the time of the drive filling up. This is supposed to abort if available free drive space is below a specified threshold, so it would appear that this needed adjusting to account for the growing website. It has now been tweaked so that an even higher amount of free space must be available before commencing the next backup. In addition we'll trial moving backups onto a storage drive that's separate to the website.