Reason for unexpected outage - we've run out of disk space!

ryanschristmaslights

Senior elf
Administrator
Joined
Jun 30, 2010
Messages
780
Location
Adelaide, Australia
Did you try to access AusChristmasLighting earlier this morning but unexpectedly greeted with a strange single line error message? The answer is our storage drive filled up to 100%, and when there's no free space things tend to fall over. How embarrassing!

As a result, for the moment at least I've had to turn off new attachment uploads. They'll be renabled once the issue of too little free disk space issue is sorted. Existing attachments are still available to view/download.

Certain safe guards with the backend systems did not prevent us completely running out of space, annoyingly.
 

djgra79

My name is Graham & I love flashing lights!
Global moderator
Generous elf
Joined
Dec 27, 2011
Messages
1,771
Location
Cranbourne West
Thanks for the update Ryan and for all your work in the background to keep things running.
 

ryanschristmaslights

Senior elf
Administrator
Joined
Jun 30, 2010
Messages
780
Location
Adelaide, Australia
Update: attachments uploads are permitted once again. :)

So what was determined? An automated scheduled site backup operation was occurring at the time of the drive filling up. This is supposed to abort if available free drive space is below a specified threshold, so it would appear that this needed adjusting to account for the growing website. It has now been tweaked so that an even higher amount of free space must be available before commencing the next backup. In addition we'll trial moving backups onto a storage drive that's separate to the website.
 
Top