Since moving to our new awesome servers following the 49ers/Raiders game, the instances of unscheduled downtime have been few. We even got through the draft without so much as a hiccup.

There was a corrupt database table that occurred during a daily backup early Thursday morning. It was the very large table that held all of the post replies, which is why you could not load thread pages. The rest of the site was not impacted. Of course, I was sick and did not notice until I woke up around 11:30 AM. I ran a fix which takes about 30 minutes on a table that large (which is pretty darn fast and impressive). However, that brought down the entire network of sites during the time as system resources were directed at the fix. Everything was back up by noon.

The odd thing is that the forum should be set up to shut down during scheduled backups, so it is odd that this occurred. Support says that there was a surge of traffic during that time Thursday morning which is odd and which is why I wanted it investigated. Either way, the forum should have automatically shut down, throwing up the familiar error message.

I submitted a ticket to support to investigate the cause of the issue and specifically told them that I had fixed the table. I guess they did not read that part because on Friday afternoon, they once again ran the fix that I had already run, which brought everything down for 30 minutes again.

Later next week, I may look at converting the table to a different type, which should lessen the possibility of it becoming corrupt. However, that will take some time since it is the largest table on the site.

I'll try to update this thread after major issues. Obviously not after minor ones.

Of course, there are scheduled downtimes.

Scheduled downtime list

The portion of the site being backed up is unavailable during these times.

Daily backups occur sometime between 2 AM and 3 AM. The time fluctuates since the forum is one of the last databases on the list that is backed up. I have seen it finish as early as 2:20 AM and as late as 3:07 AM (that one was during a backup that served as a daily and weekly).

I do a manual backup (databases only) to a local machine on Fridays sometime between 2 PM and 3 PM. It depends on my schedule. This backup takes about 5 minutes. The forum (or portion of the site being backed up) is unavailable during the backup.

Temporary reschedule

Daily backups have been temporarily rescheduled for Monday, Wednesday and Friday rather than every day. This will likely go back to the daily schedule later.

More

Please keep unnecessary comments in this thread to a minimum so that my updates can be easily found by users. I don't want to lock it though because it would just fall down the page rather quickly.