Shard revert due to database issue

Post by **Red Squirrel** » Fri Feb 02, 2018 2:48 am

Due to a database corruption issue we had to do a revert to Sun Dec 31 18:00:08 EST 2017. Given how dead the shard has been lately it only got noticed now so had to use a pretty old backup... but given the lack of activity not like much was lost anyway.

Sorry for the inconvenience this may have caused. I will investigate the cause of the corruption.

Archived topic from AOV, old topic ID:6740, old post ID:39199

ninja2007 · Post by **ninja2007** » Fri Feb 02, 2018 5:17 pm

should of left the characters wiped

i would of loved to start over from scratch with a bare server

Archived topic from AOV, old topic ID:6740, old post ID:39201

Post by **Red Squirrel** » Tue Feb 13, 2018 5:29 pm

This happened again this morning.

There's a weird issue with my network where if something is pegging disk I/O too much, stuff starts to lockup. I was trying to sync the ethereum blockchain, I think that is what was causing issues.

Had to restore a backup from this morning. (Tue Feb 13 06:00:26 EST 2018)

I stopped the blockchain sync for now to see if the problem goes away just to rule it out. If it is that, then I will simply put it on a dedicated machine instead of a VM.

Archived topic from AOV, old topic ID:6740, old post ID:39203

Post by **Red Squirrel** » Sat Feb 17, 2018 11:24 am

So this seems to keep happening. I really don't know what's going on but I won't really have much time to investigate for a while but I do need to look into this further. Nothing has actually changed for a long time so I really don't know why this keeps happening.

Archived topic from AOV, old topic ID:6740, old post ID:39216

Post by **Red Squirrel** » Sat Feb 17, 2018 11:37 am

It seems the network link to the DB server is randomly dropping. Going to keep shard down while I do a ping test. It seems to only happen when there is heavy load on the file server. Right now there is a backup job running and it's just killing everything. This issue has been a bane of my existence, it seems backup jobs put a ridiculous load on the server and then it causes stuff to crash and I just don't get why.

Edit: Decided to bring shard back up as I wanted to test backup anyway. I will monitor ping test while it's up.

Backup from Sat Feb 17 06:00:08 EST 2018 restored.

Archived topic from AOV, old topic ID:6740, old post ID:39217