Unfortunately this activity resulted in a absolute affected and poe currency new abortion approach that we had not ahead considered.

When we actualize new instance servers for players to play on, they are angled off from a so alleged "Prespawner" instance that has preloaded a lot of of the bold abstracts into memory.

This is so that instances can allotment anamnesis for changeless assets that are acclimated for the game.

Unfortunately there were some gaps in the abstracts that the prespawner instance was loading, which agency that those assets were loaded on appeal if a specific bold server bare them.

The acumen this resulted in the issues that we saw today is because rows from one table references abstracts from added tables by index.

When rows are added or removed from the boilerplate of a table, the basis of those rows in the abstracts change. This is what fabricated the references finer go out of sync.

Effectively references to rows were pointing to about approximate places. Uniques were pointing to the amiss mods.

Base types were pointing to the amiss art. Adventure rewards were pointing to the amiss abject types. Abounding added tables were analogously broken.

When we got letters of the botheration and started to see aberrant items affiliated we knew that something acquire to acquire gone amiss with the deploy.

After shutting the branch down, we had a harder accommodation to make. Do we do a database rollback?

There were abandoned 18 annual of damage, but at the time we had abandoned apparent abandoned a few aberrant items and did not acquire the admeasurement of the problem.

Rolling aback the database would acquire taken a fair bulk of time, during which we would acquire had downtime.

There are aswell a lot of added issues that yield a lot of time to array out if accomplishing a rollback.

For example, new accounts that acquire been created by users will be lost. Any purchases of microtransaction credibility would acquire been undone, but we would still acquire the money.

If we had done a acknowledgment we would acquire bare to manually array these issues out afterwards the fact.

Therefore, at that time I absitively that a acknowledgment would could cause added abuse than good, poe goods and we absitively to redeploy aback to the accepted adequate version.