A few days ago, the BPX Beacon Chain suddenly stopped generating new blocks and stalled at the height of 2105857. Now we know the causes of this failure and we are working on a solution that will get the blockchain back up and running.

The difficulty was too high…

The previous version of BPX blockchain (V2) had a netspace of over 400 petabytes at its peak and about 200 petabytes on average. Having such data, and taking into account that some users may leave during migration or hesitate to switch to the new chain, we designed the difficulty adjustment algorithm of the current V3 chain for a similar network space, with a large margin ensuring correct operation of the chain even if netspace drops 4 times. At the time it was a good decision, the BPX V3 netspace was growing and mining difficulty factor jumped from 1 to over 20 after a few weeks.

During the next months, difficult market conditions, a decrease in hard drives mining profitability and the development of new compressed plots formats, incompatible with BPX caused the BPX netspace to drop to an unpredictable value of just 11 petabytes. Mining difficulty went back to the minimum value of 1, meaning there is no way to adjust it down any further. Even though the BPX Chain was not designed to operate with such low netspace, it was running quite well without any alarming symptoms. Unfortunately, there was a ticking bomb hidden in the consensus algorithm that was just waiting for a certain condition to be met in order to lead to a blockchain disaster.

…and we accidentally increased it

About a month ago, due to changes in the project structure, we were forced to move all our servers to different locations. To ensure the continuity of BPX Chain timelords operation, we have configured new timelords as virtual machines in a new location, before the primary physical timelords were temporarily shut down. Such timelords had to share resources with other services, so they were significantly slower than physical ones. We knew that if there were no other timelords in the network apart from our official ones, the rate of generating new blocks would slow down a bit, but within 24 hours at most, the blockchain would adapt to the new timelord speed. The hardware used as timelords has been changed many times in the past, and the network has quickly responded to the change.

The network has lowered the sub-slot iterations – a parameter directly related to the timelord speed, but also used to check whether a given plot is capable of generating a block or not. Unfortunately, in a situation where the difficulty factor has reached a minimum value of 1, lowering the sub-slot iterations amount is equivalent to increasing the actual mining difficulty by an order of magnitude, even though the difficulty factor remains unchanged. As a result, the difficulty, which was already inappropriate for the current BPX netspace, has skyrocketed even further. Combined with the additional requirements of the consensus algorithm, this led to a situation where none of the plots used in the entire network are able to generate the block for several current challenges, and the same challenges are repeated over and over and will not change to different ones until a block is created, thereby completely paralyzing the chain.

Mission: Bailout

The unforeseen consequences of a relatively simple data center operation led to the first-ever unrecoverable failure of an entire blockchain. The only reasonable way to get BPX Chain back up and running is a hard fork that changes the difficulty adjustment values to appropriate for the current netspace and timelord speed.

We are currently working on the Beacon Client update that will introduce the necessary changes. The new version will be marked 4.0, as it will not be compatible with the previous ones. We are making every effort to release it as soon as possible. We will announce its availability in the separate post on this website and on social media.