How a Dormant Bug Briefly Split the Ethereum Blockchain

The Ethereum blockchain temporarily split today, caused by a dormant bug that had been partially fixed. The issues have since been largely resolved—but questions remain over the handling of the fix.

The bug caused chaos because it affected the Ethereum infrastructure provider Infura. When the service went down, citing a “major issue,” many services that use Infura were left unable to interact with the blockchain. Many other nodes were also affected. But how did this all happen?

According to Péter Szilágyi, team lead at the Ethereum Foundation, the bug had been lying dormant for two years. John Youngseok Yang, a Ph.D student at Software Platform Lab, noticed the bug and received 20,000 points (worth $20,000 and typically paid in Ethereum) for finding two serious vulnerabilities. The bug affected a version of the Ethereum blockchain, called Go-Ethereum, or Geth.

Several months ago, Ethereum developers introduced a fix and many nodes running the latest version were then protected. However, the developers didn’t reveal details of the bug, in order to prevent a bad actor from exploiting it (on any nodes that hadn’t upgraded).

What went wrong is that some nodes didn’t upgrade to the latest version. In fact, Infura was running a version of Geth (1.9.9) that was released on December 6, 2019.

And then the bug occurred.

At 7.10 AM UTC, the blockchain split into a longer version and a shorter version. To some degree, this happens all the time and nodes stick to the longer version. But in this case, the nodes that hadn’t updated stuck to the smaller chain. This meant they were showing different blocks and disagreeing with other versions of the blockchain. 

With two versions, there was a split. This was from block 11234873 onward.

“The issue is that at some point some change to the code was introduced that led to a split between those who have upgraded and those who have not,” Nikita Zhavoronkov, Ethereum lead developer, told Decrypt.

A variety of nodes were affected. Binance runs its own Ethereum node, observed the split and temporarily closed withdrawals from the exchange. Blockchair, an Ethereum block explorer, was also affected and its data was then showing blocks from the smaller blockchain.

“Ethereum-dependent services are facing outages following the recent chain split. ETH withdrawals from Nash trading channels are currently failing, but we’re working to resolve the issue as soon as possible,” tweeted Nash, a non-custodial platform.

But the biggest disruption revolved around Infura. It runs Ethereum nodes on behalf of centralized and decentralized projects in this space and has been described as Ethereum’s backbone.

According to its website, many major projects use Infura, including Ethereum wallet MetaMask, non-custodial wallet Coinbase Wallet (which is different from Coinbase’s native mobile app), cat-breeding game CryptoKitties and lending platform Compound.

So, when Infura went down, many of these services couldn’t connect to the blockchain. Those using MetaMask were struggling to make transactions (unless they manually pointed it towards a node that wasn’t affected).

Infura, which has been criticized as a potentially centralized weakness in Ethereum’s decentralized architecture came under fire again over the issue. 

Casa CTO Jameson Lopp tweeted, “Three words that strike fear in the hearts of those who build on Ethereum: "Infura is down.””

Blockstream CSO Samson Mow added, “Ethereum is Infura. Decentralization theatre can stop now.”

However, Szilágyi came to its defence, tweeting, “Yo, Internet. Stop pointing fingers at Infura using older Geth. It's only sane from an operator perspective to not surf the newest releases.”

There are a few lessons that can be learned. One is that companies running nodes on behalf of other projects should probably update their nodes on a regular basis. “This is a reminder to keep your node(s) up to date!” tweeted a Geth coder known as M H Swende. 

The other is that there should be more transparency when Ethereum implementation updates are released. However, there is a dispute here over whether it would be a security risk to provide too much information.

“That said, silently fixing a bug dormant for 2+ years has a much lower chance of causing a disruption than raising awareness to it. We strive to minimize potential damage,” said Szilágyi.

“If we thought that announcing consensus security fixes would make exploitation less likely, we would do that. As it is, we generally do not announce such fixes, but recommend that people keep their nodes up to date. Two of our recent releases have vaguely mentioned security fixes,” added Swende.

So remember, if you see any vague mentions of security fixes—it's probably best to upgrade your node.