The race to debug Constantinople

On Tuesday, while much of Europe was tuned into BBC World watching the UK’s Prime Minister lose a historic Parliamentary vote against her Brexit bill, the cryptoratti were debating their own pending split on the Ethereum network.

A hard fork, known as Constantinople, had been slated to upgrade the network at 0400 UTC on Thursday. However, the 11th-hour discovery of a security bug prompted a rethink played out in public on the Ethereum developer’s forum,  Gitter, which led—almost as dramatically—to a vote to postpone the upgrade.

The problem is that, around half the network nodes had already begun to upgrade by the time the decision to stand down was made, based on Gitter comments. So now, the Ethereum community is working overtime to ensure that new code is disseminated to everyone who already updated, to ensure they don’t follow the wrong fork. They’re hoping to head off a situation in which some node operators have implemented the code, and will attempt the hard fork that had been planned for 0400 UTC on Thursday.

Dubbed Constantinople, the update was billed as a big deal, one of the more important  upgrades expected this year. It contains five nifty changes to improve the platform’s scalability, while reducing many costs. And it had already been attempted—and delayed—once before. So the stakes were high, or at least, felt that way to many in the community.

But on Tuesday, ChainSecurity, an audit security firm and developer of secure blockchain protocols, found a bad bit of stuff buried deep in the new open-source code for smart contracts. In the wrong hands, the bug might enable attackers to steal user funds. Chillingly, it’s the same kind of big that caused the DAO hack in June 2016, when $50 million of virtual currency was almost lost.

After the DAO hack, devs rewrote the network code so that, in theory, that kind of exploit couldn’t happen again. But the experience surfaced philosophical differences, which had caused the young community to split into two, and resulted in a new cryptocurrency, Ethereum Classic. It was a valuable lesson that taught developers that extreme care is always needed during a network upgrade. Hence the decision that missing an arbitrary deadline is far better than rushing bad code into production.

“Out of an abundance of caution, key stakeholders around the Ethereum community have determined that the best course of action will be to delay the planned Constantinople fork,” wrote the Ethereum Foundation’s Hudson Jameson, following a core developer meeting to determine a course of action on Constantinople.

The decision to abort was, by all accounts, not taken lightly. And, as Ethereum is a fully decentralized organization, most of the discussion, and steps leading to it, is on the Gitter feed. Technicalities (of which there are a fair few) aside, it makes for quite a thrilling read.

In fact, the event was done and dusted in 10 hours, the time that elapsed between ChainSecurity’s initial disclosure and the final decision.

Throughout that period, the “cat herders” were on high alert. (Yes, that’s the name of the special force Ethereum has created to deal with such emergencies, and to marshall the extended community in the correct direction.)

And they’ve had a lot of marshalling to do. With the decision to abort, new code needs to be promptly deployed by those working on the upgrade—and swiftly implemented by those who had already updated to the new fork of the blockchain. This needs to done before that hard fork’s hard deadline on Thursday. Anyone who had already updated for Constantinople and then went off on holiday, without hearing about the postponement, would effectively be stranded on a branch of the blockchain tree which was about to be lopped off.

Despite the bug presenting a critical security issue, and despite the fact that so many nodes had already begun to upgrade, developers were keen to stress that, to date, no smart contracts have been compromised. It was described as a “potential vulnerability.”

But nonetheless, a profusion of (mis)information was generated as a result of the postponement. Developers were criticised for not spotting the fault sooner and lambasted for incompetence.  

At the coalface of all the action was the team from MyCrypto. Addressing criticism on twitter, they pointed out the difficulty of auditing every existing contract while that contract is on an already-updated chain. And highlighted the problems faced by researchers trying to imagine what developers could dream up that could be inadvertently exploitable.

“Yes, it would be nice to catch it earlier,” admitted MyCrypto. “But the process for dealing with these bugs will improve as a result. And nobody lost funds.”

Some people were actually impressed by the way the potential crisis was dealt with.

The exposed vulnerabilities continue to be investigated by the team of core developers and ETH security researchers. It’s easy to imagine the mountain of takeaway pizza, they will get through while they pour over the code.

Meanwhile, over on Ethereum’s reddit channel, Vitalik Buterin, the platform’s much-admired founder, offered the following advice:

“I would say my personal takeaway from this is to be much more explicit about writing down invariants (properties guaranteed by the protocol) that we rely on so we can check against them when changing things,” he wrote, helpfully.

Further announcements about the longer term fate of Constantinople are likely to come after the core developer meeting on Friday. In the meantime, all the nodes, miners, and others who updated for the postponed fork, have been issued with instructions on how to proceed, thanks to the cat herders. However a firm date for implementing a fresh fork, with the temporarily abandoned changes, is yet to be set, and is not expected to come before February or March. 

Will it be third time lucky? Let’s hope the cat herders won’t need their nine lives.