Solana Went Down Again—It Likely Won't Be the Last Time

Solana just had another major outage—and that’s OK, the developers behind the network say. It’s likely going to happen again.

That’s what happens when you’re building on the bleeding edge of tech, and at least one prominent builder on the network—Helius Labs co-founder and CEO Mert Mumtaz—says he’s prepared to take the heat that comes with it.

“What we can't do is be afraid of progress for the sake of not being harassed on Twitter,” Mumtaz told Decrypt.

Solana nearly made it a full year without a major outage—a milestone that has proven evasive amid the network’s rise, and one that will remain out of reach for at least another 12 months following Tuesday’s morning’s stretch of downtime.

Solana Price Rebounds as Network Restarts After Outage

Solana’s price rebounded late Tuesday morning after the network recovered from an outage which had, hours prior, triggered a dip in the cryptocurrency’s price. SOL is currently up to $95.90, an almost full recovery to the position of $96.02 that the token held at 5:00 am EST this morning, prior to the network’s failure. Around 5:22 am EST, Solana developers posted that the network had suffered a “major outage.” Immediately, SOL began plummeting, hitting a low of $93.36 minutes later. All in all,...

The network went offline for nearly five hours Tuesday, according to data from the official Solana Status tracker. The previous outage, on February 25, 2023, lasted for nearly 19 hours and was later blamed on “custom block-forwarded software [that] inadvertently transmitted a huge amount of data,” according to the Solana Foundation.

Tuesday’s outage doesn’t have an official cause yet, but a tweet from Matthew Sigel—VanEck’s Head of Digital Assets Research—that’s been circulated by Solana leaders and developers blames a bug that had been fixed on Solana’s testnet, but not yet deployed to the mainnet.

“This bug was noticed on testnet and a fix was created, but not yet implemented due to testing,” Sigel wrote.

It’s not just a piece of a working explanation behind Solana’s latest network outage, but also a hint at a development ethos that prioritizes network safety and security over perfect uptime. It’s a comment that has been shared in the past by Solana co-founder Anatoly Yakovenko, Mumtaz, and other notable Solana builders.

While rare and relatively short-lived in the grand scheme of things, Solana’s outages have become a key part of the narrative around the blockchain. It’s also an easy target for critics who claim that the chain can’t compete with Ethereum or other notable rivals, despite Solana’s growing use by traders and developers alike thanks to cheap, fast transactions.

 gm from Decrypt

Solana Co-Founder Says 'Long-Term Fix' to Network Outages Is in the Works

Solana co-founder Anatoly Yakovenko knows that Solana’s outages concern its users. But a solution for the network could be on the horizon. “This has been the biggest challenge for us, and the number one priority,” Yakovenko told Decrypt on the most recent episode of the gm podcast. The young proof-of-history, proof-of-stake hybrid blockchain has had five major outages since its launch in 2020. Out of those five, three have occurred this year—each one the result of either bugs in Solana’s code o...

Outages are disappointing, no doubt, and Yakovenko has spoken in the past about finding ways to overcome them for good. Even so, the Solana co-founder and others in the space have been candid about saying that outages may still happen even amid the renewed focus on stability, thanks to that ethos of focusing on long-term security.

“If we are truly in the early innings of crypto (and I believe we are), we need to improve these systems as fast as possible so we are ready when the time comes,” said Mumtaz, who’s built up a reputation for being quick to defend Solana from critics and “FUD.”

“Short-term it's risky and painful, but long-term things like this make the systems more resilient and anti-fragile. Same as any evolutionary system,” he said.

One prominent solution that Solana backers keep floating is the upcoming launch of Firedancer, a second validator client for the network that would exist separately from the current one that Solana Labs built. Announced in August 2022, Firedancer is expected to improve Solana performance and boost decentralization, all while providing a fallback for the network if the Labs validator hits an issue. 

It won’t be a matter of flipping a switch on Firedancer and expecting Solana to be failure-free going forward, however. Yakovenko tweeted in December that users running Solana validators are likely to operate both clients, and would act to stop the network if inconsistencies emerge.

Jump Crypto Plans New Solana Validator Client to Boost Performance, Decentralization

In an effort to improve the decentralization and performance of the Solana network, Web3 infrastructure developer and investor Jump Crypto has partnered with the non-profit Solana Foundation to create a new, additional validator client for the blockchain platform. Jump Crypto will lead development of a new open-source validator client called Firedancer, which will be based on the C++ programming language. It is intended to coexist alongside Solana Labs’ existing validator client, which is based...

“I actually think that Solana validator culture is to pick safety over liveness, and people are much more likely to run a setup where if Firedancer and Labs disagree, halt both,” he wrote in reply to a thread by Dragonfly Capital Managing Partner Haseeb Qureshi. “Outages suck, but state corruption is catastrophic.”

Decrypt reached out to Solana to seek comment from Yakovenko about Tuesday’s outage and the impending Firedancer launch, but did not immediately receive a response.

Firedancer could have its own hitches during rollout, too, and it may take time to work through them. But Mumtaz said he believes that the second validator client will eventually help boost the reliability of the Solana network as it continues to expand.

“Once Firedancer goes live, there will likely be some unforeseen issues in mainnet. That's just how software goes; humans make mistakes,” he told Decrypt. “But after a few iterations, we will likely set up automatic failovers, and the probability of failures will go down drastically over time.”

Edited by Guillermo Jimenez