Hetzner Online GmbH, a German cloud service provider, moved to block all Solana activity on its servers Wednesday, instantly taking over 1,000 Solana validators offline. While the affected nodes are largely back online, the action triggered a broader discussion in the crypto community about the increasing influence of centralized service providers in controlling the fate of supposedly-decentralized blockchain networks.
Approximately 40% of the Solana network’s validators—and 20% of the network's total stake—were housed on Hetzner as recently as August, according to independent analysis by Google Cloud engineer Sam Padilla.
Okay, we need to talk about Hetzner banning @solana nodes.
I monitored the infrastructure decentralization of Solana throughout all of August via a tool that I built.
~40% of Solana validators live on hetzner, adding up to ~20% of total network stake.
While the event did not crash Solana—that would have required taking 39% of the network’s total stake offline, halting its consensus mechanism and cutting off all network activity—it did highlight the increasing dependence of networks like Solana on individual service providers like Hetzner.
Hetzner, along with centralized cloud services providers Equinix and Amazon Web Services (AWS), hosted a whopping 65% of Solana’s stake in August, according to Padilla. A decision by those companies to deny service to Solana validators would instantly derail and offline the network—which touts itself as “the most censorship resistant blockchain network in the world.”
(3/10) The highest percentage of validators was concentrated on... you guessed it, Hetzner.
HTZ led the charge in validators, with ~40% of validators and 20% of the stake.
Other points of concern in terms of stake were Equinix and AWS.
In the hours following Wednesday’s ban, Solana validators—who stake large quantities of SOL with the network to earn more SOL, and keep Solana’s proof of stake consensus mechanism operating by processing blocks of the network’s transactions—raced to find new service providers and get their stake back online.
A representative of the Solana Foundation told Decrypt that Hetzner’s ban would not meaningfully impact Solana, as validators have many alternative options for service providers.
AD
AD
“This decision by Hetzner was certainly unfortunate, but not entirely unexpected,” the representative said. “There are hundreds of data centers that Solana operates out of, so those impacted are migrating to new locations.”
“Operators either managed to move or the stake was deactivated,” a separate source told Decrypt on Thursday.
The representative also pointed to the Solana Foundation’s Server Program initiative, which pairs validators with vetted and approved independent service providers in an effort to further decentralize the Solana network.
Despite such efforts, though, a majority of Solana validators still depended on centralized companies like Hetzner, Equinix, and AWS for their hosting needs.
“It was super easy to get set up on Hetzner, and as a result, many low-quality validators just defaulted to using Hetzner as it was quick and simple for them, even if it wasn’t the best choice,” the anonymous founder of Solana staking platform SolBlaze told Decrypt.
Hetzner, on its part, has long signaled its hostility towards crypto. In August, the company began banning Ethereum nodes operating on its servers, and signaled as bluntly as possible to its users that “using our products for any application related to [crypto] mining, even remotely related, is not permitted. This includes Ethereum. It includes proof of stake and proof of work and related applications. It includes trading.”
I know this is off topic, but if there are Hetzner customers who are using Ethereum, they should please read this: https://t.co/Lhlv3KAtUZ --Katie
That statement, made on August 23, would have applied to Solana proof of stake validating. And yet, a substantial percentage of Solana validators continued to utilize Hetzner servers.
AD
AD
“This is not a surprise at all. Hetzner has been advising about censoring crypto nodes for a very long time and this finally happened,” Guilherme Soubihe, CEO of internet service provider latitude.sh, told Decrypt.
“It seems Hetzner doesn’t want Solana servers on its infrastructure because [validating] is too intensive for its server disks, which tend to fail after a few months of use,” said Soubihe. “There are bare metal providers, like us, that are crypto friendly and would never censor crypto.”
So-called bare metal service providers rent entire physical servers to customers, unlike cloud servers, which host multiple tenants.
In the aftermath of Wednesday’s ban, validators of other blockchain networks that rely heavily on Hetzner, including Cardano and Cosmos, began searching for new service providers, in anticipation of similar bans on their nodes that could seriously impact network functionality.
Hetzner just took out all Solana validators on their network.
— St₳kΣ with Pride 🌈 Hosky Rug Pool & Ardana ISPO (@StakeWithPride) November 2, 2022
The move by Hetzner comes during a period when users are re-evaluating the dependence of blockchain networks—particularly proof of stake networks, which depend on ample server space to function—on centralized service providers.
Wednesday’s events similarly brought about another wave of criticism directed at networks like Solana and Cardano, and their dependence on centralized service providers.
