Today’s Arbitrum Sequencer Downtime: What Happened?
Earlier today, we experienced Sequencer downtime, which made the Arbitrum One network temporarily unavailable. The issue has now been fully resolved, and the Arbitrum Sequencer as well as all public RPC nodes are fully operational. We thank everyone for their patience as we diagnosed this issue, and apologize for inconveniences that this caused.
The core issue was a hardware failure in our main Sequencer node. While we generally have redundancies that would allow a backup Sequencer to seamlessly take control, these also failed to take effect this morning due to a software upgrade in process. As a result, the Sequencer stopped processing new transactions. We’d like to stress that Arbitrum is a Layer 2 network on Ethereum, and the network is designed to sustain even permanent Sequencer failures by falling back to using Ethereum to process transactions.
In standing the network back up, we moved cautiously and deliberately to make sure that all transactions that had been confirmed by the Sequencer before going offline were preserved and that there would be no re-org. The Arbitrum Sequencer posts transaction batches every few minutes on Ethereum, and there were 284 transactions that had been accepted by the Sequencer before it went down that had not yet been posted on-chain. We partially restarted the Sequencer to first post these transactions, and only once they were confirmed did we fully restore the Sequencer to accept new transactions.
The stability and uptime of the network is a top priority for us, and we will work to implement additional redundancies to increase stability going forward. While we will spare no effort to maximize uptime, the strongest guarantees will come once Arbitrum is fully decentralized. The Arbitrum network is still in beta, and we will keep this moniker as long as there are points of centralization that still exist in the system. The Arbitrum Sequencer is currently run by Offchain Labs, but the good news is that we have a deeply researched and credible path to fully decentralize it. In the coming days, weeks, and months, we will continue on this twofold path of minimizing Sequencer downtime, and at the same time achieving the ultimate goal of full decentralization.
