Prysm Updates Q2'23

Published in

Offchain Labs

8 min readMay 11, 2023

Hi Friends — welcome to the latest installment of our quarterly Prysm updates! Here’s a recap of some roses and thorns from Capella along with some insight into what the Prysm team’s working on now. Also, if you’re a validator using Prysm—don’t miss our UI feedback link at the end of the article!

Capella Retrospective
Deneb Now
Projects Underway
01 Hardening on MEV-boost & builder testing
02 Parallel beacon block construction
03 Reduce process attestation time
04 Backfilling blocks syncing
05 Methodical SSZ implementation
06 Extensive API support
07 Prysm gRPC remote signer removed in favor of the Web3Signer interface
08 The Future of Prysm UI

Capella Retrospective

We survived Capella! 🎊
Ethereum mainnet successfully transitioned from Bellatrix to Capella at 10:27 PM UTC on April 12, 2023, at epoch 194,048. What’s a hard fork without some bumps along the way? The Prysm team certainly got their share! Here is a summary of the main challenges the Prysm team faced:

Validators failed to propose beacon blocks using MEV-boost.
This was caused by the missing inclusion of bls-to-exec changes when submitting a blind block to the relayers. This was fixed in release v4.0.2. Full details and corrections actions were outlined in our post-mortem.
Validators failed to delete keys via key manager API.
This is not a Capella-related bug and may have happened before Capella. When attempting to delete keys from the validator keystore, such a process may only partially delete the Keystore in the memory. This was fixed in release v4.0.3. Full details are outlined in Lido’s post-mortem.
Various validator bugs when interacting with MEV-boost: one with panic when using a unique combination of builder flags, one with configuration checks, and one with builder bid checks. All these bugs were fixed in v4.0.3.

There were many things that were learned from the aforementioned incidents. We came up with a priority list of items to focus on in order to prevent similar issues from happening in the later network fork, some of which we’ll touch on in the following sections.

Overall, it was a successful upgrade amid the turbulences the Prysm team faced. On the day of the transition, the block proposal rate remained above 90%, and the attestation participation remained above 98%. We are very happy to be on the other side now. This is a testament to how strong Ethereum is and how valuable client diversity is 🎉

https://beaconcha.in/charts/participation_rate

Deneb Now

With Capella done, the client teams have been gearing up to focus on the next great network upgrade — Deneb. EIP-4844 will be the main feature in Deneb. In addition, there will be other smaller scoped EIPs, primarily on the execution layer side. As a consensus layer client, we won’t be focusing on those EIPs, but feel free to read the below:

EIP-6780: SELFDESTRUCT only in the same transaction
EIP-1153: Transient storage opcodes
EIP-6475: New SSZ type to represent optional values

Consensus client teams have been hard at work on EIP-4844: we just delivered Devnet #5 a few weeks ago. This Devnet highlighted features on separated blobs decoupled from the block with its independent gossip net. Blocks by ranging syncing and blobs by range and root syncing were tested. The devnet also utilized a new execution API that combined the payload and blobs bundled into a single API call. Lastly, there is a new crypto API to verify KZG proof of the blob.

Although devnet helps discover research spec and implementation gaps, participating in a devnet doesn’t mean we’re ready to ship Deneb anytime soon. The current client implementation is nowhere near production quality, and it often takes client implementers many months to reach that stage. Aside from consensus layer client implementations, here are some ongoing topics for those who might be interested:

Choosing blob data gas parameters.
What should be the min and the increase decrease factors?
https://ethresear.ch/t/eip-4844-fee-market-analysis/15078/12
How to handle reorg’ed blob transactions? https://github.com/ethereum/execution-apis/issues/333
How to define blob transaction signature schemes? https://hackmd.io/nz-IqXLPQ-yahFPFlPl62A

We hope you don’t think EIP-4844 is all we are working on now until Deneb — many Prysm improvements and features are coming in the next release and Q3, which we’ll highlight below!

1. Hardening MEV-boost & builder testing

Close to 85% of the mainnet validators use MEV-boost, which means sufficient time and effort must be put into MEV-boost testing from a consensus layer client’s perspective.

Relayers and builders will always be part of the consensus — whether we like it or not — until we reach in-protocol proposer-builder separation. It’s important to recognize such facts. Since the Capella incidents, we’ve been working on the following:

Adding more extensive unit tests for builder paths
Adding end-to-end tests for builder paths
Working closely with Hive tests to cover specific builder scenarios. Even considering integrating Hive tests as part of Prysm CI/CD
Operating more testnet validators with MEV-boost to mimic the 85% mainnet condition

2. Parallel beacon block construction

When a validator requests a beacon block from the node to sign over, the beacon node builds the beacon block by constructing multiple parts of the block. The current construction flow is done sequentially and takes longer than desired when using MEV-boost. This could lead to the relayer refusing to return the payload because it’s outside of the 4s window or the block is getting reorged. A straightforward optimization we’ve implemented is to build the block in parallel. Mainly building the consensus part and the execution part in parallel. We saw good results in the local setup that reduced block construction time by as much as 40%. This feature will be available in the upcoming v4.0.4 release, and we encourage stakers to try it out.

3. Reduce process attestation time

Before updating the canonical head of the chain, the beacon node must process all the new attestations. The attestations come in various unaggregated, aggregated, and block forms. This becomes an expensive process when a node subscribes to more beacon attestation subnets (i.e. serving more validators). If the process attestation time is slow, it’ll lead to the node not getting the correct head on time, which causes validators to vote for the incorrect head and build a block off the incorrect parent. We’re currently looking into methods to reduce batching and processing attestation time and are seeing positive results. Batching unaggregated attestations could result in as much as 50% improvement. We’ll have more updates on this soon!

4. Backfilling blocks syncing

We are working on backfilling blocks if a node uses weak subjectivity syncing. This ensures your Prysm beacon node is a good citizen and will serve peer blocks up to weak subjectivity period. Backfilling behavior applies to blobs as well as blocks. For blocks, it’ll be maintained for up to four months, while blobs will be maintained for up to eighteen days. Many conversations go into designing backfilling. We want good performance without compromising on the Prysm node’s reliability. If not implemented carefully, a node could run out of bandwidth or get rate limited. Once this feature is ready, we’ll look for beta testers so expect more updates from us here!

5. Methodical SSZ implementation

FastSSZ is the library currently used by Prysm and flashbots MEV code to generate golang code for SSZ support. With SSZ eventually coming to the Execution Layer, we realized we needed to invest in a code generation tool that could scale to more contributors and serve broader audiences in the Ethereum community. So we have open-sourced the methodical-ssz code generation library in alpha. We have been collaborating with members of the Geth team, who have leveraged their expertise from working with RLP to make contributions, improving the codebase, and fixing compatibility issues with their EIP-4844 transaction code. We’re also working with Flashbots developers to find more corner cases and serve broader use cases. Methodical-ssz currently generates fastSSZ compatible marshaling, unmarshaling, and merkleization method sets from the Prysm codebase, as well as spec tests (which pass!). In the future, we plan to build more merkleization and proof generation features in the library and explore further optimizations. We are open to contributors and feedback from anyone interested in using it.

6. Extensive API support

API is how stakers and validators interact with the node. Having a complete set of support for API is crucial, and the Prysm team is working towards that goal. We recognize there are a broad set of API consumers ranging from stakers, block builders, relayers, and service providers. Prysm currently implements a custom set of APIs under gRPC, and we are looking into providing a more extensive set of APIs under REST that can be used for performance monitoring, UI, and more. We want to hear all your feedback! If there are API features that you want to see, please open an issue.

7. Prysm gRPC remote signer removed in favor of the Web3Signer interface

Prior to the Beacon API standard, Prysm supported an interface for gRPC-based remote signing. Because Prysm is the only consensus client that supports gRPC, the remote signer interface could only be used by Prysm. We have since supported the Web3signer interface, which is supported by all consensus clients for remote signing and is REST-based. We have fully removed the Prysm remote signer in v4.

8. The Future of Prysm UI

The current form of our web UI is marked for deprecation, but nothing has been removed yet. Prysm was the first client to natively support a web interface; however, since the release of the Keymanager APIs and having consensus clients support generic web UIs, users may no longer benefit as much from a natively supported UI. During this period, we are re-evaluating how the Prysm Web UI can provide value, either in a new form that’s slimmed down to core or essential features, or completely separated out of Prysm. Your feedback is a crucial part of our decision-making process, so please share your thoughts in our Reddit discussion!