CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the `consensus` logging module to "debug" level (should not happen in production), and setting the log output format to JSON. The second is via RPC `dump_consensus_state`.
Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.
The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated.
As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).
This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don't set the log output to "json", leave at "plain", or don't set the consensus logging module to "debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC `dump_consensus_state`), do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in one's nginx setup).
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the `consensus` logging module to "debug" level (should not happen in production), and setting the log output format to JSON. The second is via RPC `dump_consensus_state`.
Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.
The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated.
As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).
This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don't set the log output to "json", leave at "plain", or don't set the consensus logging module to "debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC `dump_consensus_state`), do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in one's nginx setup).
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places. The first is via logs, setting the `consensus` logging module to "debug" level (should not happen in production), and setting the log output format to JSON. The second is via RPC `dump_consensus_state`.
Case 1, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.
The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated.
As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with two nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).
This regression was fixed in versions 0.34.29 and 0.37.2. Some workarounds are available. For case 1 (hitting the deadlock via logs), either don't set the log output to "json", leave at "plain", or don't set the consensus logging module to "debug", leave it at "info" or higher. For case 2 (hitting the deadlock via RPC `dump_consensus_state`), do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in one's nginx setup).
### Impact
An internal modification to the way struct `PeerState` is serialized to JSON introduced a deadlock when new function MarshallJSON is called. This function can be called from two places:
1. Via logs
* Setting the `consensus` logging module to "debug" level (should not happen in production), and
* Setting the log output format to JSON
2. Via RPC `dump_consensus_state`
Case 1 above, which should not be hit in production, will eventually hit the deadlock in most goroutines, effectively halting the node.
In case 2, only the data structures related to the first peer will be deadlocked, together with the thread(s) dealing with the RPC request(s). This means that only one of the channels of communication to the node's peers will be blocked. Eventually the peer will timeout and excluded from the list (typically after 2 minutes). The goroutines involved in the deadlock will not be garbage collected, but they will not interfere with the system after the peer is excluded.
The theoretical worst case for case 2, is a network with only two validator nodes. In this case, each of the nodes only has one `PeerState` struct. If `dump_consensus_state` is called in either node (or both), the chain will halt until the peer connections time out, after which the nodes will reconnect (with different `PeerState` structs) and the chain will progress again. Then, the same process can be repeated.
As the number of nodes in a network increases, and thus, the number of peer struct each node maintains, the possibility of reproducing the perturbation visible with 2 nodes decreases. Only the first `PeerState` struct will deadlock, and not the others (RPC `dump_consensus_state` accesses them in a for loop, so the deadlock at the first iteration causes the rest of the iterations of that "for" loop to never be reached).
This regression was introduced in versions `v0.34.28` and `v0.37.1`, and will be fixed in `v0.34.29` and `v0.37.2`.
### Patches
The PR containing the fix is [here](https://github.com/cometbft/cometbft/pull/865), and the corresponding issue is [here](https://github.com/cometbft/cometbft/pull/863)
### Workarounds
For case 1 (hitting the deadlock via logs)
* either don't set the log output to "json", leave at "plain",
* or don't set the consensus logging module to "debug", leave it at "info" or higher.
For case 2 (hitting the deadlock via RPC `dump_consensus_state`)
* do not expose `dump_consensus_state` RPC endpoint to the public internet (e.g., via rules in your nginx setup)
### References
* [Issue](https://github.com/cometbft/cometbft/pull/863) that introduced the deadlock
* [Issue](https://github.com/cometbft/cometbft/pull/524) reporting the bug via logs
An internal modification to the way PeerState is serialized to JSON introduced a deadlock when the new function MarshalJSON is called.
This function can be called in two ways. The first is via logs, by setting the consensus logging module to "debug" level (which should not happen in production), and setting the log output format to JSON. The second is via RPC dump_consensus_state.
CISA KEV lists vulnerabilities known to be exploited. FIRST EPSS estimates the likelihood of exploitation in the next 30 days. An unavailable score does not mean zero risk.
Which CVEs have scores here?
This view includes predictions for CVE identifiers explicitly listed in public NuGet advisories. Microsoft release pages have their own prediction coverage. Check source status for the scope and latest checks.
No exploitation or prediction data is available here for these CVEs. This does not mean they are safe or unlikely to be exploited.
Risk history
Compare earlier KEV listings and EPSS scores. Advisory wording and affected versions have a separate history.
When does an EPSS score appear as a change?
A prediction change is reported when EPSS rises from below 10% to 10% or higher. Every daily score change from the last 30 days stays in this history. Before that, it keeps KEV changes and moves between EPSS bands (0.1%, 1%, 5%, 10%, 20%, 50% and 90%).
Advisory history
15 saved versions
See how the publisher's advisory changed over time. Each link opens the version saved at that time. The publisher's update date can differ from the date shown here.
How advisory history works
Each link identifies one complete publisher record. History can include advisories that previously shared an identifier; an identifier removed by a publisher no longer establishes a current relationship. A historical link shows only that record and earlier saved versions. Failed checks and unchanged reads do not create new versions.
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…
CometBFT is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine and replicates it on many machines. An internal modification made in versions 0.34.28 and 0.37.1 to the way struct `PeerState` is serialized to JS…