Skip to content

Defer ChainMonitor updates and persistence to flush()#4351

Merged
TheBlueMatt merged 5 commits intolightningdevkit:mainfrom
joostjager:chain-mon-internal-deferred-writes
Mar 18, 2026
Merged

Defer ChainMonitor updates and persistence to flush()#4351
TheBlueMatt merged 5 commits intolightningdevkit:mainfrom
joostjager:chain-mon-internal-deferred-writes

Conversation

@joostjager
Copy link
Contributor

@joostjager joostjager commented Jan 27, 2026

Summary

Modify ChainMonitor internally to queue watch_channel and update_channel operations, returning InProgress until flush() is called. This enables persistence of monitor updates after ChannelManager persistence, ensuring correct ordering where the ChannelManager state is never ahead of the monitor state on restart. The new behavior is opt-in via a deferred switch.

Key changes:

  • ChainMonitor gains a deferred switch to enable the new queuing behavior
  • When enabled, monitor operations are queued internally and return InProgress
  • Calling flush() applies pending operations and persists monitors
  • Background processor updated to capture pending count before ChannelManager persistence, then flush after persistence completes

Performance Impact

Multi-channel, multi-node load testing (using ldk-server chaos branch) shows no measurable throughput difference between deferred and direct persistence modes.

This is likely because forwarding and payment processing are already effectively single-threaded: the background processor batches all forwards for the entire node in a single pass, so the deferral overhead doesn't add any meaningful bottleneck to an already serialized path.

For high-latency storage (e.g., remote databases), there is also currently no significant impact because channel manager persistence already blocks event handling in the background processor loop (test). If the loop were parallelized to process events concurrently with persistence, deferred writing would become comparatively slower since it moves the channel manager round trip into the critical path. However, deferred writing would also benefit from loop parallelization, and could be further optimized by batching the monitor and manager writes into a single round trip.

Alternative Designs Considered

Several approaches were explored to solve the monitor/manager persistence ordering problem:

1. Queue at KVStore level (#4310)

Introduces a QueuedKVStoreSync wrapper that queues all writes in memory, committing them in a single batch at chokepoints where data leaves the system (get_and_clear_pending_msg_events, get_and_clear_pending_events). This approach aims for true atomic multi-key writes but requires KVStore backends that support transactions (e.g., SQLite); filesystem backends cannot achieve full atomicity.

Trade-offs: Most general solution but requires changes to persistence boundaries and cannot fully close the desync gap with filesystem storage.

2. Queue at Persister level (#4317)

Updates MonitorUpdatingPersister to queue persist operations in memory, with actual writes happening on flush(). Adds flush() to the Persist trait and ChainMonitor.

Trade-offs: Only fixes the issue for MonitorUpdatingPersister; custom Persist implementations remain vulnerable to the race condition.

3. Queue at ChainMonitor wrapper level (#4345)

Introduces DeferredChainMonitor, a wrapper around ChainMonitor that implements the queue in a separate wrapper layer. All ChainMonitor traits (Listen, Confirm, EventsProvider, etc.) are passed through, allowing drop-in replacement.

Trade-offs: Requires re-implementing all trait pass-throughs on the wrapper. Keeps the core ChainMonitor unchanged but adds an external layer of indirection.

@ldk-reviews-bot
Copy link

ldk-reviews-bot commented Jan 27, 2026

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@joostjager
Copy link
Contributor Author

Closing this PR as #4345 seems to be the easiest way to go

@joostjager joostjager closed this Jan 27, 2026
@joostjager joostjager reopened this Feb 9, 2026
@joostjager joostjager force-pushed the chain-mon-internal-deferred-writes branch from 1f5cef4 to 30d05ca Compare February 9, 2026 14:45
@joostjager
Copy link
Contributor Author

The single commit was split into three: extracting internal methods, adding a deferred toggle, and implementing the deferral and flushing logic. flush() now delegates to the extracted internal methods rather than reimplementing persist/insert logic inline. Deferred mode is opt-in via a deferred bool rather than always-on. Test infrastructure was expanded with deferred-mode helpers and dedicated unit tests.

@joostjager joostjager force-pushed the chain-mon-internal-deferred-writes branch 9 times, most recently from 2815bf9 to 3eb5644 Compare February 11, 2026 09:37
@codecov
Copy link

codecov bot commented Feb 11, 2026

Codecov Report

❌ Patch coverage is 93.04556% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (62c7575) to head (52081b6).
⚠️ Report is 70 commits behind head on main.

Files with missing lines Patch % Lines
lightning/src/chain/chainmonitor.rs 90.22% 22 Missing and 4 partials ⚠️
lightning/src/util/test_utils.rs 92.85% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4351      +/-   ##
==========================================
+ Coverage   85.87%   86.10%   +0.23%     
==========================================
  Files         157      159       +2     
  Lines      103769   105410    +1641     
  Branches   103769   105410    +1641     
==========================================
+ Hits        89115    90767    +1652     
+ Misses      12158    12127      -31     
- Partials     2496     2516      +20     
Flag Coverage Δ
tests 86.10% <93.04%> (+0.23%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@joostjager joostjager force-pushed the chain-mon-internal-deferred-writes branch 10 times, most recently from f964466 to b140bf9 Compare February 12, 2026 08:22
@joostjager joostjager marked this pull request as ready for review February 12, 2026 10:56
@joostjager
Copy link
Contributor Author

This PR is now ready for review. LDK-node counterpart: lightningdevkit/ldk-node#782

@joostjager joostjager force-pushed the chain-mon-internal-deferred-writes branch from 2205fb5 to 710c9fe Compare March 17, 2026 20:51
@joostjager
Copy link
Contributor Author

joostjager commented Mar 17, 2026


// Flush monitor operations that were pending before we persisted. New updates
// that arrived after are left for the next iteration.
chain_monitor.get_cm().flush(pending_monitor_writes, &logger);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If kv_store.write().await fails (returns Err), the ? short-circuits and flush() is never called. The pending monitor operations remain in the queue and will be re-attempted on the next iteration — this is fine.

However, if flush() itself panics (e.g., from the debug_assert at flush count mismatch, or from a poisoned lock), the entire async future panics without returning Err, which could crash the background processor rather than triggering a graceful shutdown. Consider wrapping flush in a way that converts panics to errors, or at least document that flush is not expected to fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we document things like that for debug asserts

Comment on lines +1581 to +1594
debug_assert!(
{
let monitors = self.monitors.read().unwrap();
let in_monitors = monitors.contains_key(&channel_id);
let in_pending = pending_ops.iter().any(|op| match op {
PendingMonitorOp::NewMonitor { channel_id: id, .. } => *id == channel_id,
_ => false,
});
in_monitors || in_pending
},
}
"ChannelManager generated a channel update for a channel that was not yet registered!"
);
pending_ops.push_back(PendingMonitorOp::Update { channel_id, update: update.clone() });
ChannelMonitorUpdateStatus::InProgress
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In release builds, this debug_assert is stripped, so an update_channel call for a channel_id that exists in neither self.monitors nor pending_ops will silently queue the update. When later flushed, update_channel_internal will hit the monitors.get(&channel_id)None branch and return InProgress without actually applying the update — a silent data loss.

Consider making this a hard check (or at least logging a warning) in release mode, consistent with how the non-deferred path handles this in update_channel_internal.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked about this. We don't really support this and return InProgress as before.

Comment on lines 1548 to +1552
fn watch_channel(
&self, channel_id: ChannelId, monitor: ChannelMonitor<ChannelSigner>,
) -> Result<ChannelMonitorUpdateStatus, ()> {
let logger = WithChannelMonitor::from(&self.logger, &monitor, None);
let mut monitors = self.monitors.write().unwrap();
let entry = match monitors.entry(channel_id) {
hash_map::Entry::Occupied(_) => {
log_error!(logger, "Failed to add new channel data: channel monitor for given channel ID is already present");
return Err(());
},
hash_map::Entry::Vacant(e) => e,
};
log_trace!(logger, "Got new ChannelMonitor");
let update_id = monitor.get_latest_update_id();
let mut pending_monitor_updates = Vec::new();
let persist_res = self.persister.persist_new_channel(monitor.persistence_key(), &monitor);
match persist_res {
ChannelMonitorUpdateStatus::InProgress => {
log_info!(logger, "Persistence of new ChannelMonitor in progress",);
pending_monitor_updates.push(update_id);
},
ChannelMonitorUpdateStatus::Completed => {
log_info!(logger, "Persistence of new ChannelMonitor completed",);
},
ChannelMonitorUpdateStatus::UnrecoverableError => {
let err_str = "ChannelMonitor[Update] persistence failed unrecoverably. This indicates we cannot continue normal operation and must shut down.";
log_error!(logger, "{}", err_str);
panic!("{}", err_str);
},
if !self.deferred {
return self.watch_channel_internal(channel_id, monitor);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: load_existing_monitor() at line 1083 has a pre-0.1 fallback path that calls chain::Watch::watch_channel(self, ...), which dispatches here. In deferred mode, this queues the monitor in pending_ops and returns InProgress instead of actually loading it into self.monitors.

Since load_existing_monitor is called during node startup (before the background processor runs), pre-0.1 monitors will be stuck in the pending queue — invisible to release_pending_monitor_events, process_chain_data, list_monitors, etc. — until something explicitly calls flush().

The fallback should call self.watch_channel_internal directly to bypass the deferred queue, or the pre-0.1 path should be documented as incompatible with deferred mode.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to call watch_channel_internal

@joostjager joostjager force-pushed the chain-mon-internal-deferred-writes branch from 710c9fe to 8761e21 Compare March 18, 2026 07:46
@joostjager
Copy link
Contributor Author

Comment on lines +1308 to +1312
Err(()) => {
// `watch_channel` checks both `pending_ops` and `monitors`
// for duplicates before queueing, so this is unreachable.
unreachable!();
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unreachable!() here relies on an invariant that the channel_id was checked against both pending_ops and monitors before being queued. However, load_existing_monitor inserts directly into monitors without checking pending_ops. If load_existing_monitor were ever called while a deferred NewMonitor for the same channel_id is queued (however unlikely), this unreachable!() would panic in production.

Consider replacing with a proper error return + log_error! (matching the Err return from watch_channel_internal), or adding a doc comment on load_existing_monitor noting it must not be called while deferred ops are pending.

Suggested change
Err(()) => {
// `watch_channel` checks both `pending_ops` and `monitors`
// for duplicates before queueing, so this is unreachable.
unreachable!();
},
Err(()) => {
// This can only happen if the channel_id was inserted
// into `monitors` between queueing and flushing (e.g.,
// via `load_existing_monitor`).
log_error!(logger, "Flushed NewMonitor for channel that already exists in monitors");
debug_assert!(false, "Duplicate channel_id found during NewMonitor flush");
drop(queue);
continue;
},

Comment on lines +1300 to +1303
// Hold `pending_ops` across the internal call so that
// `watch_channel` (which checks `monitors` + `pending_ops`
// atomically) cannot race with this insertion.
match self.watch_channel_internal(channel_id, monitor) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When flush processes NewMonitor, it holds pending_ops (the queue guard) across the entire watch_channel_internal call, which includes persistence I/O and acquiring monitors.write(). For the Update case (line 1320), the lock is correctly released before I/O via drop(queue).

This asymmetry means a slow persist_new_channel blocks all concurrent deferred watch_channel and update_channel calls (even for unrelated channels) for the duration of the I/O. Consider whether the atomicity guarantee (preventing a duplicate watch_channel race) could instead be achieved by briefly holding both locks just for the monitors insertion, then releasing pending_ops before the persistence call. For example, insert a placeholder into monitors under both locks, release pending_ops, then persist.


// Flush monitor operations that were pending before we persisted. New updates
// that arrived after are left for the next iteration.
chain_monitor.get_cm().flush(pending_monitor_writes, &logger);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flush() is a synchronous method that may perform blocking persistence I/O (calls persist_new_channel / update_persisted_channel on the Persist impl). Calling it inside this async block means a sync Persist implementation will block the executor thread here.

The function-level doc comment on process_events_async mentions this, but a brief inline comment here would help developers reading the async block understand why this sync call exists inside an async context and that it's intentional.

Comment on lines +1589 to +1600
debug_assert!(
{
let monitors = self.monitors.read().unwrap();
let in_monitors = monitors.contains_key(&channel_id);
let in_pending = pending_ops.iter().any(|op| match op {
PendingMonitorOp::NewMonitor { channel_id: id, .. } => *id == channel_id,
_ => false,
});
in_monitors || in_pending
},
}
"ChannelManager generated a channel update for a channel that was not yet registered!"
);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In release builds, this debug_assert! is compiled out entirely, meaning update_channel unconditionally queues the update without validating the channel exists. If a bug elsewhere causes an update for an unregistered channel, it would be silently queued and only surface later during flush (where update_channel_internal returns InProgress in release and panic in debug for the same condition).

This matches the behavior of the non-deferred update_channel_internal (which also only has a debug_assertions panic for this case), so it's not a regression. But worth being aware of for debugging production issues — the error would be deferred from the call site to the flush site.

@joostjager
Copy link
Contributor Author

Claude bot is still quite repetitive, not making the overview over the PR better. It comes up with good points though.

Think I've now addressed all of its complaints one way or the other.

@joostjager joostjager requested a review from TheBlueMatt March 18, 2026 09:23
@TheBlueMatt
Copy link
Collaborator

Ugh, needs rebase somehow?

joostjager and others added 5 commits March 18, 2026 18:01
The previous wording implied that persisting a full ChannelMonitor
would automatically resolve all pending updates. Reword to make clear
that each update ID still needs to be individually marked complete via
channel_monitor_updated, even after a full monitor persistence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract the ChannelMonitor construction boilerplate that was
duplicated across channelmonitor test functions into a reusable
#[cfg(test)] pub(super) dummy_monitor helper, generic over the
signer type.

AI tools were used in preparing this commit.
Pure refactor: move the bodies of Watch::watch_channel and
Watch::update_channel into methods on ChainMonitor, and have
the Watch trait methods delegate to them. This prepares for adding
deferred mode where the Watch methods will conditionally queue
operations instead of executing them immediately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a `deferred` parameter to `ChainMonitor::new` and
`ChainMonitor::new_async_beta`. When set to true, the Watch trait
methods (watch_channel and update_channel) will unimplemented!() for
now. All existing callers pass false to preserve current behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the unimplemented!() stubs with a full deferred write
implementation. When ChainMonitor has deferred=true, Watch trait
operations queue PendingMonitorOp entries instead of executing
immediately. A new flush() method drains the queue and forwards
operations to the internal watch/update methods, calling
channel_monitor_updated on Completed status.

The BackgroundProcessor is updated to capture pending_operation_count
before persisting the ChannelManager, then flush that many writes
afterward - ensuring monitor writes happen in the correct order
relative to manager persistence.

Key changes:
- Add PendingMonitorOp enum and pending_ops queue to ChainMonitor
- Implement flush() and pending_operation_count() public methods
- Integrate flush calls in BackgroundProcessor (both sync and async)
- Add TestChainMonitor::new_deferred, flush helpers, and auto-flush
  in release_pending_monitor_events for test compatibility
- Add create_node_cfgs_deferred for deferred-mode test networks
- Add unit tests for queue/flush mechanics and full payment flow

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@joostjager joostjager force-pushed the chain-mon-internal-deferred-writes branch from 8761e21 to 3f1345b Compare March 18, 2026 17:05
@joostjager
Copy link
Contributor Author

Uses conflict. Rebased

Comment on lines +1308 to +1311
Err(()) => {
// `watch_channel` checks both `pending_ops` and `monitors`
// for duplicates before queueing, so this is unreachable.
unreachable!();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug (low severity): This unreachable!() is technically reachable if load_existing_monitor is called concurrently with flush for the same channel_id. load_existing_monitor bypasses the deferred queue and inserts directly into self.monitors, which would cause watch_channel_internal here to find an Occupied entry and return Err(()).

While this shouldn't happen in practice (existing monitors are loaded at startup before new channels are created), the unreachable!() is misleading. Consider replacing with a log_error! and continue (or at least changing to debug_assert!(false, ...) + graceful skip) so a future caller doesn't get a confusing panic.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh, come on.

Comment on lines +1296 to +1305
PendingMonitorOp::NewMonitor { channel_id, monitor } => {
let logger = WithChannelMonitor::from(logger, &monitor, None);
let update_id = monitor.get_latest_update_id();
log_trace!(logger, "Flushing new monitor");
// Hold `pending_ops` across the internal call so that
// `watch_channel` (which checks `monitors` + `pending_ops`
// atomically) cannot race with this insertion.
match self.watch_channel_internal(channel_id, monitor) {
Ok(status) => {
drop(queue);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit/design: For NewMonitor, pending_ops is held across watch_channel_internal which performs persistence I/O (persist_new_channel). This blocks all concurrent deferred watch_channel/update_channel calls (including unrelated channels) for the entire I/O duration.

For Update (line 1320), the lock is correctly dropped before I/O. The same approach could work here: pop the op, release pending_ops, call watch_channel_internal, then handle the result.

The stated reason for holding the lock ("so that watch_channel cannot race with this insertion") is to prevent a duplicate watch_channel from being enqueued for the same channel_id while the insertion is in progress. But this race is harmless — the duplicate would be caught by watch_channel_internal returning Err(Occupied) during its own flush. The simpler approach avoids blocking all channels on one channel's I/O.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean no we prefer to keep the assert there.

Comment on lines 1140 to +1156
@@ -1129,7 +1149,12 @@ where
CHANNEL_MANAGER_PERSISTENCE_KEY,
channel_manager.get_cm().encode(),
)
.await
.await?;

// Flush monitor operations that were pending before we persisted. New updates
// that arrived after are left for the next iteration.
chain_monitor.get_cm().flush(pending_monitor_writes, &logger);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Race window: pending_operation_count() is captured at line 1141, but encode() runs when the future is first polled (line 1150). Between these two points, ChannelManager handlers on other threads can queue new deferred monitor ops AND mutate the CM's internal state. The encode() output would include state corresponding to those new ops, but flush won't process them (count was lower).

This means: after this code runs, the persisted CM state may reference monitor updates that haven't been persisted yet. On crash recovery, the CM would detect the discrepancy and re-issue the missing updates — so this is safe (monitor-behind-CM is the safe direction). But it's worth documenting that the ordering guarantee is "eventually correct across iterations" rather than "strict per-iteration".

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be worth just moving the count fetch into the future, but this isn't worth documenting. we dont have to have a comment for every tiny detail.

Comment on lines +1262 to +1270
/// Returns the number of pending monitor operations queued for later execution.
///
/// When the `ChainMonitor` is constructed with `deferred` set to `true`,
/// [`chain::Watch::watch_channel`] and [`chain::Watch::update_channel`] calls are queued
/// instead of being executed immediately. Call this method to determine how many operations
/// are waiting, then pass the result to [`Self::flush`] to process them.
pub fn pending_operation_count(&self) -> usize {
self.pending_ops.lock().unwrap().len()
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation: list_monitors(), get_monitor(), list_pending_monitor_updates(), process_pending_events(), and rebroadcast_pending_claims() only iterate self.monitors — monitors still sitting in pending_ops are invisible to all of these. This is fine given the flush-before-use pattern, but the doc on pending_operation_count (or on those methods) should note that unflushed monitors won't appear.

Copy link
Contributor

@valentinewallace valentinewallace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAICT the only real change since my last ack is the flush lock, which looks good

@TheBlueMatt TheBlueMatt merged commit dcf0c20 into lightningdevkit:main Mar 18, 2026
17 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

weekly goal Someone wants to land this this week

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants