Worker Heartbeating

[yuandrew]

What was changed

Implements periodic worker heartbeat RPC that reports worker status, slot usage, poller info, host metrics, and sticky cache counters to the Temporal server. Includes HeartbeatManager, PollerTracker, and integration tests covering all heartbeat fields.

• HeartbeatManager — Per-namespace heartbeat scheduler. Workers register/unregister; scheduler fires at the configured interval. Gracefully shuts down if server returns UNIMPLEMENTED.
• PollerTracker — Tracks in-flight poll count and last successful poll time per worker type. Only records success when a poll returns actual work.
• WorkflowClientOptions.workerHeartbeatInterval — New option to configure heartbeat interval. Defaults to 60s. Can be set between 1-60s, or a negative duration to disable.
• Worker.getActiveTaskQueueTypes() — Reports WORKFLOW, ACTIVITY, and NEXUS (only when Nexus services are registered, matching Go SDK).
• Worker.buildHeartbeat() — Assembles the full WorkerHeartbeat proto with slot info, poller info, host metrics, sticky cache counters, and timestamps.
• Shutdown: Each worker sends ShutdownWorkerRequest with a final SHUTTING_DOWN heartbeat and active task queue types.
• NexusWorker task counters — Fixed totalProcessedTasks/totalFailedTasks to be properly incremented.
• TrackingSlotSupplier.getSlotSupplierKind() — Reports FixedSize vs ResourceBased in heartbeats.

Why?

New feature!

Checklist

1. Closes Worker Heartbeating #2716

2. How was this tested:

• WorkerHeartbeatIntegrationTest (12 tests): End-to-end against test server — basic fields, slot info, task counters, poller info, failure metrics, interval counter reset, in-flight slot tracking, sticky cache counters/misses, multiple workers, resource-based tuner, shutdown status, disabled heartbeats
• HeartbeatManagerTest: Unit tests for scheduler lifecycle, UNIMPLEMENTED handling, exception resilience
• PollerTrackerTest: Unit tests for poll tracking and snapshot generation
• TrackingSlotSupplierKindTest: Slot supplier kind detection
• WorkerHeartbeatDeploymentVersionTest: Deployment version in heartbeats
• All existing tests pass (./gradlew :temporal-sdk:test + spotlessCheck)

3. Any docs updates needed?

---

Note

Medium Risk
Adds a new periodic heartbeat RPC path and threads that run in production workers, and changes worker shutdown requests/poller instrumentation, which could affect load and shutdown behavior if misconfigured or server capability detection is wrong.

Overview
Implements periodic worker heartbeating to Temporal Server: workers now register a per-namespace scheduled heartbeat callback that reports status plus runtime stats (slot usage, poller counts/last success, sticky cache hit/miss counts, host metrics, plugin info, and optional deployment version) and disables itself on UNIMPLEMENTED.

Introduces new tracking primitives (PollerTracker, TaskCounter, NamespaceCapabilities) and wires them through workflow/activity/nexus pollers and task handlers so heartbeats can report in-flight pollers and processed/failed task counts; TrackingSlotSupplier now exposes supplier kind and used-slot count for reporting.

Extends WorkflowClientOptions with experimental workerHeartbeatInterval (default 60s, 1–60s allowed, negative disables) and updates worker shutdown to send richer ShutdownWorkerRequest including taskQueue, workerInstanceKey, active task queue types, and a final shutting down heartbeat. CI dev-server startup enables frontend.ListWorkersEnabled to support new worker listing tests.

^{Reviewed by Cursor Bugbot for commit ff47c66. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Sushisource]

All the workers now have something like this - can we abstract these out into something else? Seems like we could use a shared interface or base class.

[yuandrew]

moved into a new TaskCounter class

[Sushisource]

In my PR I've bundled a bunch of capability stuff up into a class, FYI, this will want to go in there

[yuandrew]

Thanks for the heads up, had Claude pull in your new class, so merge conflict should be minimal now. Feel free to merge first

[Sushisource]

Needs a review tho!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit ff47c66. Configure here.}

[cursor]

HeartbeatManager UNIMPLEMENTED handling prevents proper shutdown cleanup

Low Severity

When the server returns UNIMPLEMENTED, heartbeatTick calls shuttingDown.set(true) and scheduler.shutdown() (graceful). Later, when HeartbeatManager.shutdown() is called, SharedNamespaceWorker.shutdown() uses compareAndSet(false, true) which fails since shuttingDown is already true, causing it to return immediately without calling shutdownNow() or awaitTermination(). This means HeartbeatManager.shutdown() won't wait for the scheduler thread to fully terminate.

Additional Locations (1)

• temporal-sdk/src/main/java/io/temporal/internal/worker/HeartbeatManager.java#L128-L137

 

^{Reviewed by Cursor Bugbot for commit ff47c66. Configure here.}