Skip to main content

Troubleshoot SDK Worker capacity problems

View Markdown

This guide covers the failure modes where Workers stop keeping up with their Task Queues: slots that never free up, pollers that stop polling, Tasks that sit in the queue, and a Sticky Execution cache that is not doing its job. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

For recommended alert thresholds and for durations, see SDK Worker alerting. For Worker sizing and tuning, see Worker performance and Worker deployment and performance.

These failure modes are causally linked. Slots fill up, which stops polling, which lets Tasks accumulate, which stops completions. When several fire at once, work backward to the earliest link in the chain rather than treating each as independent.

Worker Task slots exhausted​

Metric: temporal_worker_task_slots_available reaching 0, tagged worker_type

Every Task execution slot for this worker_type and Task Queue is occupied, so no new Tasks can be picked up. The SDK blocks before issuing its next poll until a slot is released.

info

This metric reports meaningful values only with fixed-size slot suppliers. It cannot be used with resource-based slot suppliers — see Slot availability metrics. If your Workers use resource-based tuning, skip this section and use Workflow Task schedule-to-start latency elevated as your signal that Worker capacity is short.

Why it matters. Slots stay occupied because existing Tasks are not completing. Until they free up, schedule-to-start latency rises and, in both the Go and Java SDKs, temporal_num_pollers drops to zero as a secondary effect, because those SDKs block on slot acquisition before incrementing the poller gauge.

Impact and remediation differ by worker_type, so check that tag on the firing alert. Values are WorkflowWorker, ActivityWorker, and LocalActivityWorker.

WorkflowWorker slots​

  1. Check Workflow Task execution latency. Sustained high temporal_workflow_task_execution_latency confirms something is holding slots. See Workflow Task execution latency high.
  2. Check Worker CPU. High CPU slows Workflow Task execution directly and keeps slots occupied longer.
  3. Check for blocking calls in Workflow code. A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no async def Workflow code is blocking the event loop.
  4. Check for throttling on respond operations. See RESOURCE_EXHAUSTED on respond operations — slots are not released until the respond call succeeds.

To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options.

ActivityWorker slots​

  1. Check Activity execution latency. Sustained high temporal_activity_execution_latency for the affected activity_type confirms Activities are holding slots longer than expected.
  2. Check Worker CPU. High CPU slows Activity execution directly.
  3. Check what the Activities are waiting on. If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency.
  4. Check for throttling on respond operations. See RESOURCE_EXHAUSTED on respond operations.

To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size — but only after confirming a downstream dependency is not the constraint.

LocalActivityWorker slots​

Local Activities run inside the Workflow Task execution loop, so blocked slots hold up the entire Workflow Task. The SDK responds by sending repeated Workflow Task heartbeats to keep the Task alive. If heartbeating continues past the Workflow Task heartbeat timeout — 30 minutes by default — the Temporal Service times out the Task and reschedules it, and the Local Activities re-execute from scratch.

  1. Check what the Local Activities are waiting on. The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected activity_type. If it calls a downstream service, check whether that service is slow or throttling.
  2. Check Worker CPU.
  3. Check the Local Activity latency alert. See Local Activity latency exceeds the heartbeat timeout — if that is firing alongside this, the heartbeat timeout is imminent or already exceeded and re-execution has begun.

All pollers disconnected​

Metric: temporal_num_pollers reaching 0, tagged poller_type and task_queue

No active pollers remain for this poller type and Task Queue. Workers have stopped polling entirely and Tasks are accumulating with nothing to process them.

info

This metric is tagged poller_type, not worker_type. The values are workflow_task, workflow_sticky_task, activity_task, and nexus_task. Alert on each poller type your Workers use, and note that a Worker with an active Sticky Execution cache reports workflow_sticky_task alongside workflow_task.

Why it matters. Workflow and Activity Tasks accumulate on the Temporal Service. At scale this grows into a large backlog that puts pressure on Task matching and persistence. Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed.

Triage.

  1. Check whether Worker processes are running. Check pod status, restart counts, and logs. Workers may have crashed, been evicted, or been OOM killed — this is the most common cause.
  2. Rule out scale-down. If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further.
  3. Check whether slots are exhausted. See Worker Task slots exhausted for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first.
  4. Check for authentication failures. Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry.
  5. Check for INTERNAL errors. See UNIMPLEMENTED or INTERNAL from the Temporal Service — sustained INTERNAL errors cause Workers to back off and stop polling.
Self-hosted Temporal Service

Cross-check the server-side concurrent poller count for this Task Queue. If it has also dropped, Workers have fully disconnected from the Service's perspective. For authentication failures, check unauthorized request counts and authorization system failures — a non-zero authorization system failure count means the auth plugin itself is failing, which is more urgent than an expired credential.

Task completions dropped to zero​

Metric: temporal_request rate reaching 0 on RespondWorkflowTaskCompleted or RespondActivityTaskCompleted

The SDK increments this counter only after a successful response, so a zero rate means no responses are landing.

A zero rate on its own does not prove a problem. It is equally consistent with an idle Task Queue, a Namespace outside business hours, or a batch workload between runs. This metric is tagged namespace and operation, not task_queue, so you cannot narrow it to a single Task Queue without correlating against another signal.

Treat it as actionable only alongside evidence of demand: a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency that keeps climbing. Without one of those, there is nothing for Workers to complete and the metric is reporting the truth.

Why it matters. When there is demand, no Workflow Task completions means Workflow Executions are not making progress. Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them.

No Activity Task completions means Workflows waiting on Activity results are stalled. Activities begin timing out against their scheduleToClose timeouts, and the Temporal Service retries them within the Retry Policy — but with no Workers completing them, the retries accumulate too.

Triage.

  1. Confirm there is demand. Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed.
  2. Check whether polling also stopped. Check temporal_num_pollers for the relevant poller_type. If pollers are also at zero, Workers are down entirely — go to All pollers disconnected.
  3. Check whether Workers are failing Tasks instead of completing them. If polling is active but completions are zero, check temporal_workflow_task_execution_failed. Workers may be failing every Task. See Non-determinism error and gRPC message too large.
  4. Check Task slots. See Worker Task slots exhausted. If every slot is occupied, no new Tasks are picked up or completed.
  5. Check for throttling on respond operations. See RESOURCE_EXHAUSTED on respond operations — sustained throttling suppresses this counter directly, because the SDK only counts successful responses.
  6. Check schedule-to-start latency. See Workflow Task schedule-to-start latency elevated. If Tasks are not being dispatched, completions drop to zero even with healthy Workers.
  7. For Activity completions, check Workflow Task health first. Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers.
Self-hosted Temporal Service

Check overall cluster health: service errors by Namespace, persistence availability, and resource-exhausted counts by cause.

Workflow Task schedule-to-start latency elevated​

Metric: temporal_workflow_task_schedule_to_start_latency, tagged namespace and task_queue

Workflow Tasks are sitting in the queue before a Worker picks them up. Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue.

Two other pages give thresholds for this metric, for different purposes. Performance bottlenecks treats P95 above one second as worth investigating — use that when you are tuning Worker capacity, not paging on it. Worker health supplies ready-made Prometheus queries for both this metric and the Activity equivalent. The thresholds on this page are set higher because they are meant to page someone.

Why it matters. Schedule-to-start latency adds directly to Workflow end-to-end latency. Every second a Task waits is a second added to how long your Workflows take.

Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all. At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database. A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue.

Triage.

  1. Check Worker health. Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else.
  2. Check poller counts. Check temporal_num_pollers for poller_type=workflow_task. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See All pollers disconnected.
  3. Check Task slots. See Worker Task slots exhausted — with every slot occupied, the SDK blocks before its next poll and the effective poll rate falls.
  4. Check for throttling on poll operations. See RESOURCE_EXHAUSTED on poll operations — throttled Workers back off and poll less frequently.
  5. Scale out if the backlog is real. If Workers are healthy and slots are free but latency stays high, the Worker pool is undersized for current throughput. Scale horizontally. Expect a delay before latency recovers — the accumulated backlog has to drain first.
Self-hosted Temporal Service

Check the approximate Task backlog in the Matching Service section of your server dashboard — a growing backlog confirms Tasks are accumulating faster than Workers pick them up. Check the server-side concurrent poller count for this Task Queue to confirm whether your Worker pool has shrunk.

Activity schedule-to-start latency elevated​

Metric: temporal_activity_schedule_to_start_latency, tagged namespace and task_queue

Activity Tasks are sitting in the queue before a Worker picks them up.

As with the Workflow Task equivalent above, Performance bottlenecks gives a lower investigation threshold for this metric, and Worker health supplies the Prometheus query.

Why it matters. Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency. At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database.

Triage.

  1. Check Worker health. Confirm Activity Worker processes are running and not restarting.
  2. Check poller counts for poller_type=activity_task. See All pollers disconnected.
  3. Check Task slots for worker_type=ActivityWorker. See Worker Task slots exhausted.
  4. Check for throttling on poll operations. See RESOURCE_EXHAUSTED on poll operations.
  5. Check the Activity failure rate. See Activity execution failures elevated — high failure churn generates retry Tasks that grow the backlog and keep this latency elevated even after Worker capacity is restored.
  6. Check for Task Queue rate limits. A low TaskQueueActivitiesPerSecond caps how fast Activities are dispatched regardless of Worker capacity. See Performance bottlenecks.
Self-hosted Temporal Service

Check the approximate Task backlog and the server-side concurrent poller count for this Task Queue, as in the Workflow Task section above.

Sticky cache holding zero entries under load​

Metric: temporal_sticky_cache_size reaching 0 while the Worker is processing Workflow Tasks

This gauge reports how many Workflow Executions the Worker is currently holding in its Sticky Execution cache, not the configured cache capacity.

A zero value is normal on an idle Worker. A Worker that is not processing Workflow Tasks has nothing to cache, and a low-volume Task Queue may sit at zero for long stretches. Alerting on this gauge alone will page you on idle Workers.

The condition worth alerting on is zero occupancy while the Worker is doing work. Pair the gauge with a non-zero Workflow Task rate on the same Worker, and use a long for duration. Corroborate with elevated temporal_workflow_task_replay_latency or a high temporal_sticky_cache_miss rate, either of which confirms Workflow Tasks are replaying from history rather than resuming from cache.

Tag coverage is uneven, which limits how precisely you can scope this alert: namespace is emitted only by the TypeScript and Java SDKs, and task_queue only by TypeScript.

Why it matters. When the cache holds nothing while Workflow Tasks are running, every Workflow Task fetches all History pages from the Temporal Service and re-executes every command from scratch. Every Workflow Task becomes the equivalent of a cache miss.

At any meaningful scale this causes sustained pressure on the Temporal Service and elevated Workflow Task execution latency for every Execution on the Worker. See Workflow Task execution latency high.

Triage.

  1. Confirm the Worker is actually processing Workflow Tasks. If the Workflow Task rate is zero, the Worker is idle and there is nothing to fix.
  2. Check whether the cache is disabled by configuration. A cache size of zero in Worker options means no Execution is ever cached:
    • Go SDK — worker.SetStickyWorkflowCacheSize(int) sets the size, defaulting to 10,000. Passing 0 disables the cache entirely. Restore a non-zero value. This must be called before any Worker starts; the cache is shared across all Workers in the process.
    • Java SDK — WorkerFactoryOptions.Builder.setWorkflowCacheSize(int) sets the size, defaulting to 600. Passing 0 resets to the default rather than disabling the cache, and a negative value is rejected, so the Java SDK cannot reach a zero-size cache this way. Check setMaxWorkflowThreadCount(int) instead — a thread pool set too low starves Workflow execution and prevents the cache from being used effectively.
  3. Check the eviction rate. If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they are added. Check temporal_sticky_cache_total_forced_eviction — a high rate points at a cache too small for the number of concurrent Executions on the Worker, or at repeated Workflow Task timeouts forcing evictions.
  4. Verify the fix. After redeploying, temporal_sticky_cache_size should climb from zero under load, and temporal_workflow_task_execution_latency should fall as cold replays are replaced by cache hits.
Self-hosted Temporal Service

Check persistence latency filtered to GetWorkflowExecution — sustained high latency there confirms the Temporal Service is under pressure from repeated History reads.