# Troubleshoot SDK Worker capacity problems

> Diagnose exhausted Task slots, disconnected pollers, Task backlog, and Sticky Execution cache problems

This guide covers the failure modes where Workers stop keeping up with their Task Queues: slots that never free up, pollers that stop polling, Tasks that sit in the queue, and a [Sticky Execution](/sticky-execution) cache that is not doing its job.
It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting).
For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker).

These failure modes are causally linked.
Slots fill up, which stops polling, which lets Tasks accumulate, which stops completions.
When several fire at once, work backward to the earliest link in the chain rather than treating each as independent.

## Worker Task slots exhausted 

**Metric:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) reaching 0, tagged `worker_type`

Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up.
The SDK blocks before issuing its next poll until a slot is released.

> **📝 Note:**
>
> This metric reports meaningful values only with fixed-size slot suppliers.
> It cannot be used with resource-based slot suppliers — see [Slot availability metrics](/develop/worker-performance#slot-availability-metrics).
> If your Workers use resource-based tuning, skip this section and use [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated) as your signal that Worker capacity is short.
>

**Why it matters.**
Slots stay occupied because existing Tasks are not completing.
Until they free up, schedule-to-start latency rises and, in both the Go and Java SDKs, [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) drops to zero as a secondary effect, because those SDKs block on slot acquisition before incrementing the poller gauge.

Impact and remediation differ by `worker_type`, so check that tag on the firing alert.
Values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`.

### WorkflowWorker slots 

1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high).
1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer.
1. **Check for blocking calls in Workflow code.** A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — slots are not released until the respond call succeeds.

To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options.

### ActivityWorker slots 

1. **Check Activity execution latency.** Sustained high [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` confirms Activities are holding slots longer than expected.
1. **Check Worker CPU.** High CPU slows Activity execution directly.
1. **Check what the Activities are waiting on.** If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations).

To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size — but only after confirming a downstream dependency is not the constraint.

### LocalActivityWorker slots 

[Local Activities](/local-activity) run inside the Workflow Task execution loop, so blocked slots hold up the entire Workflow Task.
The SDK responds by sending repeated Workflow Task heartbeats to keep the Task alive.
If heartbeating continues past the Workflow Task heartbeat timeout — 30 minutes by default — the Temporal Service times out the Task and reschedules it, and the Local Activities re-execute from scratch.

1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling.
1. **Check Worker CPU.**
1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) — if that is firing alongside this, the heartbeat timeout is imminent or already exceeded and re-execution has begun.

## All pollers disconnected 

**Metric:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) reaching 0, tagged `poller_type` and `task_queue`

No active pollers remain for this poller type and Task Queue.
Workers have stopped polling entirely and Tasks are accumulating with nothing to process them.

> **📝 Note:**
>
> This metric is tagged `poller_type`, not `worker_type`.
> The values are `workflow_task`, `workflow_sticky_task`, `activity_task`, and `nexus_task`.
> Alert on each poller type your Workers use, and note that a Worker with an active Sticky Execution cache reports `workflow_sticky_task` alongside `workflow_task`.
>

**Why it matters.**
Workflow and Activity Tasks accumulate on the Temporal Service.
At scale this grows into a large backlog that puts pressure on Task matching and persistence.
Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed.

**Triage.**

1. **Check whether Worker processes are running.** Check pod status, restart counts, and logs. Workers may have crashed, been evicted, or been OOM killed — this is the most common cause.
1. **Rule out scale-down.** If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further.
1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first.
1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry.
1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) — sustained INTERNAL errors cause Workers to back off and stop polling.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Cross-check the server-side concurrent poller count for this Task Queue.
> If it has also dropped, Workers have fully disconnected from the Service's perspective.
> For authentication failures, check unauthorized request counts and authorization system failures — a non-zero authorization system failure count means the auth plugin itself is failing, which is more urgent than an expired credential.
>

## Task completions dropped to zero 

**Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted`

The SDK increments this counter only after a successful response, so a zero rate means no responses are landing.

**A zero rate on its own does not prove a problem.**
It is equally consistent with an idle Task Queue, a Namespace outside business hours, or a batch workload between runs.
This metric is tagged `namespace` and `operation`, not `task_queue`, so you cannot narrow it to a single Task Queue without correlating against another signal.

Treat it as actionable only alongside evidence of demand: a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency that keeps climbing.
Without one of those, there is nothing for Workers to complete and the metric is reporting the truth.

**Why it matters.**
When there is demand, no Workflow Task completions means Workflow Executions are not making progress.
Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them.

No Activity Task completions means Workflows waiting on Activity results are stalled.
Activities begin timing out against their `scheduleToClose` timeouts, and the Temporal Service retries them within the Retry Policy — but with no Workers completing them, the retries accumulate too.

**Triage.**

1. **Confirm there is demand.** Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed.
1. **Check whether polling also stopped.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are also at zero, Workers are down entirely — go to [All pollers disconnected](#all-pollers-disconnected).
1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large).
1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — sustained throttling suppresses this counter directly, because the SDK only counts successful responses.
1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers.
1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check overall cluster health: service errors by Namespace, persistence availability, and resource-exhausted counts by cause.
>

## Workflow Task schedule-to-start latency elevated 

**Metric:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency), tagged `namespace` and `task_queue`

Workflow Tasks are sitting in the queue before a Worker picks them up.
Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue.

Two other pages give thresholds for this metric, for different purposes.
[Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) treats P95 above one second as worth investigating — use that when you are tuning Worker capacity, not paging on it.
[Worker health](/cloud/worker-health#schedule-to-start-latency) supplies ready-made Prometheus queries for both this metric and the Activity equivalent.
The thresholds on this page are set higher because they are meant to page someone.

**Why it matters.**
Schedule-to-start latency adds directly to Workflow end-to-end latency.
Every second a Task waits is a second added to how long your Workflows take.

Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all.
At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database.
A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue.

**Triage.**

1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else.
1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `poller_type=workflow_task`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected).
1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) — with every slot occupied, the SDK blocks before its next poll and the effective poll rate falls.
1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations) — throttled Workers back off and poll less frequently.
1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, the Worker pool is undersized for current throughput. Scale horizontally. Expect a delay before latency recovers — the accumulated backlog has to drain first.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the approximate Task backlog in the Matching Service section of your server dashboard — a growing backlog confirms Tasks are accumulating faster than Workers pick them up.
> Check the server-side concurrent poller count for this Task Queue to confirm whether your Worker pool has shrunk.
>

## Activity schedule-to-start latency elevated 

**Metric:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency), tagged `namespace` and `task_queue`

Activity Tasks are sitting in the queue before a Worker picks them up.

As with the Workflow Task equivalent above, [Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_activity_schedule_to_start_latency-spike) gives a lower investigation threshold for this metric, and [Worker health](/cloud/worker-health#schedule-to-start-latency) supplies the Prometheus query.

**Why it matters.**
Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency.
At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database.

**Triage.**

1. **Check Worker health.** Confirm Activity Worker processes are running and not restarting.
1. **Check poller counts** for `poller_type=activity_task`. See [All pollers disconnected](#all-pollers-disconnected).
1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted).
1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations).
1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) — high failure churn generates retry Tasks that grow the backlog and keep this latency elevated even after Worker capacity is restored.
1. **Check for Task Queue rate limits.** A low `TaskQueueActivitiesPerSecond` caps how fast Activities are dispatched regardless of Worker capacity. See [Performance bottlenecks](/troubleshooting/performance-bottlenecks).

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the approximate Task backlog and the server-side concurrent poller count for this Task Queue, as in the Workflow Task section above.
>

## Sticky cache holding zero entries under load 

**Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 while the Worker is processing Workflow Tasks

This gauge reports how many Workflow Executions the Worker is **currently** holding in its [Sticky Execution](/sticky-execution) cache, not the configured cache capacity.

**A zero value is normal on an idle Worker.**
A Worker that is not processing Workflow Tasks has nothing to cache, and a low-volume Task Queue may sit at zero for long stretches.
Alerting on this gauge alone will page you on idle Workers.

The condition worth alerting on is zero occupancy *while the Worker is doing work*.
Pair the gauge with a non-zero Workflow Task rate on the same Worker, and use a long `for` duration.
Corroborate with elevated [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) or a high [`temporal_sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) rate, either of which confirms Workflow Tasks are replaying from history rather than resuming from cache.

Tag coverage is uneven, which limits how precisely you can scope this alert: `namespace` is emitted only by the TypeScript and Java SDKs, and `task_queue` only by TypeScript.

**Why it matters.**
When the cache holds nothing while Workflow Tasks are running, every Workflow Task fetches all History pages from the Temporal Service and re-executes every command from scratch.
Every Workflow Task becomes the equivalent of a cache miss.

At any meaningful scale this causes sustained pressure on the Temporal Service and elevated Workflow Task execution latency for every Execution on the Worker.
See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high).

**Triage.**

1. **Confirm the Worker is actually processing Workflow Tasks.** If the Workflow Task rate is zero, the Worker is idle and there is nothing to fix.
1. **Check whether the cache is disabled by configuration.** A cache size of zero in Worker options means no Execution is ever cached:
   - **Go SDK** — `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` disables the cache entirely. Restore a non-zero value. This must be called before any Worker starts; the cache is shared across all Workers in the process.
   - **Java SDK** — `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets to the default rather than disabling the cache, and a negative value is rejected, so the Java SDK cannot reach a zero-size cache this way. Check `setMaxWorkflowThreadCount(int)` instead — a thread pool set too low starves Workflow execution and prevents the cache from being used effectively.
1. **Check the eviction rate.** If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they are added. Check [`temporal_sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction) — a high rate points at a cache too small for the number of concurrent Executions on the Worker, or at repeated Workflow Task timeouts forcing evictions.
1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero under load, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check persistence latency filtered to `GetWorkflowExecution` — sustained high latency there confirms the Temporal Service is under pressure from repeated History reads.
>