# Troubleshoot SDK execution failures

> Diagnose non-determinism errors, oversized payloads, failing Workflow and Activity code, and Local Activity latency

This guide covers failures that occur while your Workflow and Activity code is executing on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that run past the Workflow Task heartbeat timeout.
It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service.

For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting).
For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics).

[`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`.
The reason determines what the Temporal Service does next, and the difference matters: two of the three cause indefinite retries, one causes immediate termination.
Alert on each `failure_reason` separately.

## Non-determinism error 

**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError`

Replay produced a different command sequence than the one recorded in Event History.
The Worker detected that the Workflow code it is running does not match the commands the Workflow Execution has already produced.

**Why it matters.**
Affected Executions are not progressing.
The Temporal Service retries the Workflow Task continuously, adding pressure to your Workflow Workers, and by default the Executions stay in Running status — prolonging their end-to-end time indefinitely.
A non-determinism error does not resolve on its own.

**Triage.**

1. **Identify the affected Workflow Executions.** This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures.
1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause.
1. **Determine whether this is a code change or a deploy artifact.** Common causes:
   - A code change added, removed, or reordered commands — Activity scheduling, Timers, Signals, Child Workflows — without a versioning guard. In-flight Executions that built History under the old code fail on the new code.
   - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes.
   - Changed Activity or Timer parameters in existing Workflow code without versioning.
1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning).
1. **Watch Worker pressure.** Continuous retries put sustained load on Workflow Workers. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high) — a high volume of retries can saturate capacity and affect healthy Executions on the same Task Queue.

## gRPC message too large 

**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge`

The Workflow Task response payload exceeded the gRPC message size limit.
The Worker attempted `RespondWorkflowTaskCompleted` and the response was rejected — by the gRPC library on the SDK side, by a proxy or load balancer in the path, or by the gRPC library on the Temporal Service side on receive.

Because the Temporal Service never saw the original request, the SDK sends a follow-up `RespondWorkflowTaskFailed` with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`.
Since replay would produce the same oversized response on every attempt, the Temporal Service terminates the Workflow Execution rather than retrying it.

For the payload size limits themselves and the equivalent Activity Task behavior — which retries rather than terminating — see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error).

**Why it matters.**
Affected Executions are terminated immediately and permanently, with `TERMINATED` status and no retry.
Any in-progress work in those Executions is lost, and they must be restarted manually.

This is the one `failure_reason` on this page that ends Executions rather than retrying them, which is why it warrants a short `for` duration.

**Triage.**

1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the `WorkflowTaskFailed` and `WorkflowExecutionTerminated` events in Event History.
1. **Find what is oversized.** The fix depends entirely on which part of the response is too large:
   - **Oversized Activity inputs or outputs.** Move large payloads out of band — store them in blob storage and pass a reference through Event History instead. See [External Storage](/external-storage) for the pattern.
   - **Accumulated Signals or Updates.** A large number buffered into a single Workflow Task. Rate-limit senders or batch Signals.
   - **Too many commands in one response.** A Workflow scheduling a very large fan-out of Activities or Child Workflows in a single step. Break the fan-out into smaller batches across multiple Workflow Tasks.
1. **Fix and deploy before restarting anything.** Terminated Executions do not retry. Restarting them before the cause is fixed means hitting the same limit and being terminated again. Once the corrected Worker is deployed and verified, restart the affected Executions from the Temporal UI or CLI.

> **📝 Note:**
> Self-hosted Temporal Service
>
> Check the Workflow terminate rate on your server dashboard — a spike alongside this metric confirms Executions are being terminated at volume.
>

## Workflow Task execution failures elevated 

**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError`

Sustained Workflow Task failures from unhandled exceptions and panics in Workflow code that the SDK catches and reports.
`WorkflowError` is the catch-all reason: it covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors.

**Why it matters.**
The Temporal Service retries the Workflow Task.
If the error is deterministic and reproduces on every replay, the Execution is stuck retrying indefinitely, consuming Worker capacity and staying in a permanently unhealthy state.

At high rates the retry pressure saturates Workflow Worker slots and affects healthy Executions on the same Task Queue.
Unlike `GrpcMessageTooLarge`, the Temporal Service does not terminate the Execution, so the impact compounds until you resolve it.

**Triage.**

1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. The `workflow_type` tag on the metric narrows which Workflow is failing.
1. **Determine which failure mode this is.** `WorkflowError` covers several:
   - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool, caused by `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` being too low for the number of concurrent Executions. New Workflow Tasks are rejected before they can execute. Raise the thread count, and check whether the Worker pool needs to scale out as well.
   - **Unhandled exception in Workflow code.** A bug or unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event identifies the error.
   - **Data Converter error.** A failure serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration.
1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion often occur together under load, and a CPU-starved Worker completes Workflow Tasks more slowly, accelerating both.
1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running.

## Workflow Task execution latency high 

**Metric:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency), tagged `namespace`, `task_queue`, and `workflow_type`

Workflow Tasks are taking too long to execute on the Worker.
The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks.

A batch workload where Workflow Tasks routinely run long may sit above this threshold without any problem.
Set the threshold against your own observed p99, and treat the default as a signal only if your Workflows are latency-sensitive.

**Why it matters.**
The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue.
Each timeout forces a [Sticky Execution](/sticky-execution) cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay.

If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats.
Non-idempotent Local Activities produce duplicate side effects with real business impact.

At scale this compounds: more timeouts cause more cold replays, cold replays drive latency higher, and higher latency causes more timeouts.

**Triage.**

1. **Check replay latency.** Check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-executing Event History rather than running new commands — usually caused by large histories, slow Data Converter execution during replay, or a high cache eviction rate forcing cold replays.
1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load).
1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker.
1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop.
1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — the SDK holds the slot until the respond call succeeds, inflating this metric even when the Workflow code finished quickly.

## Activity execution failures elevated 

**Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed), tagged `activity_type`

Activities are explicitly failing — returning failures rather than timing out — at a sustained rate.

`ApplicationFailure` instances marked with category `BENIGN` are excluded and do not increment this counter, so this metric tracks unexpected failures only to the extent your application uses benign failures correctly.

**Why it matters.**
A high failure rate drives a burst of retry Tasks.
If Workers cannot keep up with the retry volume, the Activity Task backlog grows — see [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated).
At scale, sustained retry bursts put significant pressure on Task matching and the underlying database.

**Triage.**

1. **Identify which Activity is failing.** The `activity_type` tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids.
1. **Determine whether this is transient or a bug.** A downstream service outage, network partition, or database timeout recovers on its own — watch whether the rate falls. A persistent code bug does not.
1. **Check downstream service health.** A degraded dependency is a common cause of sustained failure bursts. If the dependency is throttling, confirm your [Retry Policy](/encyclopedia/retry-policies) has appropriate backoff — without it, retry bursts amplify the pressure you are already applying.
1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops.
1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`, which suppresses this metric for them and lets the alert track unexpected failures without per-`activity_type` threshold tuning. Confirm your SDK version supports the category before relying on it.

Note that internal failures — a context propagation error or a context timeout, rather than an Activity returning a failure — increment this counter regardless of category.

## Unregistered Activity invocation 

**Metric:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type`

A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for.

This metric is emitted by the Go SDK only.

**Why it matters.**
The Activity cannot execute. It will keep being retried against a Worker that cannot run it until the Activity's `scheduleToClose` timeout expires, or indefinitely if no such timeout is set — so the Workflow Execution waiting on it makes no progress.

This is almost always a deployment error rather than a runtime condition: Workflow code that schedules an Activity the deployed Worker does not register. It does not resolve on its own.

**Triage.**

1. **Identify the Activity and the Task Queue.** The `activity_type` and `task_queue` tags name both. The `workflow_type` tag identifies which Workflow is scheduling it.
1. **Check whether the Activity is registered on the right Worker.** Confirm the Worker polling that Task Queue registers that Activity type. A common cause is registering the Activity on a Worker polling a different Task Queue.
1. **Check for a partial rollout.** If Workflow code that schedules a new Activity deployed ahead of the Worker that implements it, some Workers will be running without the registration. Complete the rollout.
1. **Check for a renamed Activity.** Changing an Activity's registered name while Executions are in flight leaves those Executions scheduling the old name. Register both names until the in-flight Executions drain, or use a versioning guard.

## Local Activity latency exceeds the heartbeat timeout 

**Metric:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency), tagged `activity_type`

A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes.

### How Workflow Task heartbeating works 

A Local Activity executes inside the Workflow Task rather than as a separately scheduled Activity Task.
That means the Workflow Task stays open for as long as the Local Activity runs, which would normally exceed the Workflow Task timeout.

To keep the Task alive, the SDK sends Workflow Task heartbeats — repeated `RespondWorkflowTaskCompleted` calls that tell the Temporal Service work is still in progress and request more time.
The Service allows this up to the Workflow Task heartbeat timeout.
Past that, it times out the Task and reschedules it on the normal Task Queue.

Local Activities cannot heartbeat individually the way regular Activities can, and their results are not recorded in Event History between Workflow Task heartbeats.
So when the Task is rescheduled, every Local Activity in it runs again from the beginning.

**Why it matters.**
When the Temporal Service times out the heartbeating Workflow Task, the Local Activity re-executes from scratch.
A non-idempotent Local Activity produces duplicate side effects with real business impact.

Any pending Signals, Updates, or other events are delayed until the retried Workflow Task completes, so end-to-end Execution latency rises significantly.

The Local Activity also occupies an executor slot for its entire duration.
Several in this state at once can occupy every available slot, blocking new Local Activities from starting.
See [LocalActivityWorker slots](/troubleshooting/sdk-worker-capacity#localactivityworker-slots).

Local Activities are designed for short, fast operations.
A single attempt running for 30 minutes is a design problem, not a tuning problem.

**Triage.**

1. **Identify the affected Local Activity.** The `activity_type` tag narrows it down. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids.
1. **Find what it is blocked on.** A Local Activity running this long is almost always blocked on a downstream call — a slow service, a slow query, or a network call with a very long timeout. Fix the dependency, or shorten the timeout on the call so the Local Activity fails fast instead of hanging.
1. **Check whether a retry chain is accumulating.** A high failure rate paired with an aggressive Retry Policy can push total elapsed time past the heartbeat timeout even when every individual attempt is short. Check [`temporal_local_activity_execution_failed`](/references/sdk-metrics#local_activity_execution_failed) for the same `activity_type`, and fix the underlying failure first.
1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have already timed out heartbeating Workflow Tasks. Check Worker logs for timeout errors and Event History for `WorkflowTaskTimedOut` events. If they are present, Local Activities have already re-executed — verify idempotency and address any duplicate side effects.
1. **Fix the design.** If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a `scheduleToCloseTimeout` below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed.