Moving PyTorch to Trainium: How TorchNeuron’s three execution modes differ

A native backend with three migration paths

AWS launched TorchNeuron’s public beta on October 20, 2026, bringing native PyTorch support to Trainium2 and Trainium3. It offers eager execution and torch.compile, plus an experimental adaptive eager path. A migration experiment should establish which path preserves a training loop’s numerical behavior and delivers useful performance. Sources: AWS, AWS Neuron, AWS Neuron Documentation

The launch results do not establish which mode wins on the same workload. AWS describes testing approximately 100,000 Hugging Face models with reduced configurations, providing compatibility evidence rather than throughput or convergence results. Hugging Face reports support for 89% of the top 100 most-used text-generation architectures, versus 46% with Optimum Neuron v0.4.5. Neither result selects an execution mode; that requires a controlled comparison. Sources: PyTorch, Hugging Face

Eager execution still requires compilation

TorchNeuron’s eager path lowers individual operations to an intermediate representation, compiles executables and queues execution asynchronously. Previously encountered operations and shapes can reuse cached artifacts. Adaptive eager fuses eligible queued operations in execution order, while the Neuron backend for torch.compile captures and compiles graphs. The table below connects these differences to measurements worth collecting. Sources: AWS Neuron Documentation

Adaptive eager is experimental and disabled by default; enablement must happen before Python starts. Buffered work flushes on a timeout, synchronization or an operation that cannot join a fusion group. The compiled path supports static shapes and the default compilation mode; other modes warn and use that default. Non-contiguous inputs require a contiguous copy on every compiled call, and Inductor options do not apply. The repository’s explicit Neuron backend and static-shape settings provide a compiled baseline. Sources: AWS Neuron, AWS Neuron Documentation

compiled_model = torch.compile(
    model,
    backend="neuron",
    dynamic=False,
)

Freeze the environment and mathematical workload

The repository distinguishes native TorchNeuron from the older backend built on Accelerated Linear Algebra, which shares the torch-neuronx name. Component release notes are dated October 19, 2026, one day before the public-beta announcement. They identify Neuron SDK 2.33.0 and native package families 2.13.3.0.* and 2.14.3.0.*, matching PyTorch 2.13.0 and 2.14.0, respectively. They list Python 3.12–3.14 support. Sources: AWS, AWS Neuron, AWS Neuron Documentation

The release notes restrict standalone wheels to Ubuntu 26.04. They direct Ubuntu 24.04 users to the supplied Deep Learning Amazon Machine Image and require source builds for manual installation on Ubuntu 22.04 or 24.04. The repository gives broader Ubuntu installation instructions, so record the actual image and installed versions. Sources: AWS Neuron, AWS Neuron Documentation

Start with a small decoder-style training loop containing embedding, causal attention, normalization, a multilayer perceptron, output projection, loss, backward execution and AdamW. Publish its dimensions and source, keeping it small enough to inspect intermediate results. Generate weights, inputs and labels on CPU once, disable dropout initially and restore identical model and optimizer state before each comparison. Test 32-bit floating point and bfloat16 separately. This workload can expose migration problems but cannot establish large-model scaling.

1. Record the instance, allocated Neuron cores, logical-core configuration, image, package versions, compiler, runtime and environment variables.
2. Keep CPU allocation and thread settings fixed across runs.
3. Run each mode in a separate process, enabling adaptive fusion only for adaptive eager before startup.
4. Keep the compilation boundary fixed across repeated runs and disclose whether it includes the optimizer.
5. Repeat with a second sequence length, non-contiguous inputs and periodic loss logging, applying each variation equally.

Check numerical agreement before measuring speed

Compare outputs, loss, every parameter gradient and post-update weights against CPU execution and between Neuron modes. Declare tolerances first, then report maximum absolute error, normalized error, the fraction outside tolerance and non-finite counts. AWS warns that adaptive fusion can alter intermediate precision and recommends disabling it and comparing identical CPU inputs when diagnosing discrepancies. TorchNeuron executes float64 as float32 with a warning, so it cannot provide a higher-precision reference through float64. A matching loss alone does not establish gradient agreement or convergence. Sources: AWS Neuron Documentation

Operator-level inspection can locate differences hidden by an overall score. The compatibility table documents bfloat16 linspace differences from CPU, collision-sensitive scatter_add behavior and float32 tan differences near poles. It also lists CPU fallbacks for torch.multinomial, torch.take and torch.scatter_reduce. Data type and layout matter: torch.randint with int64 falls back, and non-contiguous torch.lerp inputs can fall back. Record fallback paths separately for forward execution, backward execution and the optimizer. Sources: AWS Neuron

Compiled execution needs a separate inventory: torch.range and torch.logspace have eager fallback entries but registered compile decompositions. AWS forces graph breaks for some operations it cannot lower, allowing eager execution between compiled regions or raising an error when a full graph is required. Use full-graph capture diagnostically and disclose partial compilation in performance results. AWS also provides eager fallback inspection and a setting that exposes execution errors while retaining fallback for unimplemented operations. A graph break marks a compilation boundary; a CPU fallback identifies where computation runs. Sources: AWS Neuron Documentation, AWS Neuron

Separate startup costs from completed training steps

AWS documents persistent executable caching and a separate Dynamo precompile cache that avoids retracing. Executable cache keys depend on lowered computation and compiler configuration, including target, logical-core configuration, compiler version and flags. Measure a fresh process with empty experiment-specific caches, a fresh process with populated caches and steady execution within one process. Include the first backward pass and optimizer update in startup measurements. Synchronize timed regions to measure completed work rather than asynchronous submission. Sources: AWS Neuron Documentation

Report step latency and throughput separately, using repeated measurement windows and showing variability. New compiled input shapes can trigger recompilation; the repository recommends padding or bucketing inputs. Test sequence-length changes alongside layout changes and logging. Calculate compilation break-even by dividing incremental startup overhead relative to eager by the measured per-step time saved. If compiled steps are not faster, report no break-even and track migration hours separately. Sources: AWS Neuron

Profiler captures should explain those measurements. The launch example collects CPU, accelerator, device and runtime activity, synchronizes before profiler steps and sorts summaries by device self-time. Capture cold and warmed traces for every mode, using the table below as an analysis checklist. Keep profiling and diagnostic printing outside primary throughput measurements. Printing synchronizes execution and copies tensors to the host, changing behavior. Sources: AWS Neuron Documentation, PyTorch

Choose a mode by measured savings

Use eager execution to locate compatibility and numerical problems before adding graph capture. Give adaptive eager a separate trial and check whether eligible operations actually fused. AWS exposes inspection APIs for fusion status, grouped operations and execution counts. Favor compiled execution when measured savings repay startup costs within the intended run and its shape constraints fit the workload. Sources: AWS Neuron Documentation

Published migration examples show why engineering effort belongs beside execution time. Hugging Face’s generation example uses bfloat16, tensor parallelism, a static cache, a maximum sequence length of 1,024 and prefill chunks of 512. Deepgram’s launch testimonial reports approximately 60% more input audio seconds per dollar than its existing GPU baselines after a multi-month port. Keep a migration ledger covering device changes, padding, layout fixes, unsupported APIs, graph-break workarounds and custom kernels. Record time spent and measured effects so the decision accounts for both machine time and engineering work. Sources: PyTorch, Hugging Face