Ahad Jawaid

NAT: Scaling Causal Neural Action Tokenizers Across Embodiments

Alexander SwerdlowStanford University
Yu XiangThe University of Texas at Dallas
nat

Action tokenizers let robot policies treat control as autoregressive next-token prediction. As these policies are trained on larger and more diverse datasets, they will need a universal tokenizer that can encode actions from many embodiments and data sources, including ones it has not seen. We study action tokenization at scale, training on 49k hours of action data from real-world robots, handheld grippers, simulation, and humans. We introduce Neural Action Tokenizer (NAT), a causal tokenizer whose decoder lets actions be executed as their tokens are generated.

Across four simulation benchmarks, NAT outperforms FAST and prior neural action tokenizers when tokenizing unseen robot embodiments, and in vision-language-action models it outperforms FAST+ across four benchmarks and on a real robot when acting through its tokens. NAT tokens also close the performance gap to a flow-matching action head on some benchmarks while lowering the inference latency for first actions. Our experiments also find that robot data may not be necessary to learn an effective action space: a NAT trained only on human hand motion performs close to tokenizers trained on robot data.

Figure 1: Time to first action on a 0.45B VLA (RTX 4090, batch size 1), slowed 50× with actions at 30 Hz. FAST+ must generate every token before decoding; a flow-matching head denoises the whole chunk; NAT decodes actions as soon as their tokens are generated.

Contributions

  • State-of-the-art neural action tokenization. NAT outperforms FAST and prior neural action tokenizers (OAT, ActionCodec, and NAC) in mean success on unseen robot embodiments.
  • A universal streaming action tokenizer. We train NAT on 49k hours of actions from real-world robots, handheld grippers, simulation, and humans, yielding a tokenizer for diverse action data. Its causal decoder supports immediate execution, and its time-ordered tokens let a policy condition its next chunk on the actions still executing without retraining.
  • Human data for action tokenizers. Our experiments suggest that robot data may not be necessary to train an action tokenizer: a NAT trained only on human hand motion is nearly as effective as one trained on all sources.
  • Pretraining vision-language-action models with NAT tokens. We show that NAT tokens can help pretrain vision-language-action models: further training a pretrained vision-language model on 1.8k hours of robot data through NAT tokens, before fine-tuning, raises mean success on four simulation benchmarks over fine-tuning the vision-language model directly.
  • Flow action heads and action tokens. Acting through NAT tokens produces first actions sooner than acting through a flow-matching action head, and on some benchmarks matches its success: on LIBERO, the two are within the 95% intervals.

Method

NAT flattens an action chunk into a one-dimensional signal, encodes it with a causal convolutional encoder, and quantizes the latents with multi-scale FSQ into coarse-to-fine tokens. Tokens are emitted in time order, so the causal decoder can turn the first group of tokens into the first actions while the policy is still generating the rest of the chunk.

NAT architecture: encoder, multi-scale FSQ, decoder, and the causal layer stacks
Figure 2: NAT architecture. Left: an action chunk is flattened, encoded into latents, quantized by multi-scale FSQ into coarse-to-fine tokens, and decoded back; NAT is trained end to end with a mean-squared reconstruction loss. The inset shows the 14 tokens of one chunk of the universal NAT, numbered in the order they are emitted. Right: the causal encoder and decoder layer stacks, read bottom to top.

Varying action dimensions. Robots differ in their number of action dimensions, while NAT encodes chunks of a fixed width. Each action vector is split into groups of at most 7 dimensions, one per arm where the dataset defines arms and consecutive dimensions otherwise, and each group is tokenized as its own chunk.

An action chunk split into groups of 7 dimensions, each tokenized by NAT
Figure 3: Varying action dimensions. The action dimensions are split into groups of 7, one per arm where the dataset defines arms and consecutive dimensions otherwise, and the last group is zero-padded. The groups are placed one after another in time, NAT encodes each group, and their tokens form one sequence. Padded slots are excluded from the reconstruction loss.

Streaming and real-time chunking. As soon as the tokens of a latent are generated, NAT decodes that latent's actions and the robot executes them, while the policy continues generating the rest. While the policy predicts the next chunk, the robot keeps executing the current one, so the next chunk must continue from the actions that run in the meantime. Because the tokens are ordered in time and the decoder is causal, this works through the tokens alone: NAT encodes the actions still executing as the start of a new chunk, their first tokens form a fixed prefix, and the policy generates only the remaining tokens from the new observation. The robot then switches to the new chunk with less of a jump (Table 8), without any change to training or sampling.

Real-time chunking with action tokens: executed tokens, a frozen prefix for the actions still executing, and the rest of the chunk generated from the new observation
Figure 4: Real-time chunking with action tokens. Gray tokens encode actions already executed. When inference starts, the next actions will execute before the new chunk is ready, so NAT encodes these actions as the start of the new chunk and freezes the resulting first tokens. The policy generates only the remaining tokens, given the new observation. Because the decoder is causal, the frozen prefix decodes to the actions being executed.

Data

The corpus has 49k hours (5.5B action frames) from 156 datasets: real-robot teleoperation, simulation, handheld grippers (UMI), and human hands. Each dataset is normalized per dimension to [−1, 1]. The corpus excludes the evaluation robots (Franka Panda and Sawyer), so the tokenizers are not trained on them.

Corpus hours by data source and by action representation
Figure 5: Action corpus. Hours by data source (top) and by action representation (bottom).
One example episode of each action representation in the corpus, from joint positions to actuator commands
Figure 6: Action representations in the corpus, one example episode each.

Results

Comparison with prior action tokenizers

With one cross-embodiment policy per tokenizer, NAT has the highest mean success (47.0%), ahead of FAST (43.4%), the other neural tokenizers (29.0–40.9%), and the diffusion policy (35.9%). NAT is best on LIBERO-Long and RoboCasa, FAST on Meta-World, and OAT on RoboMimic. OAT and NAC solve under 10% of Meta-World, whose actions fill a small part of the normalized range and so magnify reconstruction error. With a LIBERO-only policy (Table 2), NAT reaches 77.9% (FAST: 66.2%).

MethodLIBERO-LongMeta-WorldRoboMimicRoboCasaMean
NAT (ours)74.4 ± 3.838.2 ± 6.638.6 ± 2.536.8 ± 3.347.0 ± 1.5
FAST68.6 ± 0.739.4 ± 1.435.1 ± 3.430.8 ± 7.943.4 ± 1.7
OAT70.3 ± 2.99.2 ± 3.140.0 ± 3.726.8 ± 4.736.6 ± 2.3
ActionCodec61.8 ± 3.832.2 ± 2.339.1 ± 1.630.5 ± 4.540.9 ± 1.6
NAC57.2 ± 1.37.9 ± 2.526.1 ± 4.225.0 ± 2.329.0 ± 1.2
Diffusion Policy57.5 ± 4.631.8 ± 4.329.2 ± 3.525.2 ± 5.035.9 ± 2.6
Table 1: Success (%) of one cross-embodiment policy per tokenizer, trained jointly on the four benchmarks; the evaluation robots are unseen by the tokenizers. Highlighted: best in each column.

Tokens and generation rate. On LIBERO, which runs at 20 Hz, one second of action takes 9 tokens with the universal NAT, 24 with FAST+ (2.8× more), and 140 with per-dimension binning (16× more).

What data does an action tokenizer need?

These studies train tokenizers on subsets of the corpus and evaluate them on LIBERO-Long with a fixed LIBERO-only policy.

Human data alone. Trained on human hands only, NAT keeps 74.4% and FAST, whose DCT basis is not learned, is unchanged, while OAT, ActionCodec, and NAC lose 18–33 points. NAT's reconstruction error on LIBERO grows by 22%, versus 27% for OAT and 40% for ActionCodec. A human-only NAT is also close to the all-data NAT with a cross-embodiment policy (38.0% vs. 38.5% mean success).

MethodRecon. errorSuccess (%)
AllHumanAllHuman
NAT (ours)1.441.7677.9 ± 1.874.4 ± 2.2
FAST0.950.9566.2 ± 2.167.0 ± 3.5
OAT3.103.9361.5 ± 1.838.6 ± 1.3
ActionCodec2.473.4558.4 ± 1.240.9 ± 2.8
NAC2.564.0457.8 ± 2.825.1 ± 1.9
Table 2: Tokenizers trained on all source types (All) or on human-hand data only (Human). Reconstruction error: mean absolute error (×10−2) on LIBERO-10 arm actions. Highlighted: best in each column (lowest error, highest success).

Data mix. Every subset of source types stays within 4.1 points of the full mix and above FAST. Removing human-hand data causes the largest single-source drop (3.1 points).

Source typeOnly this typeThis type removed
Real robot77.8 ± 4.075.9 ± 3.2
Sim robot–77.0 ± 5.1
UMI76.0 ± 2.477.2 ± 3.6
Human hands74.4 ± 2.274.8 ± 2.5
UMI and human hands–73.8 ± 2.7
Table 3: NAT trained on one source type only or with source types removed. Trained on all source types, NAT reaches 77.9 ± 1.8%.

Data amount. With the policy fixed, training NAT on 0.1% to 100% of the corpus (50 to 49k hours, same number of updates) keeps success between 74.3% and 76.4%.

LIBERO-Long success as NAT is trained on growing shares of the corpus
Figure 7: NAT is data-efficient. LIBERO-Long success of a fixed policy as NAT is trained on growing shares of the zero-shot corpus with the same number of updates. Error bars are 95% confidence intervals.

Which architectural choices matter?

These ablations, unlike the other tokenizer studies, train on a 1% subset of an earlier corpus that includes the evaluation robots. Success rises from 12.9% with one FSQ scale to 75.8% with four, with no gain from eight. Replacing FSQ with NAC's residual vector quantization costs 8.8 points. We suspect two causes, not isolated here: RVQ's learned codebooks are prone to collapse, leaving codes unused, and FSQ's fixed grid makes codes that share most channel levels decode to similar latents, so a policy's confusions between them cost less, while RVQ's codes have no such structure. The ISTFT and linear output heads perform similarly (75.6% vs. 75.8%), so NAT uses the linear head, and a non-causal NAT scores 1.7 points higher, within the 95% intervals.

ComponentNATSuccessAlternativeSuccess
QuantizerMS-FSQ69.3 ± 2.2MS-RVQ60.5 ± 1.2
Output headLinear75.8 ± 3.6ISTFT75.6 ± 0.5
CausalityCausal75.8 ± 3.6Non-causal77.5 ± 3.1
Scales (tokens)1 (8)2 (12)4 (28)8 (60)
Success12.9 ± 0.856.3 ± 4.375.8 ± 3.674.9 ± 1.6
Table 4: Architecture ablations on LIBERO-Long: success (%) of NAT's choice and an alternative for each component, and of the number of scales (tokens per chunk). The quantizer comparison was run on an earlier reference. Highlighted: best number of scales.

NAT VLA

Two LFM2-VL-450M VLAs are pretrained on 4.8B tokens of MolmoAct2's seven-robot data mixture (1.8k hours), which includes DROID, BC-Z, RT-1, and BridgeData V2: one with a universal NAT (4 latents, 14 tokens per chunk, 6,400 codes per scale, trained for one pass over the full zero-shot corpus) and one with FAST+, otherwise identical. On held-out data, NAT's action error under greedy decoding is 16% lower. Each is then fine-tuned jointly on four simulation benchmarks and on a real robot. Acting through tokens, NAT leads on all four benchmarks and raises mean success from 37.4% to 61.4%; on LIBERO it reaches 90.8% against 62.0% for FAST+, while on Meta-World the intervals overlap. Without robot pretraining, the NAT VLA reaches 54.9%, still above the pretrained FAST+ VLA; robot pretraining through NAT tokens adds 6.5 points (4.8 on LIBERO), with non-overlapping intervals on Meta-World and RoboMimic.

TokenizerActs viaLIBEROMeta-WorldRoboMimicRoboCasaMean
NAT (ours)Tokens90.8 ± 2.622.8 ± 2.281.2 ± 2.450.9 ± 3.361.4
FAST+Tokens62.0 ± 4.220.9 ± 2.429.1 ± 2.837.6 ± 3.237.4
NAT, no robot pretrainingTokens86.0 ± 2.815.6 ± 1.672.5 ± 2.645.4 ± 3.154.9
NAT (ours)Flow expert89.2 ± 2.845.7 ± 2.589.8 ± 2.069.5 ± 3.073.6
FAST+Flow expert89.0 ± 2.543.9 ± 2.584.7 ± 2.367.9 ± 3.071.4
Table 5: Success (%) at 50k steps with 95% bootstrap intervals, acting through action tokens or a flow-matching expert. All rows use robot-pretrained backbones except no robot pretraining, which is fine-tuned from the base LFM2-VL-450M with the same recipe. Highlighted: best within each group (tokens, flow expert).

Flow action head. Through a flow-matching expert trained with knowledge insulation, NAT VLA reaches 73.6% mean success and FAST+ VLA 71.4%; the intervals overlap except on RoboMimic. These runs add a mid-training stage and use MolmoAct2's post-training learning rates, so they differ from the token runs in more than the action head. The expert raises NAT's mean by 12.2 points, mostly on Meta-World and RoboCasa, and FAST+'s by 34.0. On LIBERO, NAT tokens and the expert are within the 95% intervals (90.8% vs. 89.2%). The expert adds 77.7M parameters, a mid-training stage, and a denoising process at inference, while NAT's first actions arrive sooner than a chunk from our flow head (19.0 vs. 42.0 ms) and, in our latency analysis, than the heads of GR00T N1.7 and MolmoAct2, and its tokens keep streaming and real-time chunking.

Real robot and latency

Figure 8: Autonomous rollout on a Trossen AI Solo arm, 1× speed.
The four real-world tasks with their start and goal states
Figure 9: Real-world tasks on the Trossen AI Solo arm, with the start and goal state of each task.
The Trossen AI Solo arm with its wrist camera, parallel gripper, and six joints
Figure 10: The Trossen AI Solo arm used for the real-world evaluation.

Both VLAs are fine-tuned on 481 teleoperated demonstrations; the universal NAT's corpus includes our own Trossen AI recordings. Each policy runs 10 trials per task from the demonstrations' median start pose, each scored by an operator. NAT succeeds in 55% of trials and FAST+ in 35%, leading on three of the four tasks. FAST+ is better at placing grapes (70% vs. 20%), although NAT's reconstruction error on the grape demonstrations is small (MSE 0.0005) and its policy's greedily sampled actions have lower reconstruction error than FAST+'s (MSE 0.021 vs. 0.038). In failed trials NAT VLA did not appear to lower the gripper far enough and closed above the grapes; because each chunk's 16 actions execute open loop, such an offset may go uncorrected.

TokenizerFold towelTurn on scaleMarker in binGrapes on plateMean
NAT (ours)9070402055
FAST+502007035
Table 6: VLA success on a real Trossen AI Solo arm (%, 10 trials per task). Highlighted: best in each column.

FAST+ uses 6 to 73 tokens per chunk (38.8 on average) and decodes only after the whole chunk. NAT uses 14, and its first 4 decode to 8 actions, which take 267 ms to execute at 30 Hz; the rest of the chunk, and the next one, are generated while the robot moves. NAT VLA has its first actions after 19.0 ms, 5.1× sooner than FAST+ VLA and 2.2× sooner than its own flow expert. On 3–5B models, generating NAT's first 4 tokens would take 45–101 ms, comparable to or faster than their action heads, and a full chunk 85–226 ms, under the 267 ms the first actions take to execute.

ModelTokenizerSizePer tokenFirst actionsFull chunk (tokens)Full chunk (flow head)
NAT VLA (ours)NAT0.45B2.0 ± 0.019.0 ± 0.5*39.5 ± 0.6*42.0 ± 0.9
FAST+ VLAFAST+0.45B2.2 ± 0.097.5 ± 37.697.5 ± 37.646.5 ± 12.6
GR00T N1.7NAT†3B4.1 ± 0.044.7 ± 1.085.2 ± 1.056.6 ± 1.2
π0.5NAT†3B6.5 ± 0.069.0 ± 1.7134.5 ± 1.772.6 ± 0.2
MolmoAct2NAT†5B12.6 ± 0.0100.8 ± 0.3226.4 ± 0.4132.4 ± 1.3
Table 7: Action latency (ms) on an RTX 4090 (batch size 1, CUDA graphs); mean ± s.d. over 50 frames for our models and 30 runs for the others. First actions: 8 actions from 4 NAT tokens; full chunk: 14 NAT tokens. *Includes detokenization. †Latency analysis only: these models are not trained with NAT; we time 4 and 14 decoding steps of their language models. Highlighted: the faster route to the first actions, tokens or flow head.

Real-time chunking

With real-time chunking, the NAT VLAs' switches from one chunk to the next come with smaller jumps. The difference between the last executed action of a chunk and the first action of the next falls from 0.126 to 0.065 for the VLA fine-tuned on the four benchmarks and from 0.046 to 0.0096 for the robot-pretrained VLA on held-out data. A switch is still larger than the change between two actions of the same chunk.

NAT VLAAcross a chunk switchWithin one chunk
With RTCWithout RTC
Fine-tuned0.065 ± 0.0070.126 ± 0.0100.031 ± 0.003
Pretrained0.0096 ± 0.00070.046 ± 0.0090.0024 ± 0.0002
Table 8: Difference between consecutive actions, |at − at−1|, averaged over the arm's action dimensions (in [−1, 1]), steps, and benchmarks; ±: 95% bootstrap interval. Across a chunk switch: from the last executed action of a chunk to the first of the next; within one chunk: two consecutive actions of the same chunk. Fine-tuned: LIBERO, RoboMimic, Meta-World; pretrained: held-out Bridge, RT-1, BC-Z, SO-100/101. Measured by running recorded demonstrations through the LeRobot NAT VLA, open loop.

Takeaways

  • A single action tokenizer can serve robots it has never seen, outperforming FAST and prior neural tokenizers without training on the evaluation robots.
  • An action tokenizer may not need robot data: human hand motion alone yields a tokenizer close to one trained on all sources, so large-scale human data could be used to train action tokenizers.
  • NAT tokens are an effective action representation for vision-language-action models, both for robot pretraining and for acting, where they outperform FAST+.
  • Action tokens can be faster than flow action heads, and on some domains acting through tokens alone closes the performance gap to a flow head, though a gap remains on others.

Limitations and future work

Acting through NAT tokens still trails a flow-matching head on some benchmarks; further tuning and design choices, such as the token budget, decoding, and fine-tuning recipe, may close this gap fully. NAT encodes every chunk in a fixed number of tokens; a future study could set the number of tokens per chunk dynamically. NAT is also trained on a single chunk length, so tokenizing chunks of varying length remains to be explored. Our data-scaling experiment scales the tokenizer's training data while keeping the policy's LIBERO data fixed; a future study could explore the benefits of scaling the tokenizer corpus together with the policy's training corpus.

Quickstart

pip install nat-tokenizer
from nat_tokenizer import NATActionTokenizer

tokenizer = NATActionTokenizer.from_pretrained()

Resources

BibTeX

@misc{jawaid2026nat,
  title={NAT: Scaling Causal Neural Action Tokenizers Across Embodiments},
  author={Ahad Jawaid and Alexander Swerdlow and Yu Xiang},
  year={2026}
}

back to top