← Home

Abstract

Motivation: Actions as Signals

The action tokenizer is a critical design choice in VLA pretraining. Early methods relied on uniform per-dimension binning, producing prohibitively long token sequences for high-frequency control. Subsequent approaches like FAST used frequency-domain compression to shorten sequences. The primary difficulty lies in capturing the statistical regularities that let a policy model the underlying action distribution; a secondary challenge is latency, dictated by the compression rate and the tokenizer's decoding speed.

The audio generation domain has extensively studied and largely mitigated these same challenges. Audio and robotic actions share a continuous time-series structure but differ notably: action sequences operate at lower frequencies (\(30\text{–}60\,\text{Hz}\)) than audio (\(16\text{–}48\,\text{kHz}\)), and are multi-channel (typically 7–14 dimensions for robot joints or end-effectors). Despite these differences, the core objective of compressing continuous spatio-temporal signals remains fundamentally similar. Our key finding is that adapting audio codecs to actions primarily requires rethinking frequency-domain objectives: by removing mel-frequency losses, multi-scale RVQGAN models compress action sequences without major architectural changes.

Method: Neural Action Codec

NAC maps continuous action chunks to 1D signals, compresses them via a convolutional encoder and multi-scale RVQ, and decodes them with a Vocoder-style decoder. We then train an autoregressive behavioral cloning policy that leverages NAC's structured token space.

Tokenizer

• 1D signal representation: The continuous action chunk is flattened along the temporal and feature dimensions into a single-channel pseudo-waveform of length \(L = H_a \cdot D_a\), where Ha is the action horizon and Da the action dimensionality.
• SEANet encoder: A fully convolutional encoder with strided downsampling, residual blocks, ELU activations, weight normalization, and reflection padding maps the waveform to a continuous latent \(z\).
• Multi-scale RVQ (MRVQ): Per-stage temporal pooling forces earlier stages to compress coarse, global trajectory structure over wide windows, while later stages capture fine-grained, high-frequency residuals. An aggressively scaled commitment loss (\(V_{\text{bins}}\) codebooks, λ_commit = 10³) bounds the latent space and prevents codebook collapse.
• Decoder & ISTFT head: A Vocos-style backbone predicts STFT magnitude and phase; an inverse short-time Fourier transform reconstructs the 1D signal, with the ISTFT hop length synchronized to the encoder's hop length.
• Adversarial training: We adapt audio discriminators for 1D action signals; ablations show the DAC discriminator gives the strongest signal for robotic actions. Critically, we drop the mel-spectrogram loss entirely, using MSE, L1, or an unscaled spectrogram loss instead.

Behavioral Cloning Policy

We train an autoregressive policy on top of the frozen NAC tokenizer. To provide structured generation, NAC uses offset codebooks: with \(n_q\) scales, the policy vocabulary size is \(|\mathcal{V}| = n_q \times V_{\text{bins}} + 1\), accounting for a beginning-of-sequence token. All tokens for scale 0 are predicted first, followed sequentially by scale 1, and so forth. At inference, the policy generates tokens with this fixed layout, partitions them into per-scale segments, recovers code indices via modulo arithmetic, and decodes through the frozen detokenizer—executing the first few steps in a receding-horizon fashion.

Experiments

We compare NAC against continuous-control and discrete-token baselines—Bin, Diffusion Policy, FAST, VQ-VLA, and OAT—spanning naive binning, diffusion-based control, hand-designed compression, and learned tokenization. All policy comparisons share observation inputs, action horizons, and training protocols, differing only in action parameterization. We evaluate on LIBERO-10 and RoboMimic in simulation and a suite of real-world manipulation tasks.

Overall Manipulation Performance

NAC achieves the highest success rate on every benchmark. On LIBERO-10, it outperforms FAST by 11.71 points and OAT by 5.56 points, with similar gains on RoboMimic. In the real world, NAC reaches 50% total success versus 40% for both OAT and FAST, with the largest gains on tasks requiring precise, localized corrections such as grasping grapes and stacking blocks.

Can Audio Codecs Model Robot Actions?

Audio codecs are viable for action tokenization, provided audio-specific assumptions are removed. Mel-spectrogram training collapses downstream performance to nearly zero, while simple signal-domain or non-mel frequency objectives produce strong policies (MSE yields the strongest control; spectrogram loss the best reconstruction MSE). Removing the discriminator causes complete downstream failure, and replacing the ISTFT head with a linear decoder worsens both reconstruction and policy success.

Table 1: Action tokenization ablations on LIBERO-10. Performance (%) is downstream policy task success rate; MSE is reconstruction error on 14,000 validation action chunks.

Compression & Latency

NAC compresses each action chunk to 12 tokens, matching the best learned tokenizers while reducing token count by nearly 19× relative to Bin and 3× relative to FAST. Though slower than hand-designed tokenizers, NAC is significantly faster than VQ-VLA and operates within a practical range for real-time control.

Reconstruction Quality

The video below compares the reconstruction fidelity of each tokenizer—how faithfully decoded trajectories track the ground-truth action signal across methods.

Real-World Manipulation

We evaluated all methods on 8 physical manipulation tasks spanning fine grasping, object placement, and deformable control, with 10 trials per task. NAC's compressed token space transfers effectively to physical control, outperforming alternatives on average.

The video below shows downstream policy rollouts, comparing closed-loop task execution across all tokenizers.

Citation

If you find this work useful, please consider citing our paper:

@misc{jawaid2026nac,
  title={NAC: Neural Action Codec for Vision-Language-Action Models},
  author={Ahad Jawaid and Yu Xiang},
  year={2026},
  institution={The University of Texas at Dallas}
}