Using AudioContext suspend/resume as a Streaming A/V Sync Gate

TL;DR: Buffering N frames before playback merely guesses the network speed. AudioContext.suspend() stops audio output and suspends the progression of time in the audio context, so it can act as a synchronization gate: suspend the timeline when the next frame is missing and resume it when that frame arrives.

This article assumes familiarity with the Web Audio API, requestAnimationFrame, and frame-by-frame streaming.

The problem: frames and audio run on separate tracks

Consider a streamed lip-sync player. A server sends video frames over SSE or WebSocket while the client receives, decodes, and draws them alongside the corresponding audio.

The direct implementation starts audio immediately and uses audioContext.currentTime to choose the frame:

audioSource.start(0);
const startTime = audioContext.currentTime;

function renderLoop() {
  const elapsed = audioContext.currentTime - startTime;
  const targetFrame = Math.floor(elapsed * fps);

  if (frames[targetFrame]) {
    drawFrame(frames[targetFrame]);
  }

  requestAnimationFrame(renderLoop);
}

It works on a healthy connection. When frames arrive more slowly than audio advances:

Audio clock:   0s ───────── 2s ───────── 4s ─────────
Frames ready:  ████████████░░░░░░░░░░░░░░░░░░░░
What users see: [normal]     [frozen mouth, audio continues]

The A/V drift is immediately visible.

Why a fixed buffer threshold is not enough

A common fix waits for a fixed number of frames:

const BUFFER_THRESHOLD = 60;

onFrameDecoded(() => {
  decodedCount += 1;

  if (!started && decodedCount >= BUFFER_THRESHOLD) {
    started = true;
    audioSource.start(0);
    renderLoop();
  }
});

This postpones the problem:

• The threshold depends on network and server speed. Sixty frames may be too few or may only add startup latency.
• Any stall after playback begins can still put frames behind audio.
• Buffering for the worst case makes healthy connections wait too.

The underlying issue is an audio clock that advances without knowing whether the next frame exists.

The key property: suspend stops time progression

MDN defines AudioContext.suspend() as suspending the progression of time in the audio context while temporarily halting audio hardware access. It is more than mute:

const audioContext = new AudioContext();

// After playing for a while:
console.log(audioContext.currentTime); // for example, 2.341

await audioContext.suspend();
await sleep(3000);

console.log(audioContext.currentTime); // remains near the suspended position

await audioContext.resume();
// The timeline continues from the suspended position.

That makes AudioContext useful as a synchronization primitive:

Is the next sequential frame ready?
  yes → resume()  → time advances → draw when its timestamp arrives
  no  → suspend() → time freezes  → wait for the frame-arrival event

Initialization: suspend before scheduling audio

Audio decoding and frame delivery can run in parallel. Suspend the new context first, then schedule the buffer while it is suspended:

const AudioContextConstructor =
  window.AudioContext ??
  (
    window as typeof window & {
      webkitAudioContext: typeof AudioContext;
    }
  ).webkitAudioContext;

const audioContext = new AudioContextConstructor();
await audioContext.suspend();

const buffer = await audioContext.decodeAudioData(audioBytes);
const source = audioContext.createBufferSource();

source.buffer = buffer;
source.connect(audioContext.destination);
source.start(0);

const startTime = audioContext.currentTime;

start(0) can schedule the source while the context is suspended, but the timeline does not advance. The scheduler calls resume() after both audio and the first frame are ready.

Browser autoplay policy still applies. The first resume() that permits audible output generally needs to follow a user gesture. Production code should not swallow its rejection; show a “play” affordance or propagate the error to the caller.

Gate functions: track the desired state locally

Both resume() and suspend() are asynchronous. Reading only audioContext.state introduces a transition window: a command has been issued, but the property has not updated yet.

A synchronously updated flag can represent the desired state:

let audioContext: AudioContext | null = null;
let audioRunning = false;

function ensureRunning() {
  if (!audioRunning && audioContext) {
    audioRunning = true;
    audioContext.resume().catch((error) => {
      audioRunning = false;
      reportAudioError(error);
    });
  }
}

function ensureSuspended() {
  if (audioRunning && audioContext) {
    audioRunning = false;
    audioContext.suspend().catch(reportAudioError);
  }
}

The flag prevents duplicate commands on every animation frame. It does not replace error handling: when autoplay policy rejects resume(), reset the flag and update the UI.

The scheduler: let frames drive the clock

Two events can wake the scheduler:

1. The next sequential frame finishes decoding.
2. Audio decoding finishes.

let audioReady = false;
let lastDrawnFrame = -1;

function tryAdvance() {
  if (!isPlaying) return;

  const nextNeeded = lastDrawnFrame + 1;

  if (streamEnded && nextNeeded >= totalFrames) {
    ensureSuspended();
    isPlaying = false;
    onPlayEnd();
    return;
  }

  const nextFrame = frames[nextNeeded];

  if (nextFrame === null) {
    // A frame gap: freeze time and wait for onFrameArrived.
    ensureSuspended();
    return;
  }

  if (!audioReady || !audioContext) {
    requestAnimationFrame(tryAdvance);
    return;
  }

  ensureRunning();

  const elapsed = audioContext.currentTime - startTime;
  const frameTime = nextNeeded / fps;

  if (elapsed >= frameTime) {
    draw(nextFrame);
    lastDrawnFrame = nextNeeded;
  }

  // Wait if the timestamp is early; otherwise continue with the next frame.
  requestAnimationFrame(tryAdvance);
}

The same logic handles both directions:

• Server slower than real time: a frame gap suspends audio.
• Server faster than real time: the frame exists, but elapsed < frameTime prevents early drawing.

Wake only for the frame that matters

function onFrameArrived(index: number, bitmap: ImageBitmap) {
  frames[index] = bitmap;

  if (!isPlaying && index === 0) {
    isPlaying = true;
    tryAdvance();
    return;
  }

  if (isPlaying && index === lastDrawnFrame + 1) {
    tryAdvance();
  }
}

If frames arrive as 0, 1, 3, 2, 4, frame 3 is stored without waking playback. Frame 2 then matches lastDrawnFrame + 1 and wakes the scheduler, which proceeds through 2, 3, and 4. The index enforces order without a separate sort.

Completion and cleanup

Stopping playback should cancel scheduling, close the audio context, and release GPU resources owned by each ImageBitmap:

function stop() {
  isPlaying = false;

  if (rafId !== null) {
    cancelAnimationFrame(rafId);
    rafId = null;
  }

  audioContext?.close().catch(reportAudioError);
  audioContext = null;

  for (const bitmap of frames) {
    bitmap?.close();
  }
}

A zero-frame stream needs an explicit completion path. Because normal playback starts when frame zero arrives, totalFrames === 0 never produces that event. Handle stream completion directly instead of waiting for tryAdvance() to wake.

Audio duration, fps, and totalFrames must also describe the same timeline. If audio ends while video frames remain, currentTime still advances but users hear nothing. The gate cannot replace input-metadata validation.

Trade-offs

Benefits	Limitations
Strict A/V sync under network jitter	A stall pauses both sound and picture
No fixed N-frame startup buffer	Autoplay policy and async state need handling
No AudioWorklet required	Frequent suspension can create perceptible pauses
Out-of-order frames naturally wait by index	Unnecessary for local or fully buffered media

For local files or stable HLS playback, driving video from the audio clock is simpler. This gate is valuable when frame arrival is inherently unpredictable.

Source and further reading

• Original TIL issue
• Complete TypeScript implementation
• MDN: AudioContext.suspend()
• MDN: BaseAudioContext.currentTime