# What Do Audio-Visual Synchronization Metrics Actually Measure?

> A joint reliability audit of the deployed audio-visual synchronization metrics
> (AV-Align, ImageBind AV-relevance, JavisScore, Synchformer/DeSync). The metrics
> measure different, incompatible reliability axes, so AV-sync should be reported
> as a Reliability Card rather than one bare score.

Authors: Jai Kumar Sharma (Virginia Tech, jaisharma@vt.edu, corresponding author);
         Peeyush Tapadiya (Accenture, peeyush.tapadiya@accenture.com)
Venue:   Gen4AVC Workshop at ECCV 2026 — accepted as a poster (Submission 2)
Date:    September 8, 2026
Status:  Peer-reviewed workshop paper (poster); authors retain copyright
License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
Keywords: audio-visual synchronization, synchronization metrics, evaluation metrics,
          meta-evaluation, metric reliability, audio-visual generation

## Links

- Project page: https://jaishrm07.github.io/avsync-reliability-card/
- Paper (PDF): https://jaishrm07.github.io/avsync-reliability-card/static/paper.pdf
- Workshop: https://gen4avc.github.io/eccv2026/
- OpenReview (forum, incl. reviews): https://openreview.net/forum?id=z84NDA7WHH
- OpenReview PDF: https://openreview.net/pdf?id=z84NDA7WHH
- Google Scholar (this paper): https://scholar.google.com/citations?view_op=view_citation&hl=en&user=IRdgHHsAAAAJ&citation_for_view=IRdgHHsAAAAJ:u5HHmVD_uO8C
- Google Scholar (author, Jai Kumar Sharma): https://scholar.google.com/citations?user=IRdgHHsAAAAJ
- Blog post (plain-language summary of this paper, by the first author):
  https://medium.com/@deeplearningjai/we-audited-the-four-av-sync-metrics-everyone-uses-they-dont-agree-df4416ccb629
- Code: not public yet (TODO: add repository link once the audit harness is released)

## TL;DR

Deployed audio-visual synchronization metrics measure different, incompatible
reliability axes. No metric wins both temporal-offset tracking and PEAVS-proxy
alignment, the four deployed metrics disagree with one another almost completely
(Krippendorff alpha = 0.066, against an inter-annotator anchor of about 0.71), and
neither linear nor simple nonlinear fusion improves on the best single metric.
Report AV-sync as a Reliability Card, not one bare score.

## Abstract

Automatic AV-sync metrics are widely used to rank and train audio-visual generators,
but they are rarely audited as measurement instruments. We jointly audit AV-Align,
ImageBind AV-relevance, JavisScore, and Synchformer/DeSync under a common reliability
protocol: controlled-distortion monotonicity, preprocessing sensitivity, rank
uncertainty, cross-metric agreement, PEAVS-proxy agreement, and learned fusion. The
result is an axis split, not a single winner: Synchformer/DeSync is the strongest
temporal-offset tracker (tau = 0.84), ImageBind/JavisScore better match the PEAVS
human-aligned proxy (tau = 0.20) and content-disruption families, and AV-Align is the
weakest standalone metric. The metrics mutually disagree (Krippendorff alpha = 0.066),
and neither linear nor simple k-NN fusion improves PEAVS agreement over the best
individual metric. We recommend reporting AV-sync as a Reliability Card (metric-family
breakdowns with confidence intervals) rather than a single bare synchronization score.

## Method (protocol)

- Synthetic-desync oracle: known-magnitude desync imposed on real, well-synced clips
  from four families (temporal shift, audio speed, fragment shuffle, intermittent
  mute), each an increasing-severity grid, so the ground-truth ordering is known by
  construction and no human annotation is needed. Scored by per-clip Kendall tau
  against the ideal "more distortion is worse" order.
- Preprocessing-sensitivity harness: random fixed-length crops and length truncation
  (changes that should not alter true sync), scored by coefficient of variation, plus
  the bootstrap rank-flip probability between conditions one desync step apart.
- Cross-metric agreement: each metric treated as a rater; pairwise Kendall tau and
  Krippendorff alpha on per-metric z-scored clip scores. PEAVS's human inter-annotator
  alpha of about 0.71 is the anchor.
- Learned meta-metric: ridge regression of the PEAVS score on the four metric scores,
  strictly out-of-fold (5-fold and leave-one-out), with split-conformal prediction
  intervals; compared against every single metric.
- Clips: AVSync15 (15 classes, 1,500 real highly-synced clips from VGGSound); primary
  audit on 75 clips (5 per class); 150-clip replication; 75 in-the-wild VGGSound clips
  as a second domain.
- Statistics: Kendall tau-b, clip-bootstrap 95% CIs at >= 1k resamples, preprocessing
  CV, bootstrap rank-flip.

## Key findings (with numbers)

1. Synchformer/DeSync is the strongest temporal-offset tracker: temporal-shift
   Kendall tau = 0.84 [0.78, 0.90] and audio-speed tau = 0.76, versus 0.16-0.39 for the
   others. It is the only metric with adjacent-level discriminability d' >= 1 (d' = 1.6
   on temporal shift); elsewhere adjacent levels overlap by more than 16%.
2. Competence is axis-specific. Synchformer does not generalize to content disruption:
   on fragment shuffle ImageBind leads (0.38 vs. 0.27) and on intermittent mute
   JavisScore leads (0.73 vs. 0.59).
3. ImageBind AV-relevance and JavisScore tie for the highest PEAVS-proxy agreement
   (tau = 0.20); Synchformer reaches only 0.07 and AV-Align 0.00. No metric is strong on
   both the temporal-oracle axis and the PEAVS-proxy axis; the two axes are nearly
   orthogonal.
4. AV-Align is the weakest and most preprocessing-sensitive standalone metric: near
   chance on audio speed (0.04) and fragment shuffle (0.01), preprocessing CV = 0.54
   against <= 0.15 for the rest, temporal rank-flip 0.66, PEAVS proxy 0.00. The official
   TempoTokens implementation reproduces every aggregate conclusion (temporal tau about
   0.22, CV 0.34, temporal flip 0.68), so this is not a reimplementation artifact.
5. The metrics barely agree: Krippendorff alpha = 0.066 on z-scored per-clip scores,
   against an inter-annotator anchor of alpha about 0.71 among PEAVS's human raters.
   Every pairwise Kendall tau is near zero except ImageBind-rel. vs. JavisScore (+0.78),
   which share an ImageBind backbone (r = 0.93) and are not independent. A PCA shows the
   four metrics span three orthogonal axes. Alpha by setting: 0.07 on clean clips
   (n = 75), 0.21 on the controlled desync grid (n = 1650), 0.11 on generated clips
   (n = 45).
6. Fusion does not help: a ridge meta-metric evaluated strictly out-of-fold with
   split-conformal intervals never beats the best single metric (held-out tau <= 0.12
   vs. 0.20 against PEAVS), and a leave-one-out k-NN combiner does no better (<= 0).
7. Leaderboard resolution: on audio generated for 45 AVSync15 videos with three MMAudio
   checkpoints, a far model pair is easy for every metric (paired clip-bootstrap flip
   probability <= 0.09), but on a close pair (large-44k vs. large-44k-v2) only
   Synchformer separates the generators (flip probability 0.08, versus 0.35 for
   AV-Align, 0.33 for ImageBind-rel. and 0.35 for JavisScore). Leaderboard gaps below a
   metric's minimum detectable difference (13-17% of range) are noise.
8. Replication: a 150-clip run reproduces every structural conclusion (Synchformer
   temporal tau = 0.87, alpha = 0.07), and 75 in-the-wild VGGSound clips reproduce the
   ordering (Synchformer temporal tau = 0.63 versus <= 0.27 for the others; embedding
   metrics lead on content disruption; AV-Align weakest).

## Main reliability scorecard (AVSync15, Table 1 of the paper)

Columns: oracle tracking Kendall tau (shift, speed, shuffle, mute; higher is better),
preprocessing CV (lower is better), rank-flip probability on pure shift and on the
worst family (lower is better), PEAVS-proxy tau (higher is better).

Metric              shift  speed  shuffle  mute   CV     flip-shift  flip-worst  PEAVS
AV-Align (reimpl.)  0.21   0.04   0.01     0.24   0.54   0.66        0.96        0.00
ImageBind-rel.      0.16   0.37   0.38*    0.61   0.15   0.54        0.69*       0.20*
JavisScore          0.39   0.51   0.31     0.73*  0.14*  0.46        0.90        0.20*
Synchformer/DeSync  0.84*  0.76*  0.27     0.59   n/a    0.19*       1.00        0.07

* = best in column. Synchformer CV is n/a (fixed >= 5 s input).

## The Reliability Card (the paper's reporting recommendation)

Report an AV-sync metric on all seven axes, not as one bare number:

axis                       what it tests                     report
Temporal oracle            global-offset sensitivity         tau + CI
Distortion oracle          speed/shuffle/mute sensitivity    tau + CI
Preprocessing sensitivity  crop and length invariance        CV + CI
Ranking resolution         leaderboard uncertainty           flip probability
PEAVS proxy                perceptual alignment              tau + CI
Cross-metric consistency   agreement with peers              rank tau / alpha
Direct human calibration   direct preference check           pairwise accuracy

## Practical guidance

- Do not report AV-Align alone: weakest standalone tracker, most preprocessing-sensitive.
- For temporal synchronization prefer Synchformer/DeSync (tau = 0.84), but state that
  temporal offset is what was measured; its competence is axis-specific and it is weak
  on content-disruption families.
- Do not expect a meta-metric to combine the metrics into a human-aligned score.
- Report a Reliability Card: metric-family scores, uncertainty, and rank resolution.

## Limitations

- Scope: primary audit on 75 curated AVSync15 clips, with a 150-clip replication and a
  75-clip in-the-wild VGGSound rerun; absolute tau is lower in the wild, but the
  ordering holds.
- PEAVS is a proxy, not human labels: it is a learned, human-aligned model (Pearson
  0.79) that may share representational biases with the embedding metrics, so that axis
  is read as PEAVS agreement rather than perceptual ground truth. Direct human
  preference annotation remains future work.
- The oracle imposes controlled distortions, not generative artifacts; the MMAudio study
  is a leaderboard-resolution stress test on 45 clips, not a full generator benchmark.
- Fusion was tested, not exhausted: supervised calibration with direct human labels is
  untested.

## BibTeX

@misc{sharma2026avsync,
  title        = {What Do Audio-Visual Synchronization Metrics Actually Measure?},
  author       = {Sharma, Jai Kumar and Tapadiya, Peeyush},
  year         = {2026},
  howpublished = {Gen4AVC Workshop at ECCV 2026},
  note         = {Workshop poster}
}