Gen4AVC @ ECCV 2026 · Poster
1 Virginia Tech · 2 Accenture · ✉ corresponding author
Kendall τ on the temporal-shift oracle (x) against Kendall τ with the PEAVS proxy (y), 75 AVSync15 clips. ImageBind-rel. and JavisScore tie on the perceptual axis; they share an ImageBind backbone (r = 0.93) and are not independent readings.
Deployed AV-sync metrics measure different, incompatible reliability axes. No metric wins both temporal-offset tracking and PEAVS-proxy alignment, the four disagree with each other almost completely (Krippendorff α = 0.066), and fusing them does not help. Report AV-sync as a Reliability Card, not one bare score.
Automatic AV-sync metrics are widely used to rank and train audio-visual generators, but they are rarely audited as measurement instruments. We jointly audit AV-Align, ImageBind AV-relevance, JavisScore, and Synchformer/DeSync under a common reliability protocol: controlled-distortion monotonicity, preprocessing sensitivity, rank uncertainty, cross-metric agreement, PEAVS-proxy agreement, and learned fusion.
The result is an axis split, not a single winner: Synchformer/DeSync is the strongest temporal-offset tracker (τ = 0.84), ImageBind/JavisScore better match the PEAVS human-aligned proxy (τ = 0.20) and content-disruption families, and AV-Align is the weakest standalone metric. The metrics mutually disagree (Krippendorff α = 0.066), and neither linear nor simple k-NN fusion improves PEAVS agreement over the best individual metric. We recommend reporting AV-sync as a Reliability Card (metric-family breakdowns with confidence intervals) rather than a single bare synchronization score.
Each finding is filed under the reliability axis it belongs to. Every number below is measured on AVSync15 clips under the shared protocol, with clip-bootstrap confidence intervals.
Synchformer/DeSync tracks a known global audio-visual shift at τ = 0.84 [0.78, 0.90] and audio speed at 0.76, against 0.16–0.39 for every other metric. It is also the only metric whose adjacent-level discriminability clears d′ ≥ 1 (d′ = 1.6 on temporal shift); everywhere else adjacent desync levels overlap by more than 16%.
On fragment shuffle ImageBind leads (0.38 vs. Synchformer's 0.27) and on intermittent mute JavisScore leads (0.73 vs. 0.59). Both embedding metrics also give the highest agreement with the PEAVS human-aligned proxy, tying at τ = 0.20 while Synchformer reaches only 0.07. Competence is axis-specific: the offset predictor and the embedding metrics are good at different things.
It is near chance on audio speed (0.04) and fragment shuffle (0.01), scores 0.00 against the PEAVS proxy, and is the only high-variance metric under should-not-change resampling: CV = 0.54 against ≤ 0.15 for the rest. Its temporal rank-flip probability is 0.66 — a coin toss on the ordering it is asked to produce. Running the official TempoTokens implementation through the same oracle reproduces every aggregate conclusion, so this is not a reimplementation artifact.
Treating each metric as a rater of clip sync, cross-metric agreement is Krippendorff α = 0.066 on z-scored per-clip scores — against an inter-annotator anchor of α ≈ 0.71 among PEAVS's human raters. Every pairwise Kendall τ is near zero except ImageBind-rel. vs. JavisScore (+0.78), which share a backbone and are not independent.
A PCA shows the four metrics span three orthogonal axes. A single sync number projects three dimensions onto one.
A ridge meta-metric over the four scores, evaluated strictly out-of-fold with split-conformal prediction intervals, never beats the best single metric (held-out τ ≤ 0.12 vs. 0.20 against PEAVS). This is not a limitation of linearity: a leave-one-out k-NN combiner does no better (≤ 0). Naive fusion is insufficient here, though it does not rule out supervised calibration against direct human labels.
We generated audio for 45 AVSync15 videos with three MMAudio checkpoints and asked each metric to rank them under a paired clip-bootstrap. A far pair (small-16k vs. large-44k) is easy for everyone. On the close pair (large-44k vs. large-44k-v2) — the leaderboard regime of similar systems — only Synchformer separates the generators.
| Metric | Far model gap | Close model gap |
|---|---|---|
| AV-Align | ≤ 0.09 | 0.35 |
| ImageBind-rel. | ≤ 0.09 | 0.33 |
| JavisScore | ≤ 0.09 | 0.35 |
| Synchformer/DeSync | ≤ 0.09 | 0.08 |
Table 2 — Leaderboard risk on generated outputs. Paired clip-bootstrap flip probability (lower is better).
Re-running the oracle on the 45 generated clips reproduces the ranking (Synchformer best, τ = 0.76; AV-Align worst), so the split is not an artifact of natural clips.
One table, six axes, four deployed metrics. Best per column in bold; higher τ is better, lower CV and rank-flip are better. Read down the columns: the leader changes from column to column.
| Metric | Oracle tracking — Kendall τ ↑ | Preproc. CV ↓ | Rank-flip ↓ | PEAVS proxy τ ↑ | ||||
|---|---|---|---|---|---|---|---|---|
| Shift | Speed | Shuffle | Mute | Shift | Worst fam. | |||
| AV-Align (reimpl.) | 0.21 | 0.04 | 0.01 | 0.24 | 0.54 | 0.66 | 0.96 | 0.00 |
| ImageBind-rel. | 0.16 | 0.37 | 0.38 | 0.61 | 0.15 | 0.54 | 0.69 | 0.20 |
| JavisScore | 0.39 | 0.51 | 0.31 | 0.73 | 0.14 | 0.46 | 0.90 | 0.20 |
| Synchformer/DeSync | 0.84 | 0.76 | 0.27 | 0.59 | n/a | 0.19 | 1.00 | 0.07 |
Table 1 — Main reliability scorecard on AVSync15. Synchformer/DeSync leads temporal tracking; ImageBind and JavisScore lead PEAVS-proxy agreement (a tie); AV-Align is the weakest standalone metric. No metric wins both the temporal-oracle and the PEAVS-proxy axis. Rank-flip Shift / Worst fam. are the pure-shift and maximum-family adjacent mis-ordering probabilities. Synchformer CV is n/a (fixed ≥ 5 s input). The AV-Align row is our reimplementation; official TempoTokens scoring gives the same aggregate conclusion. Confidence intervals are in the paper's supplement.
Worst-family flip is high for every metric — but it is each metric's weak family. On the temporal-shift ranking that matters most, Synchformer flips at 0.19 (and 0.00 on speed) where the others flip at 0.46–0.66. Leaderboard gaps below a metric's minimum detectable difference (13–17% of range) are noise.
The audit distills into a reusable reporting standard. An AV-sync metric should be characterized on all seven axes below — we recommend authors report this card for any metric they use to rank or train models.
| Reliability axis | What it tests | Report |
|---|---|---|
| Temporal oracle | global-offset sensitivity | τ + CI |
| Distortion oracle | speed / shuffle / mute sensitivity | τ + CI |
| Preprocessing sensitivity | crop and length invariance | CV + CI |
| Ranking resolution | leaderboard uncertainty | flip prob. |
| PEAVS proxy | perceptual alignment | τ + CI |
| Cross-metric consistency | agreement with peers | rank τ / α |
| Direct human calibration | direct preference check | pairwise acc. |
The first two rows are the axes this audit shows are not interchangeable; the last row is the one nobody can fill in yet, and the reason the perceptual column is read as PEAVS agreement rather than perceptual ground truth.
The AV-generation community ranks — and increasingly trains — models on synchronization metrics that mutually disagree at α = 0.066, against ≈ 0.71 among PEAVS's human annotators. An unreliable metric does not just mis-rank a leaderboard; in a preference-optimization pipeline it corrupts the optimization itself.
It is the weakest standalone tracker and the most preprocessing-sensitive metric in the set (CV 0.54, temporal rank-flip 0.66, PEAVS τ 0.00). If it appears at all, it should appear beside metrics that behave differently.
It is far and away the best offset tracker (τ = 0.84, temporal rank-flip 0.19) and the only metric that separates close generator checkpoints. But its competence is axis-specific: it is weak on content-disruption families and agrees with the perceptual proxy at only τ = 0.07.
Neither ridge fusion nor a k-NN combiner beats the best single metric against PEAVS. There is no free combination of these four scores that recovers a human-aligned one.
Metric-family scores, uncertainty, and rank resolution — the seven axes above. And check whether the gap you are claiming exceeds the metric's minimum detectable difference before you claim it.
Scope of the clip pool. The primary audit is 75 AVSync15 clips, a curated, highly-synced pool. A 150-clip replication and a 75-clip in-the-wild VGGSound rerun both reproduce the ordering, and absolute τ is lower in the wild as expected — but this is not a claim about every AV domain.
PEAVS is a proxy, not human labels. Perceptual grounding uses the PEAVS metric (human-aligned, Pearson 0.79) as a scalable stand-in for fresh human ratings. PEAVS is itself a learned model and may share representational biases with the embedding metrics, so this axis is read as PEAVS agreement, not perceptual ground truth. Direct human preference annotation remains future work before any claim of perceptual superiority.
Controlled distortions, not generative artifacts. The oracle imposes known synthetic desync, which is what makes it annotation-free. The MMAudio study complements it with real generated outputs, but it is a leaderboard-resolution stress test on 45 clips, not a full generator benchmark.
Fusion was tested, not exhausted. Ridge and k-NN combiners fail here; supervised calibration against direct human labels is untested and remains open.
Peer-reviewed and accepted as a poster; cite it as a workshop presentation. Authors retain copyright; the paper is released under CC BY 4.0.
@misc{sharma2026avsync,
title = {What Do Audio-Visual Synchronization Metrics Actually Measure?},
author = {Sharma, Jai Kumar and Tapadiya, Peeyush},
year = {2026},
howpublished = {Gen4AVC Workshop at ECCV 2026},
note = {Workshop poster}
}