How do you grade one video?

1. Two different questions, two different toolboxes

Video evaluation research answers two questions that look similar and are not.

“Is the model good?” is a distributional question. You generate a corpus of videos, take a corpus of real ones, embed both, and measure the distance between distributions. That is FVD [FVD]: embed both sets with an I3D action classifier, fit Gaussians, compute the Fréchet distance. Its successor JEDi [JEDi] swaps in V-JEPA features and a kernel distance that drops the Gaussian assumption, and needs roughly a sixth of the samples.

“Is this clip good?” is a per-clip, reference-free, online question. A set of one video has no distribution, so the entire FVD family is structurally useless here. This is the question Take has to answer every time you press the button, and it needs a completely different toolbox: deterministic gates, per-clip metrics, and a judge.

There is a second reason not to lean on FVD-style thinking even for regression testing: it barely measures time. A CVPR 2024 study [FVD-bias] showed FVD is close to insensitive to temporal corruption and can be gamed by resampling from motion-free videos. The root cause is that I3D features are appearance-dominated; most Kinetics action classes are recognizable from single frames. Hold that thought, because the same failure reappears in a different costume two sections down.

2. What “good video” decomposes into

Nobody serious scores video with one number. VBench [VBench] decomposes quality into sixteen disentangled dimensions, each with its own purpose-built evaluator: subject consistency is DINO cosine across frames, background consistency is CLIP cosine, temporal flickering is frame differencing, dynamic degree is optical-flow magnitude, aesthetics is a learned predictor, and so on. VBench-2.0 [VBench-2.0] extends this to intrinsic faithfulness: human fidelity, controllability, creativity, physics, commonsense. EvalCrafter [EvalCrafter] found that a fitted combination of seventeen metrics tracks human opinion better than naive averaging. T2V-CompBench [T2V-CompBench] makes the sharpest version of the point: for compositional skills, the evaluator type has to be matched per dimension. MLLM judges for some, object detectors for others, trackers for motion binding. No single judge covers everything.

Two findings from this literature shaped Take directly.

First, physics is the open frontier. On VideoPhy and VideoPhy-2 [VideoPhy] [VideoPhy-2], the best video generators score roughly 22–40% on physical commonsense. Shadows that detach, liquids that flow uphill, hands that pass through cups. That's why physics is one of Take's six judge axes, and why it's restricted to violations visible in sampled frames.

Second, models cheat by standing still. DEVIL [DEVIL] documents that generators inflate quality scores by under-animating: a nearly static clip flickers less, drifts less, and scores higher on consistency. A motion-quantity check is therefore an anti-gaming signal, not a nice-to-have. Take measures optical flow on every clip and the judge is told explicitly that near-zero flow is a defect unless the prompt asked for stillness.

Synthesizing across VBench, EvalCrafter, T2V-CompBench, VideoPhy and the reward-model literature, six axes keep reappearing: per-frame fidelity, aesthetics, temporal consistency, motion quantity and plausibility, prompt semantics, and physics. Models trade these off against each other, which is exactly why Take reports all six per-axis instead of collapsing to a single score and calling it objective.

3. The judge's blind spot: time

The tempting architecture is one big multimodal model: hand it the video, ask “is this good?”, done. The published evidence says no, in a very specific way.

TempCompass [TempCompass] showed that video LLMs lean on single frames plus language priors and fail on speed, direction, and event-order variants of otherwise identical content. Vinoground [Vinoground] is the cleaner kill: temporal-counterfactual pairs on short videos, where the same events happen in a different order. Humans score about 90%. GPT-4o scores about 50%, which is chance. CLIP-based models are also at random. A vision-language judge cannot reliably verify “X happens, then Y” even on a ten-second clip.

There is also a sampling-theory problem (this one is synthesis, not a single citation): a judge that sees 6–16 sampled frames cannot detect 8–24 Hz flicker, because it aliases away between samples. It is the FVD content-bias failure again, in a different costume: an appearance-dominated evaluator confidently grading a temporal property it physically cannot observe.

Take's answer is a strict division of labor. Everything temporal is owned by deterministic code that sees every frame at full framerate: flicker by frame differencing, motion quantity by optical flow, identity drift by per-frame embeddings. The judge is explicitly instructed not to grade flicker, strobing, speed, or ordering from the contact sheet, and to read the deterministic measurements for those axes instead. The judge grades only what is actually visible in static frames: anatomy, artifacts, composition, whether the required scene elements exist, whether a shadow points the wrong way.

4. The deterministic lanes, and why these four

CLIPScore: what cosine similarity actually measures

CLIP [CLIP] trains two encoders, an image ViT and a text Transformer, on ~400M web image-text pairs with a contrastive objective: in every batch, maximize the cosine similarity of correct image-text pairs and minimize all the wrong pairings. The training objective literally is “cosine is high if and only if the text describes the image.” That is the entire reason CLIPScore [CLIPScore] works: cosine in CLIP space is a direct readout of the quantity the model was trained to encode. Take computes it between your prompt and each sampled frame, then averages.

The same objective dictates the weaknesses. Features only need to be good enough to discriminate captions within a batch, so they are semantic and categorical: word order, attribute binding, counting, and spatial relations are largely invisible to it. CLIPScore ranks on-prompt versus off-prompt usefully, but it does not calibrate to an absolute “goodness,” and it cannot tell “a red cube on a blue sphere” from the reverse. Those compositional checks belong to the judge's semantics axis.

DINO, not CLIP, for identity

Why use a second embedding model for subject consistency? Because CLIP is invariant to exactly the thing we need to detect. The argument is from DreamBooth [DreamBooth] and is now the VBench convention: CLIP features are class-level, so swapping in a different dog of the same breed barely moves CLIP similarity. DINO [DINO] [DINOv2] is trained with self-distillation and no labels: a student network must match a teacher's output across different crops of the same image. The only learning signal is “these views are the same instance,” so the features are forced to encode what makes this particular object itself: texture, parts, geometry. That makes DINO instance-discriminative where CLIP is class-discriminative. Take embeds every sampled frame with DINOv2 and tracks cosine similarity against frame zero. If the subject quietly morphs into a different subject by frame forty, this lane catches it.

Optical flow: motion as a number

VBench's dynamic degree and DEVIL's dynamics scores are statistics over optical-flow magnitude, with RAFT [RAFT] as the standard estimator. Per-clip, the readings are diagnostic at both extremes: near-zero flow is the frozen “cheat” output DEVIL warns about, and extreme chaotic flow usually means the clip fell apart. Take runs on CPU, so it uses OpenCV's Farneback estimator as a proxy; the bands below were calibrated against that estimator, not RAFT, which is the honest way to use a substitute.

Flicker: full-framerate frame differencing

Mean absolute pixel difference between consecutive frames, computed over every transition in the clip, not the sampled subset. This is VBench's temporal-flickering measure, and it exists in Take precisely because of the sampling blind spot in section 3: it is the lane the judge can never own.

Why not a learned video reward model?

The strongest per-clip graders in the literature are learned reward models: VideoScore [VideoScore] (Spearman 77.1 against human ratings), VideoScore2 [VideoScore2] (chain-of-thought before scoring), VisionReward [VisionReward] (a hierarchical checklist of binary judgments, +17.2% over VideoScore, and notably resistant to reward hacking because each judgment is interpretable), and VideoReward [VideoReward]. They are all open-weights and single-GPU-class. Take's pipeline runs on a CPU-only box today, so they are a v2 lane, not a v1 lane. The cascade is designed so a GPU reward model can slot in as an additional L1 lane without changing the architecture.

5. How the judge is built

Take's L2 judge is a frontier vision-language model reading a timestamped contact sheet: a single grid image of frames sampled uniformly across the clip, first and last frames always included. Each design choice has a paper behind it.

The contact sheet itself. IG-VLM [IG-VLM] showed that tiling sampled frames into one grid and prompting a single-image VLM beats dedicated video-LM methods on 9 of 10 zero-shot video-QA benchmarks. The grid preserves coarse temporal order through reading order, and it costs one model call instead of sixteen.

Uniform sampling, first and last pinned. Uniform is the published default for short clips (VideoScore, VBench, IG-VLM all use it, typically 6–16 frames). Adaptive keyframe sampling [AKS] matters for long video, but for a single-shot 5–10 second clip it degenerates to: spread uniformly, and always include the first and last frames. One nuance worth flagging honestly: codec keyframes (I-frames) are a compression artifact, placed by encoder budget, and are semantically meaningless for sampling decisions. Content keyframes mean scene cuts [TransNet V2], and a generated single-shot clip should have zero of them; an unexpected hard cut inside one is itself a defect.

Discrete levels, not decimals. The judge outputs excellent / good / fair / poor / bad per axis. This is the Q-Align result [Q-Align]: language models score visual quality more reliably through discrete text-defined levels than through raw numbers, because levels are how human raters actually annotate. A “7.3/10” from a language model is fake precision.

Rationale before rating. Each axis requires the judge to write its reasoning first and commit to a level second. LiFT [LiFT] showed rationale-supervised critics generalize better, and VideoScore2 [VideoScore2] trains chain-of-thought-then-score explicitly. Asking for the level first and the justification second invites post-hoc rationalization; ordering it rationale-first is free accuracy.

Decomposed rubric, not a holistic score. This is the most convergent finding in the whole dossier: VBench's disentangled dimensions expose differences a single score hides, EvalCrafter's fitted combination beats naive averaging, T2V-CompBench matches evaluator types per dimension, and VisionReward's interpretable checklist beats a monolithic scorer by double digits. No credible paper argues the reverse.

Bounded retakes. The accept-or-retake decision at L3 is deliberately deterministic: a fixed level-to-points mapping, a fixed threshold (no axis at poor or below), at most three attempts. The judge's retake advice is folded into the next composition, which makes the loop a crude best-of-N search guided by a critic, the same shape the reward-model alignment literature uses [VideoReward], with the budget capped so a bad prompt fails fast instead of burning money.

6. The cascade as economics

The four levels are ordered by cost per verdict.

1. L0 gates (ffprobe decode, duration, resolution, framerate, then black-frame and frozen-frame pixel checks) cost milliseconds and catch outright failures. A clip that fails here never wastes anything downstream.
2. L1 lanes (flicker, flow, CLIPScore, DINOv2 drift) cost CPU seconds and own every full-framerate temporal question.
3. L2 judge costs a frontier-model call and owns the things only a generalist can see: anatomy, composition, composition-level semantics, visible physics.
4. L3 decision costs nothing and exists so the expensive parts run at most three times.

This shape, deterministic filters in front of learned evaluators in front of a bounded controller, is not novel; it is what the benchmark suites institutionalize internally. What Take adds is showing you the whole trace live, per clip, while it runs.

7. First-party calibration: 14 clips, real thresholds

Papers give the architecture; they do not give thresholds for this generator, this flow estimator, and this CLIP checkpoint. So before launch we ran a 14-clip reference set through the full pipeline: prompts deliberately chosen to produce good clips, ambient near-static clips, busy multi-subject scenes, strobing, identity breaks, frozen output, and pure darkness. The numbers below are our measurements, and they are the exact bands the production judge reads.

Three lessons from the calibration that no paper handed us:

Read the minimum, not the mean. Identity breaks are single-frame events. Two clips in the set had means that looked acceptable while one sampled frame had completely lost the subject. The production judge is instructed to find and explain any frame where the per-frame DINO cosine drops below 0.5.

High flicker is not strobe. Busy multi-subject scenes produce large frame-to-frame differences continuously; strobe produces spikes. Mean-versus-max separates them cleanly, and a judge told only “flicker is high” would have failed two perfectly good clips.

Some separations belong in the judge, not the gate. Our frozen-frame gate threshold sits at a frame-diff of 0.35. The degenerate still-wall clip measured 0.70 and the genuinely good cloud timelapse measured 0.97. Raising the gate to catch the first would put the second at risk, and a gate is a blind instrument. We left the gate alone and gave the judge the ambient-flow band instead: a clip with flow 0.3–1.5 is graded against intent, not against a motion quota. Gates should only kill what is unambiguously dead.

Darkness, for the record, dies at L0: the pure-darkness clip failed both the black-frame and frozen-frame gates and never reached a judge.

8. Honest limits

The cascade has known blind spots, and pretending otherwise would defeat the point of publishing the receipts.

Event-order verification is unsolved at every level: the lanes measure how much things change, not what happened in what order, and Vinoground says the judge can't do it either. Compositional binding is covered only by the judge's semantics axis, which is better than CLIPScore but not a detector-grade check of counts and spatial relations the way T2V-CompBench does it. Physics is graded only where a violation is visible in sampled frames. The DINO identity lane is meaningless for crowd scenes, where frame zero anchors nothing, and the judge is told to ignore it there. And the learned reward models in section 4 are absent purely for hardware reasons. Each of these is an upgrade path, not a surprise.

One more disclosure: the claim that VLM judges miss high-frequency flicker is our synthesis from sampling theory plus the FVD content-bias mechanism, not a single published experiment, and the observation that generated clips degrade toward their final frames is practitioner lore. Both informed the design anyway; both are labeled.

The best way to understand the cascade is to watch it run. Roll a take and the page will show you every gate, every lane reading, and the judge thinking in real time.