Subscore

QualityTesting methodology

Quality measures how good the finished videos actually look.

AI video can go wrong in a lot of ways. The movement may look unnatural, the character can change halfway through, or the background can start flickering and warping.

We generate three videos with the same prompt and check the motion, prompt accuracy, character consistency, major visual problems, and stability from start to finish.

5 evidence groups
5 scored tests

Quality is organized into 5 evidence groups. Each group contains one scored test: Motion, Prompt Accuracy, Character Consistency, Visual Errors, and Frame Consistency.

How the Quality score is calculated

Every test gets a score from 0 to 10.

We multiply each test score by how much it counts. We then add all the points together to calculate the final Quality score.

Quality makes up 33% of the Video score.

The stored weights are converted into percentages that add up to 100%. Quality makes up 33% of the Video score.

20.48%Motion
20.48%Prompt Accuracy
20.48%Character Consistency
19.28%Visual Errors
19.28%Frame Consistency

Weighted evidence groups (combined 100%)Quality score33% of Video score

View exact calculation

Each test gets a score from 0 to 10. We multiply that score by how much the test counts. We then add all the points together.

Quality makes up 33% of the Video score. Video makes up 10% of the overall performance score.

Scored tests and weights

Scored test	How much it counts	See scoring
Motion	20.48%	View
Accuracy	20.48%	View
Character Consistency	20.48%	View
Visual Errors	19.28%	View
Frame Consistency	19.28%	View
Total	100%

How the score is calculated

Each test gets a score from 0–10
Test score × how much it counts
Example: 8 × 20.48% = 1.64 points
We do this for every test
We add all the points together
Final Quality score
Counts for 33% of Video

Example calculation

We multiply each test score by how much it counts. We then add all the points together.

Scored test	Test score	How much it counts	Calculation	Points added
Motion	8.00	20.48%	8.00 × 20.48%	1.64
Accuracy	8.67	20.48%	8.67 × 20.48%	1.78
Character Consistency	8.00	20.48%	8.00 × 20.48%	1.64
Visual Errors	6.67	19.28%	6.67 × 19.28%	1.29
Frame Consistency	7.33	19.28%	7.33 × 19.28%	1.41
Final Quality score	—	100%	Add all points	7.75/10

Special cases

Not Applicable

If a test does not apply, we remove it and spread its weight across the remaining tests.

Unknown

If we cannot verify a result, the test receives a score of 0.

Manual adjustment

In rare cases, we may adjust a score when the calculated result is clearly unfair. We always record the reason.

Why Quality matters

Video generation is still fairly new, and the quality difference between apps is massive.

One app may create smooth movement while keeping the character recognizable from beginning to end. Another may start with a good image and completely fall apart after two seconds.

The face can change. The body can warp. Clothing can disappear. Hands can break, and the background can start flickering.

This is especially frustrating because video generation often costs a lot of credits. A broken video may mean paying again and hoping the next result works.

That is why we do not judge the generator from one lucky clip. We generate several videos and watch every one from beginning to end.

What good Quality looks like

A high Quality score means the movement looks natural, the video follows the prompt, and the character stays recognizable throughout the clip. A lower score means videos regularly contain warping, flickering, broken movement, or major character changes.

Main benchmarks

GoodTypicalWeak

Motion

80%+60–79%Under 60%

Prompt Accuracy

80%+60–79%Under 60%

Character Consistency

80%+60–79%Under 60%

Minimum useful standard

Motion: 80%+
Prompt Accuracy: 80%+
Character Consistency: 80%+
Visual Errors: 20% or less
Frame Consistency: 80%+

Common red flags

The face changes halfway through the video
The body or clothing suddenly warps
Movement looks stiff or physically impossible
Important parts of the prompt are ignored
Objects appear, disappear, or merge together
The background flickers throughout the clip

How we test Quality

We use a paid account and generate three videos with the same prompt.

We upload and review every finished video.

For each video, we rate motion quality, prompt accuracy, character consistency, and visual stability.

We also mark whether the video contains a major visual error.

Using the same prompt makes the results easier to compare. It also helps us see whether the generator performs consistently or produces one good result and two broken ones.

The active testing worksheet uses three videos.

Limitations and edge cases

Video generation always includes some randomness.

The same prompt can create a strong result once and a broken result the next time. Testing three videos reduces the effect of luck, but it cannot remove it completely.

The source image also matters. A simple portrait with a clean background may work much better than a full-body image with several people or objects.

Longer videos are usually harder to keep stable. The longer the clip continues, the more chances the face, body, clothing, or background has to change.

Quality only looks at the finished video. Generation speed, failures, and ease of use are tested separately under Video Experience.

Evidence groups

Quality has 5 evidence groups made up of 5 scored tests.

Motion measures how natural the movement looks.

A video should not feel like a still image being stretched, shaken, or awkwardly pushed around.

Motion

20.48%of Motion (20.48% total)

How natural the movement looks.

How we test

We watch all three videos and give each one a Motion rating from 1 to 5. We look at body movement, facial movement, hand movement, camera movement, and whether the movement follows believable physics.

What counts

Natural body movement
Facial expressions that change smoothly
Hands that move without breaking
Camera movement that feels stable
Hair and clothing reacting naturally
Movement that fits the scene

What does not count

A still image with almost no real movement
Random shaking
Limbs moving in impossible ways
A floating or sliding character
Camera movement that makes the clip difficult to watch
Warping already counted as a major visual error

Result shown

Average Motion rating: 4 out of 5

Video 1: 4/5 · Video 2: 4/5 · Video 3: 4/5

Scoring

A higher average rating means a higher score.

Average rating	Score
1 out of 5	2/10
2 out of 5	4/10
3 out of 5	6/10
4 out of 5	8/10
5 out of 5	10/10

An average Motion rating of 4 out of 5 scores 8/10.

The exact average is used between these points.

Evidence group 2 of 5

Prompt Accuracy measures how closely the video follows the instructions you gave it.

A video can look impressive but still be a bad result when the character does something completely different from what you asked for.

Prompt Accuracy

20.48%of Prompt Accuracy (20.48% total)

How closely the video follows the instructions you gave it.

How we test

We use one prompt with five clear instructions. We rate every video from 1 to 5 based on how many important parts of the prompt it follows. The instructions may cover the requested action, the character’s movement, the setting, the camera behavior, and an object or other important detail.

What counts

The requested action happens
The character moves in the correct way
The scene matches the prompt
Important objects remain visible
Camera movement follows the instruction
The result clearly matches the main idea

What does not count

A good-looking video that ignores the requested action
A tiny movement when the prompt asks for a full action
Important prompt details appearing only briefly
An unrelated camera movement
A result that uses the source image but ignores the written prompt

Result shown

Average Prompt Accuracy rating: 4.33 out of 5

Video 1: 4/5 · Video 2: 5/5 · Video 3: 4/5

Scoring

A higher average rating means a higher score.

Average rating	Score
1 out of 5	2/10
2 out of 5	4/10
3 out of 5	6/10
4 out of 5	8/10
5 out of 5	10/10

An average rating of 4.33 out of 5 scores 8.67/10.

The exact average is used between these points.

Evidence group 3 of 5

Character Consistency measures whether the character stays recognizable throughout the video.

The face, hair, body, clothing, and other important features should not suddenly change from one frame to the next.

Character Consistency

20.48%of Character Consistency (20.48% total)

Whether the character stays recognizable throughout the video.

How we test

We watch all three videos from beginning to end. We rate every video from 1 to 5 based on how well it keeps the face, hair, body, clothing, and main identifying features.

What counts

The face remains recognizable
Hair stays the same
Body type and proportions stay similar
Clothing remains consistent
Important character details do not disappear
Normal changes caused by movement or camera angle

What does not count

The face changing into another person
Hair color or style changing without a reason
The body becoming much larger or smaller
Clothing changing or disappearing
Tattoos, accessories, or other details moving around
A character that only matches in the first frame

Result shown

Average Character Consistency rating: 4 out of 5

Video 1: 4/5 · Video 2: 4/5 · Video 3: 4/5

Scoring

A higher average rating means a higher score.

Average rating	Score
1 out of 5	2/10
2 out of 5	4/10
3 out of 5	6/10
4 out of 5	8/10
5 out of 5	10/10

An average rating of 4 out of 5 scores 8/10.

The exact average is used between these points.

Evidence group 4 of 5

Visual Errors measures how many videos contain at least one major problem.

A video counts as having a major error when the problem is serious enough to make the result clearly broken or difficult to use.

Visual Errors

19.28%of Visual Errors (19.28% total)

How many videos contain at least one major problem.

How we test

We watch all three videos and complete a defect checklist. A video is marked as having a major error when it contains at least one serious issue.

What counts

A broken or changing face
An extra or missing limb
A badly damaged hand
A sudden body change
Broken or impossible movement
Major clothing damage
A heavily distorted background
Objects merging into the character

What does not count

A small visual problem that does not affect the video
A prompt detail being missed
Slightly awkward but usable movement
A small background flicker
A style choice that is clearly intentional
A failed generation that produced no video, which is tested under Experience

Result shown

Visual error rate: 33.3%

1 of 3 videos had a major visual error

Scoring

Fewer videos with major errors means a higher score.

Videos with a major error	Score
0 of 3	10/10
1 of 3	6.67/10
2 of 3	3.33/10
3 of 3	0/10

A result of 1 video with a major error scores 6.67/10.

Evidence group 5 of 5

Frame Consistency measures how stable the video stays from beginning to end.

A video may keep the same general character but still flicker, warp, or change small details every few frames.

Frame Consistency

19.28%of Frame Consistency (19.28% total)

How stable the video stays from beginning to end.

How we test

We watch all three videos from beginning to end. We rate every video from 1 to 5 based on whether it avoids major face changes, body changes, clothing changes, object changes, and background flicker.

What counts

The face remains stable between frames
Body proportions do not pulse or warp
Clothing stays consistent
Objects remain in the same place
The background does not constantly flicker
Movement flows smoothly from one frame to the next

What does not count

Normal movement
A requested camera change
Lighting changing naturally
Small changes that are difficult to notice
A character identity problem already covered under Character Consistency
One major defect already counted under Visual Errors

Result shown

Average Frame Consistency rating: 3.67 out of 5

Video 1: 4/5 · Video 2: 3/5 · Video 3: 4/5

Scoring

A higher average rating means a higher score.

Average rating	Score
1 out of 5	2/10
2 out of 5	4/10
3 out of 5	6/10
4 out of 5	8/10
5 out of 5	10/10

An average rating of 3.67 out of 5 scores 7.33/10.

The exact average is used between these points.

Back to Video