Overslop Benchmark

We give AI coding agents real tickets from real products and grade what they code. Bugs fixed checks if the bugs are actually fixed. Slop Score counts the garbage that came with it, lower is better. The best agents fix bugs and implement features without adding unnecessary junk.

What counts as slop

  • Duplicate tests
  • Needless checks
  • Overbuilt code
  • Blocks valid use
  • Leftovers
  • Extras nobody asked for
  • Needless reformatting
  • Dropped safety checks

Leaderboard

How it’s scored

Slop Score=100⋅S20+S\text{Slop Score} = 100\cdot\frac{S}{20 + S}
S
Each kind of slop found in a case counts once at its worst: Severe 20, major 4, hygiene 1. The size of the change doesn’t count.
Bugs fixed
Share of hidden checks passed, shown beside the score. Checks that protect what already worked count double.

Missing the ticket’s own example scores 100.

Every case counts the same. Readability is graded separately.

Three AI judges flag slop line by line. Two AI reviewers check every finding. Severe findings are confirmed by running the code.

Tests run on my personal subscriptions, so each entry only runs once. High variance is expected.

LevelSlop ScoreBugs fixedAPI cost

About

We give AI coding agents real tickets from real products and grade what they code. Bugs fixed checks if the bugs are actually fixed. Slop Score counts the garbage that came with it, lower is better. The best agents fix bugs and implement features without adding unnecessary junk.

What counts as slop

  • Duplicate tests
  • Needless checks
  • Overbuilt code
  • Blocks valid use
  • Leftovers
  • Extras nobody asked for
  • Needless reformatting
  • Dropped safety checks
  1. 1GPT 6.1 TerraX-High13
  2. 2GPT 6.1 TerraMedium19
  3. 3GPT 6.1 TerraMax20
  4. 4Opus 5.5X-High20
  5. 5GPT 6.1 SolLow21
  6. 6GPT 6.1 SolMedium22
  7. 7Opus 5.5High23
  8. 8Opus 5.5Medium23
  9. 9GPT 6.1 SolHigh24
  10. 10GPT 6.1 SolX-High25
  11. 11GPT 6.1 TerraLow26
  12. 12GPT 6.1 TerraHigh26
  13. 13Opus 5.5Max28
  14. 14Opus 5.5Low38
  1. Bugs fully fixed
    4 of 5
    Features delivered
    3 of 3
  2. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  3. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  4. Bugs fully fixed
    2 of 5
    Features delivered
    3 of 3
  5. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  6. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  7. Bugs fully fixed
    2 of 5
    Features delivered
    3 of 3
  8. Bugs fully fixed
    2 of 5
    Features delivered
    3 of 3
  9. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  10. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  11. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  12. Bugs fully fixed
    3 of 5
    Features delivered
    3 of 3
  13. Bugs fully fixed
    1 of 5
    Features delivered
    3 of 3
  14. Bugs fully fixed
    2 of 5
    Features delivered
    3 of 3

By case

The bug: A service archives livestream audio in 300-second chunks. A chunk that comes back slightly short gets thrown out, even though the audio is perfectly fine and the service is about scale, not each microsecond.

The slop: Demanding chunks that are exactly 300 seconds long. Checks for the Check God, audio for the bin.

  1. GPT 6.1 TerraX-High
    17
    Fixed
    +173−18
    • Needless checks major
  2. GPT 6.1 TerraMedium
    100
    Not fixed
    +80−17

    Missed the ticket’s own example

    • Blocks valid use severe
    • Needless checks major
    • Overbuilt code hygiene
  3. GPT 6.1 TerraMax
    56
    Not fixed
    +622−28
    • Blocks valid use severe
    • Needless checks major
    • Duplicate tests hygiene
  4. Opus 5.5X-High
    100
    Not fixed
    +118−2

    Missed the ticket’s own example

    • Needless checks major
  5. GPT 6.1 SolLow
    100
    Not fixed
    +96−16

    Missed the ticket’s own example

    • No slop found
  6. GPT 6.1 SolMedium
    100
    Not fixed
    +101−22

    Missed the ticket’s own example

    • Needless checks major
  7. Opus 5.5High
    100
    Not fixed
    +113−9

    Missed the ticket’s own example

    • Leftovers hygiene
    • Duplicate tests hygiene
  8. Opus 5.5Medium
    100
    Not fixed
    +126−23

    Missed the ticket’s own example

    • Needless checks major
    • Duplicate tests hygiene
  9. GPT 6.1 SolHigh
    100
    Not fixed
    +166−25

    Missed the ticket’s own example

    • Needless checks major
  10. GPT 6.1 SolX-High
    100
    Not fixed
    +248−16

    Missed the ticket’s own example

    • Needless checks major
    • Duplicate tests hygiene
  11. GPT 6.1 TerraLow
    100
    Not fixed
    +73−11

    Missed the ticket’s own example

    • Needless checks major
  12. GPT 6.1 TerraHigh
    100
    Not fixed
    +132−24

    Missed the ticket’s own example

    • Blocks valid use severe
  13. Opus 5.5Max
    100
    Not fixed
    +84−3

    Missed the ticket’s own example

    • No slop found
  14. Opus 5.5Low
    100
    Not fixed
    +113−7

    Missed the ticket’s own example

    • Needless checks major
    • Extras nobody asked for major