Coding benchmark

SWE-bench Verified leaderboard

5 models have a published score. Reflection Beam 1.0 (Reflection AI) leads with 81.6, and the spread to the lowest score, gpt-oss-120b at 62.4, is 19.2 points. The highest-scoring model you can deploy on OpenLLM Buddy is Qwen3.6 27B, ranked 3 of 5 with 77.2. Sources report different settings for this test, such as test year, version, or reasoning effort. Each setting is listed in the source table, so compare close scores with care.

Best openllm buddy catalog

77.2

Qwen3.6 27B

1 models scored

Best open weights

81.6

Reflection Beam 1.0

4 models scored

Best closed api

No published score

0 models scored

SWE-bench Verified scores

Percentage of issues resolved. Scores depend on the agent scaffold, which labs describe in their reports.

  • OpenLLM Buddy catalog
  • Open weights
  • Closed API
  1. 1Reflection Beam 1.0BF1681.6
  2. 2DeepSeek V4 ProV4-Pro (Preview) Max, resolved80.6
  3. 3Qwen3.6 27BAlibaba Qwen77.2
  4. 4Nemotron 3 UltraNVIDIA70.7
  5. 5gpt-oss-120bhigh effort62.4

What it measures

500 issues from 12 open-source Python projects, each checked by human engineers to be solvable and fairly tested. The model reads the repository, writes a patch, and the project's own tests decide pass or fail.

How it is scored

Percentage of issues resolved. Scores depend on the agent scaffold, which labs describe in their reports.

Why it matters

It is the most widely reported test of practical software-engineering ability.

Official benchmark page

More coding tests

Sources

ModelTestScoreSettingSource
Reflection Beam 1.0SWE-bench Verified81.6BF16Hugging Face model card (Beam-1.0-501B-A23B)
DeepSeek V4 ProSWE-bench Verified80.6V4-Pro (Preview) Max, resolvedHugging Face model card (DeepSeek-V4-Pro)
Qwen3.6 27BSWE-bench Verified77.2—Qwen3.6-27B model card (Hugging Face)
Nemotron 3 UltraSWE-bench Verified70.7—Hugging Face model card (NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16)
gpt-oss-120bSWE-bench Verified62.4high effortgpt-oss model card paper (arXiv 2508.10925), Table 3

Scores are published results from model cards, lab reports, and public leaderboards, each linked to its source. Settings such as thinking mode or test version differ between sources and are shown next to each score. OpenLLM Buddy catalog models are uncensored fine-tunes; their scores are those of the official base model. Data collected 2026-10-30. How the index works