Coding benchmark

SWE-bench Pro leaderboard

9 models have a published score. Claude Opus 5.5 (Anthropic) leads with 89.9, and the spread to the lowest score, Qwen3.6 27B at 53.5, is 36.4 points. The best open-weight result, Reflection Beam 1.0 at 66.1, trails the best closed model, Claude Opus 5.5, by 23.8. The highest-scoring model you can deploy on OpenLLM Buddy is Qwen3.8 27B, ranked 7 of 9 with 61.7. Sources report different settings for this test, such as test year, version, or reasoning effort. Each setting is listed in the source table, so compare close scores with care.

Best openllm buddy catalog

61.7

Qwen3.8 27B

2 models scored

Best open weights

66.1

Reflection Beam 1.0

3 models scored

Best closed api

89.9

Claude Opus 5.5

4 models scored

SWE-bench Pro scores

Percentage of tasks whose patch passes the hidden tests.

  • OpenLLM Buddy catalog
  • Open weights
  • Closed API
  1. 1Claude Opus 5.5SWE-bench Pro; adaptive thinking max effort; avg of 5 trials89.9
  2. 2Claude Sonnet 5.5SWE-bench Pro; adaptive thinking max effort; avg of 5 trials81.3
  3. 3Claude Fable 5.1SWE-bench Pro; adaptive thinking max effort; avg of 5 trials81.2
  4. 4Reflection Beam 1.0SWE-Bench Pro v1, BF16 (v2-Hard: 76.5)66.1
  5. 5Tencent Hy4 PreviewTencent65.7
  6. 6Claude Haiku 5.5SWE-bench Pro; adaptive thinking max effort; avg of 5 trials64.8
  7. 7Qwen3.8 27BAlibaba Qwen61.7
  8. 8DeepSeek V4 ProV4-Pro (Preview) Max, resolved55.4
  9. 9Qwen3.6 27BAlibaba Qwen53.5

What it measures

Long-horizon issues drawn from a wider mix of repositories, including copyleft and commercial code, that often require changes across several files and languages.

How it is scored

Percentage of tasks whose patch passes the hidden tests.

Why it matters

As SWE-bench Verified saturates, Pro shows which models hold up on enterprise-scale changes.

Official benchmark page

More coding tests

Sources

ModelTestScoreSettingSource
Claude Opus 5.5SWE-bench Pro89.9SWE-bench Pro; adaptive thinking max effort; avg of 5 trialsAnthropic Claude Opus 5.5 System Card (Table 8.1.A / sec 8.2)
Claude Sonnet 5.5SWE-bench Pro81.3SWE-bench Pro; adaptive thinking max effort; avg of 5 trialsAnthropic Claude Sonnet 5.5 System Card (Table 8.1.A / sec 8.2)
Claude Fable 5.1SWE-bench Pro81.2SWE-bench Pro; adaptive thinking max effort; avg of 5 trialsAnthropic Claude Fable 5.1 / Mythos 5.1 System Card (Table 8.1.A / sec 8.2)
Reflection Beam 1.0SWE-bench Pro66.1SWE-Bench Pro v1, BF16 (v2-Hard: 76.5)Hugging Face model card (Beam-1.0-501B-A23B)
Tencent Hy4 PreviewSWE-bench Pro65.7—Hugging Face model card (tencent/Hy4-preview), evaluation results
Claude Haiku 5.5SWE-bench Pro64.8SWE-bench Pro; adaptive thinking max effort; avg of 5 trialsAnthropic Claude Haiku 5.5 System Card (Table 8.1.A / sec 8.2)
Qwen3.8 27BSWE-bench Pro61.7—Qwen3.8-27B model card (Hugging Face)
DeepSeek V4 ProSWE-bench Pro55.4V4-Pro (Preview) Max, resolvedHugging Face model card (DeepSeek-V4-Pro)
Qwen3.6 27BSWE-bench Pro53.5—Qwen3.6-27B model card (Hugging Face)

Scores are published results from model cards, lab reports, and public leaderboards, each linked to its source. Settings such as thinking mode or test version differ between sources and are shown next to each score. OpenLLM Buddy catalog models are uncensored fine-tunes; their scores are those of the official base model. Data collected 2026-10-30. How the index works