Reasoning benchmark

GPQA Diamond leaderboard

19 models have a published score. GPT-6 Astra (OpenAI) leads with 96.1, and the spread to the lowest score, gpt-oss-120b at 80.1, is 16.0 points. The best open-weight result, Kimi K3.1 Pro at 94.2, trails the best closed model, GPT-6 Astra, by 1.9. The highest-scoring model you can deploy on OpenLLM Buddy is Kimi K3, ranked 5 of 19 with 93.5. Sources report different settings for this test, such as test year, version, or reasoning effort. Each setting is listed in the source table, so compare close scores with care.

Best openllm buddy catalog

93.5

Kimi K3

6 models scored

Best open weights

94.2

Kimi K3.1 Pro

10 models scored

Best closed api

96.1

GPT-6 Astra

3 models scored

GPQA Diamond scores

Accuracy over the 198 questions. Random guessing scores about 25%; domain experts score around 65%.

  • OpenLLM Buddy catalog
  • Open weights
  • Closed API
  1. 1GPT-6 AstraGPQA Diamond; AA independent run (Max)96.1
  2. 2Gemini 3.8 FlashGPQA Diamond; competitor figure reported by OpenAI95.3
  3. 3Kimi K3.1 Prono tools, max effort (vendor-reported)94.2
  4. 4Claude Fable 5.1GPQA Diamond; AA independent run (Max, Default Fallback)93.7
  5. 5Kimi K3max reasoning effort93.5
  6. 6Qwen3.8 2.4T-A95Bopen checkpoint Qwen3.8-2.4T-A95B (xhigh); competitor comparison column93.5
  7. 7Step 5 PreviewHigh effort (vendor-reported)93.5
  8. 8Tencent Hy4 PreviewTencent92.3
  9. 9GLM 5.3GLM 5.3 (max); competitor comparison column91.7
  10. 10DeepSeek V4.1 FlashPass@1, max reasoning effort90.9
  11. 11Reflection Beam 1.0BF1690.6
  12. 12DeepSeek V4 ProV4-Pro (Preview) Max, pass@190.1
  13. 13Qwen3.8 27BAlibaba Qwen89.2
  14. 14Muse Spark Openxhigh effort89.0
  15. 15Qwen3.6 27BAlibaba Qwen87.8
  16. 16Nemotron 3 Ultralabelled 'GPQA (no tools)' on card87.0
  17. 17Gemma 4 31BGoogle84.3
  18. 18Gemma 4 26B A4BGoogle82.3
  19. 19gpt-oss-120bhigh effort, no tools80.1

What it measures

The Diamond split holds 198 multiple-choice questions written and cross-checked by PhD-level domain experts. Questions were kept only when experts answered correctly and skilled non-experts with web access did not, so search alone does not solve them.

How it is scored

Accuracy over the 198 questions. Random guessing scores about 25%; domain experts score around 65%.

Why it matters

It is one of the clearest signals of whether a model can reason through unfamiliar science rather than repeat memorised text.

Official benchmark page

More reasoning tests

Sources

ModelTestScoreSettingSource
GPT-6 AstraGPQA Diamond96.1GPQA Diamond; AA independent run (Max)Artificial Analysis
Gemini 3.8 FlashGPQA Diamond95.3GPQA Diamond; competitor figure reported by OpenAIOpenAI GPT-6 Astra launch post (Academic table)
Kimi K3.1 ProGPQA Diamond94.2no tools, max effort (vendor-reported)DataCamp (relaying Moonshot's release table)
Claude Fable 5.1GPQA Diamond93.7GPQA Diamond; AA independent run (Max, Default Fallback)Artificial Analysis
Kimi K3GPQA Diamond93.5max reasoning effortKimi K3 model card (Hugging Face)
Qwen3.8 2.4T-A95BGPQA Diamond93.5open checkpoint Qwen3.8-2.4T-A95B (xhigh); competitor comparison columnMeta Muse Spark Open model card (comparison column)
Step 5 PreviewGPQA Diamond93.5High effort (vendor-reported)cellcog.ai (relaying StepFun's release table)
Tencent Hy4 PreviewGPQA Diamond92.3—Hugging Face model card (tencent/Hy4-preview), evaluation results
GLM 5.3GPQA Diamond91.7GLM 5.3 (max); competitor comparison columnMeta Muse Spark Open model card (comparison column)
DeepSeek V4.1 FlashGPQA Diamond90.9Pass@1, max reasoning effortDeepSeek-V4.1-Flash model card (Hugging Face)
Reflection Beam 1.0GPQA Diamond90.6BF16Hugging Face model card (Beam-1.0-501B-A23B)
DeepSeek V4 ProGPQA Diamond90.1V4-Pro (Preview) Max, pass@1Hugging Face model card (DeepSeek-V4-Pro)
Qwen3.8 27BGPQA Diamond89.2—Qwen3.8-27B model card (Hugging Face)
Muse Spark OpenGPQA Diamond89.0xhigh effortHugging Face model card (Muse-Spark-Open-816B-A42B)
Qwen3.6 27BGPQA Diamond87.8—Qwen3.6-27B model card (Hugging Face)
Nemotron 3 UltraGPQA Diamond87.0labelled 'GPQA (no tools)' on cardHugging Face model card (NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16)
Gemma 4 31BGPQA Diamond84.3—Gemma 4 31B IT model card (Hugging Face)
Gemma 4 26B A4BGPQA Diamond82.3—Gemma 4 31B IT model card (Hugging Face), 26B A4B column
gpt-oss-120bGPQA Diamond80.1high effort, no toolsgpt-oss model card paper (arXiv 2508.10925), Table 3

Scores are published results from model cards, lab reports, and public leaderboards, each linked to its source. Settings such as thinking mode or test version differ between sources and are shown next to each score. OpenLLM Buddy catalog models are uncensored fine-tunes; their scores are those of the official base model. Data collected 2026-10-30. How the index works