Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Humanity’s Last Exam
Show others and affiliations
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace indifficulty: LLMs now achieve over 90% accuracy on popular benchmarks likeMMLU, limiting informed measurement of state-of-the-art LLM capabilities. Inresponse, we introduce HUMANITY’S LAST EXAM (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-endedacademic benchmark of its kind with broad subject coverage. HLE consists of2,500 questions across dozens of subjects, including mathematics, humanities, andthe natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading.Each question has a known solution that is unambiguous and easily verifiable, butcannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap betweencurrent LLM capabilities and the expert human frontier on closed-ended academicquestions. To inform research and policymaking upon a clear understanding ofmodel capabilities, we publicly release HLE at https://lastexam.ai.

Keywords [en]
benchmarks, AI, LLM
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:su:diva-246355OAI: oai:DiVA.org:su-246355DiVA, id: diva2:1994017
Available from: 2025-09-01 Created: 2025-09-01 Last updated: 2025-09-15Bibliographically approved

Open Access in DiVA

fulltext(2041 kB)561 downloads
File information
File name FULLTEXT01.pdfFile size 2041 kBChecksum SHA-512
6404c1fa97dbef35677a39a66f552aabd01382adf99bc0c9bedecbdabb3bd9a9b902728e5fb4984a835d474d7a8dd34e19c1410f5f422da715129b19c96ddc4c
Type fulltextMimetype application/pdf

Other links

https://arxiv.org/pdf/2501.14249

Search in DiVA

By author/editor
Piperski, Alexander
By organisation
Slavic Languages
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 563 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 758 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf