2023 Human vs. Computer Open
Date: July 23, 2023 (Sunday), 9am–12pm Location: University of Maryland, College Park, MD — Iribe Building, Ground/2nd floor Supported by: Meta
The QANTA Project hosted a competition challenging participants to write trivia questions that stump AI systems while remaining answerable by human experts.
Deadlines
| Event | Deadline |
|---|---|
| Question submissions | July 20, 2023 |
| Computer system submission | July 20, 2023 |
| Tournament | July 23, 2023 |
Competition Tracks
Track 1: Question Writing
Submit adversarial quiz bowl questions that stump AI while remaining answerable by human experts.
Submission methods:
- Web interface: qantanew.web.app/play
- Google Sheets formatted per the provided template
Scoring (25 points each):
| Criterion | Description |
|---|---|
| Discriminability | Distinguishes player skill levels via item response theory |
| Difficulty | Questions where humans outperform AI |
| Diversity | Varied entities, locations, and named entities |
| Quality / Quantity | Number of questions passing editorial review |
Prizes:
- 1st place: $500
- 2nd place: $250
- 3rd place: $150
Track 2: Human Team Competition
Best-performing human trivia team: $200
Track 3: Computer System Submission
Submit a QA system to be evaluated for accuracy on adversarial questions, calibration, and in-person competition performance.
Submission: Dynabench — RQA task or email
Prizes:
- 1st place: $500
- 2nd place: $250
- 3rd place: $150
Rules
- Questions must be original (“clean”) and not previously published
- Answers must be matchable to Wikipedia page titles
- No communication between question writers and computer competition teams
- Register via separate Google Forms for each track