one year on
Mysterious 'gpt2-chatbot' appears on LMSYS Arena, matches GPT-4 Turbo quality
An unlabeled and undocumented model has appeared on the LMSYS Chatbot Arena, producing responses comparable to GPT-4 Turbo, sparking intense speculation about its origin.
A new, unlabeled model called ‘gpt2-chatbot’ appeared today on the LMSYS Chatbot Arena, and early testers report it performs at or above the level of GPT-4 Turbo. The model is undocumented and offers no information about its architecture, context length, or capabilities beyond a system prompt that reads: ‘You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture. Knowledge cutoff: 2023-11. Current date: 2024-04-29. Image input capabilities: Enabled. Personality: v2.’ This has led to widespread speculation that the model is a stealth preview from OpenAI.
Independent developer Simon Willison noted that the model ‘hallucinates less’ and provides ‘more specific details’ than GPT-4 Turbo in his tests. He suggests the model may be a preview of OpenAI GPT-4.5.
Until OpenAI or LMSYS clarifies the model’s origin, the community continues a distributed ‘vibe check’—a mix of amusement and frustration that even basic questions remain unanswered.
The record
Suggests the model may be a preview of OpenAI GPT-4.5, noting it hallucinates less and returns more specific details in his tests.
One year later — open only if you can handle spoilers
The model was later confirmed to be a test version of GPT-4o, which OpenAI released in May 2024. Altman's tweet was an in-joke about the internal codename. The model was pulled from LMSYS on May 1.