one year on
Stanford releases Alpaca, an instruction-following AI model built for under $600
The 7B-parameter model, fine-tuned from Meta's LLaMA on 52K GPT-generated examples, behaves similarly to OpenAI's text-davinci-003 and signals that instruction tuning is now feasible for academic labs on a shoestring budget.
Researchers at Stanford’s Center for Research on Foundation Models (CRFM) today released Alpaca, a 7-billion-parameter language model fine-tuned from Meta’s LLaMA that performs instruction following at a level comparable to OpenAI’s text-davinci-003 — all for a total training cost under $600.
Alpaca was trained on 52,000 instruction-output pairs generated by prompting OpenAI’s text-davinci-003 with the self-instruct methodology. Data generation cost less than $500 via the OpenAI API, and the fine-tuning run on eight 80GB A100 GPUs took three hours, adding less than $100 in cloud compute. In a blind evaluation by the five student authors, Alpaca matched text-davinci-003, winning 90 comparisons against 89. The team acknowledges the evaluation is small in scale and diversity, and they caution that Alpaca exhibits common failure modes including hallucination and toxicity.
The release includes training code, the 52K instruction dataset, and an interactive demo. The model weights are not yet released; the team says they are consulting Meta on licensing. Alpaca carries a non-commercial restriction inherited from LLaMA and is intended solely for academic research. The paper notes that the approach could be replicated by others, potentially lowering the barrier for producing capable instruction-following models.
The team says the release is meant to enable controlled scientific studies of instruction-following models in academia, while also noting risks including harmful content, spam, fraud, and disinformation.
The record
Co-lead author expressed surprise at Alpaca's performance given its small size and modest data, noting the blind pairwise comparison showed Alpaca winning 90 vs 89 comparisons against text-davinci-003.
As senior author, highlighted the goal of enabling academic research on instruction-following models, which had been difficult due to lack of accessible high-capability models.
One year later — open only if you can handle spoilers
Alpaca kicked off a wave of derivative open-source instruction-tuned models, including Vicuna, Koala, and others, collectively known as the 'barnyard.' Within a year, LLaMA-based models became a standard baseline for academic instruction-following research, and the cost of fine-tuning dropped further with techniques like LoRA.