LLM - Detect AI Generated Text
- Generative AI
Overview
In this session, a Kaggle Grandmaster walks through his second-place solution to the LLM Detect AI Generated Text competition, a binary classification task to identify whether an essay was written by a student or a large language model. He covers the two biggest modeling challenges, generating positive training examples when only human essays were provided, and avoiding overfitting on a small, topic-correlated leaderboard. The talk details a two-stage approach: pretraining a DeBERTa model on LLM-completed text from a public corpus, then fine-tuning it on essays generated by LLMs that were themselves fine-tuned, using H2O LLM Studio, to mimic student writing.
Key takeaways
- How the competition's binary classification setup (human vs. LLM-generated essay, scored on AUC) created a data problem: only human-written essays were provided, so positive (AI-generated) examples had to be created from scratch.
- How a pretraining step used the SlimPajama dataset, truncating about half a million documents at a random point and asking 15 LLMs (mostly 7-billion-parameter models) to complete the text, creating a large, diverse set of synthetic positive examples paired with the original human continuations as negatives.
- Why this pretraining step alone, with no in-domain competition data, was strong enough to rank around 8th on the private leaderboard.
- How H2O LLM Studio was used to fine-tune 5 additional LLMs directly on the ~9,000 student essays so their generations would mimic student citations, quoting, and even spelling errors, producing around 18,000 harder-to-distinguish positive examples for the second training stage.
- How six DeBERTa models (V3 and large), trained on this generated data plus data shared by other competitors, were ensembled into a final classifier that scored 0.9834 AUC for 2nd place.
- How isolating each stage after the competition showed the fine-tuning step alone would have ranked 5th and the pretraining step alone 8th, with the combination reaching 2nd.
Start owning your AI