LLM - Detect AI Generated Text

Name: LLM - Detect AI Generated Text
Uploaded: 2024-02-20
Duration: 42 min
Description: A Kaggle Grandmaster walks through his second-place solution to LLM Detect AI Generated Text, telling student essays from LLM-written ones.

Generative AI

Overview

In this session, a Kaggle Grandmaster walks through his second-place solution to the LLM Detect AI Generated Text competition, a binary classification task to identify whether an essay was written by a student or a large language model. He covers the two biggest modeling challenges, generating positive training examples when only human essays were provided, and avoiding overfitting on a small, topic-correlated leaderboard. The talk details a two-stage approach: pretraining a DeBERTa model on LLM-completed text from a public corpus, then fine-tuning it on essays generated by LLMs that were themselves fine-tuned, using H2O LLM Studio, to mimic student writing.

Key takeaways

How the competition's binary classification setup (human vs. LLM-generated essay, scored on AUC) created a data problem: only human-written essays were provided, so positive (AI-generated) examples had to be created from scratch.
How a pretraining step used the SlimPajama dataset, truncating about half a million documents at a random point and asking 15 LLMs (mostly 7-billion-parameter models) to complete the text, creating a large, diverse set of synthetic positive examples paired with the original human continuations as negatives.
Why this pretraining step alone, with no in-domain competition data, was strong enough to rank around 8th on the private leaderboard.
How H2O LLM Studio was used to fine-tune 5 additional LLMs directly on the ~9,000 student essays so their generations would mimic student citations, quoting, and even spelling errors, producing around 18,000 harder-to-distinguish positive examples for the second training stage.

Start owning your AI

One platform. Complete sovereignty. Measurable results.

Request a demo Talk to H2O