LLM Data Prep: Clean, Filter, and Structure Text
- Generative AI
- H2O Cascade
This lesson walks through preparing a raw text dataset, a collection of LinkedIn influencer posts, for downstream large language model tasks. It shows the Python steps first: filtering out profanity, scoring reading level with Flesch-Kincaid, trimming by length, and keeping only the top-performing posts, which together remove more than a third of the dataset. It then uses a generative model to write titles for the cleaned posts in a zero-shot setup, and shows how the same curate, prepare, and augment steps can be run through H2O's visual data preparation tool instead of custom code.
Key takeaways
- How profanity filtering and Flesch-Kincaid reading level scores are used to remove low-quality text
- Why cleaning a raw text dataset for LLM tasks can remove more than a third of the original rows
- How a generative model can write titles for cleaned posts in a zero-shot setup with no examples given
- What the curate, prepare, and augment steps do when preparing unstructured documents for fine-tuning
- How a visual pipeline can replace custom Python functions for cleaning, filtering, and formatting text data
Start owning your AI