17 Data Prep Functions for LLM Training Data
- Generative AI
- H2O Cascade
This lesson walks through the individual data preparation functions used to get raw text ready for large language model tasks, from cleaning and augmentation through profanity, quality, length, and sensitive-information checks, to deduplication and toxicity filtering. Each function is explained with a plain example, such as trimming tweets to a character limit or stripping credit card numbers before fine-tuning. The lesson closes by addressing unstructured PDFs, explaining that they first need to be converted into question-and-answer pairs before any cleaning function is useful.
Key takeaways
- How the profanity, toxicity, and sensitive information checks each filter a dataset for a different risk
- What the Valid Question function does to confirm a question column actually contains a question
- Why data deduplication and the length checker both help avoid wasting a model's training budget on redundant text
- How an unstructured PDF gets converted into question-and-answer pairs before it can be used for fine-tuning
- The role RLHF protection plays in appending human feedback data for reinforcement learning steps
Start owning your AI