:quality(75))
:quality(75))
:quality(75))
- H2O LLM DataStudio's new Custom Eval component automatically generates evaluation datasets with Question Type, Multichoice, or Token Presence formats from documents or existing datasets.
- Custom Eval accepts existing question and answer CSV datasets or documents such as PDFs, DOC files, audio, video, or website URLs as input for generating evaluation datasets.
- Custom Eval's Question Type generator produces conditional questions, compressed questions, and multihop reasoning questions with customizable distributions across each type.
- Generated evaluation datasets from Custom Eval can be downloaded as CSV or JSON and imported directly into H2O Eval Studio's Tests tab to launch a new leaderboard.
- H2O AI is building a GitHub repository of Custom Eval generated evaluation datasets covering industries including banking, health, legal, government, finance, and telecom.
Introducing new 'Custom Eval' - Create your own evaluation datasets effortlessly from various document formats, or existing datasets.
H2O LLM DataStudio’s new Custom Eval component is a no-code capability that empowers users to generate evaluation datasets with different evaluation types (Question Type, Multichoice, Token Presence) from any unstructured data or existing datasets within their organization. Leveraging the H2OGPTe Large Language Model, Custom Eval automatically generates diverse evaluation datasets suitable for evaluating LLMs across various document types, including annual reports, presentations, audio or video files.
Key Functionalities of LLM DataStudio Custom Eval
1. Variety of data input
Users can utilize existing Question and Answer datasets (in CSV format) or any document in formats like PDF, DOC, audio, video files or website URLs linking to documents to create evaluation datasets.
2. Different types of evaluation dataset types
- Question Type - Generates three different types of questions with customizable distributions.
- Conditional Question: Increases question complexity by incorporating scenarios or conditions affecting the context.
- Compress Question: Makes the question more indirect and shorter while retaining its essence.
- Multihop Reasoning Question: Increases question complexity by requiring multiple logical connections or inferences.
- Multichoice - Provides multiple choices for questions, with one correct and three incorrect choices.
- Token Presence - Extract the minimum tokens necessary for accurate answers for the question.
3. View and customize the output
- Flag: Flag whether a row is relevant or irrelevant.
- Edit: Customize and update any row in the generated evaluation dataset.
4. Download evaluation dataset
The generated evaluation dataset can be downloaded in CSV or JSON format. These evaluation datasets can be easily integrated into Eval Studio using JSON downloads.
5. Tight Integration between Curate and Custom Eval component.
The QA pairs generated using a project in the Curate component can be used to create a new project in Custom Eval and it can be used to generate the evaluation dataset of the desired evaluation type.
Integrate the downloaded evaluation dataset with Eval Studio
LLM DataStudio has been designed to ensure an ease of integration into H2O EvalStudio. Once you’ve finished reviewing and finalizing your eval dataset, you can take the json output and effortlessly import it into EvalStudio's "Tests" tab. From there, you can explore your test cases and documents, and launch a new leaderboard by filling in essential details like name, description, model selection and evaluator type (e.g., Answer Correctness or Token Presence).
What’s Next?
In addition to the LLM DataStudio Custom Eval, a Github Repository with the collection of evaluation datasets of different evaluation types for a range of different documents from different industries including Banking, Risk Management, Telecom, Government, Legal, Health, Finance, and Security is created. (Stay tuned as we will soon be adding 50 evaluation datasets to the repository). Examples of the different types of documents used for some of these different industries are as follows,
- Banking - Financial Statements, Annual Reports, Broker Agreements.
- Health - Healthcare Standards, Healthcare Regulations
- Legal - EU AI act, IRS Filing policy document, Contracts, Agreements
- Government - Home Affairs document, Constitution, Veterans Affairs document
- Finance - Financial records, Product Disclosure Statements
- Telecom - Technical reports, Regulation documents.