Agenda

Registration

Keynote

Sri Ambati, CEO & Founder, H2O.ai

EvalStudio Benchmarking

Srinivas Neppalli, Sr. AI Engineer, H2O.ai
Jon McKinney, Director of Research, H2O.ai

Eval Studio represents a significant advancement in the evaluation of large language models (LLMs), addressing the critical need for robust and customizable benchmarking tools in this field. Designed to cater to a variety of tasks such as question answering, conversations, and Retrieval-Augmented Generation (RAG) analyses, the platform allows users to design and implement their own benchmarks and evaluations. Unique in its approach, Eval Studio offers both manual and automated evaluation options, the latter utilizing advanced LLMs like GPT-4 or Llama2-70B. This dual-mode functionality facilitates comprehensive analysis, catering to the diverse requirements of our sophisticated audience.

The toolkit’s capability extends beyond mere evaluation. It systematically collects and stores results along with relevant metadata, enabling effective tracking and debugging of model performance over time. This feature is particularly valuable for continuous improvement and understanding of evolving LLM capabilities.

Large Language Model Interpretability

Kim Montgomery, KGM + Sr. AI Engineer, H2O.ai

Recent advances in large language models have also brought new challenges in terms of model interpretability. Generative models produce novel content, which generally doesn’t correspond to a single correct answer, so simply determining the accuracy of a result is more complicated than it is for supervised machine learning.

Also, text output can be quantified in terms of tone, toxicity, accuracy, fairness, privacy preservation, and other metrics that may or may not have an analog in supervised learning. Kim will compare model interpretability for supervised learning to methods available for understanding large language models.

Introduction to Enterprise h2oGPTe, LLM Studio and GenAI App Store

Hands-On Advanced LLM Workshop, Training, and Certification

Lunch Break

Fireside Chat

Sri Ambati, CEO & Founder, H2O.ai
Andy Markus, Chief Data Officer, AT&T

Academic Panel on GenAI Opportunities and Challenges

Srijan Kumar, Assistant Professor
School of Computational Science and Engineering, College of Computing, Georgia Institute of Technology

Tianming Liu, Research Professor of Computer Science
School of Computing, University of Georgia

Vijay Nair, Donald A. Darling Professor Emeritus of Statistics
University of Michigan

Industry Panel on GenAI Governance and Model Validation

Moderator: Elizabeth Mays, Chief Model Risk Officer, PNC
Bradley Currell, Financial Model Risk Executive, Ally
Jacob Kosoff, Data Science & Model Development Executive, Bank of America
Tarun Joshi, Quantitative Analytics Manager, Wells Fargo

Fireside Chat with US Regulators

Moderator: Dr. Agus Sudjianto, Wells Fargo
David Palmer, Board of Governors of the Federal Reserve
Loren Bushkar, Board of Governors of the Federal Reserve
Neil Desai, Business Finance Senior Analyst, Federal Reserve

What Responsible AI Means for Financial Services and Beyond

Dr. Doug Hague, Executive Director, School of Data Science at University of North Carolina at Charlotte

Closing

Networking Happy Hour