In April 2024, Kaggle hosted the LEAP-Atmospheric Physics using AI (ClimSim) competition. The competition aimed to use AI to improve climate modeling, challenging participants to develop machine learning models that could enhance climate projections and reduce uncertainty in future climate trends. The goal was to employ faster ML models to deepen our understanding of climate risks, such as extreme weather events and long-term climate changes. This, in turn, would provide policymakers with better information to take action on climate change, whether it involves preparing for future hazards or working to reduce its impacts.

Current Challenges in Climate Science

Climate models are essential tools for understanding how Earth’s climate works. Since the climate system is very complex, these models use estimates (called parameterizations) to represent small-scale physical processes, like cloud formation or turbulence, that are too tiny to model directly due to the grid size of the model. These estimates aren’t perfect, leading to uncertainty about how much warming we can expect, how precipitation patterns will change, or how extreme weather events may shift. One approach, the Multi-scale Modeling Framework (MMF), aims to include more detailed processes for improved accuracy. However, this method demands so much computing power that it’s impractical for regular climate predictions. Finding a sweet spot between detailed representation and computational cost is a key focus in climate modeling. This balance involves using a grid size that captures enough important physical processes without overwhelming computing resources. This is where machine learning comes in, and this was the main idea behind the competition.

Competition

Participants in the competition were asked to create machine learning models that could imitate small-scale weather processes like storms, clouds, turbulence, rainfall, and radiation within E3SM-MMF, a multi-scale climate model supported by the U.S. Department of Energy. Due to the model's grid size, these processes are too small to be directly modeled in a traditional climate model.

By developing ML models that can emulate (or approximate) these small-scale processes, scientists can significantly speed up the simulations. ML models are much faster to run than traditional models like MMF. If accurate enough, they can help make high-resolution, reliable climate predictions more accessible to a broader range of scientists and policymakers. 

This competition was part of the 2024 ICML Machine Learning for Earth System Modeling (ML4ESM) Workshop. It was based on the paper and dataset, which earned the Outstanding Datasets and Benchmarks Paper award at NeurIPS 2023.

Solutions

Three of the H2O.ai Kaggle Grandmasters - Nischay Dhankar, Nikhil Mishra and Ryan Chesler, participated in the competition. They competed in two different teams and finished in 9th and 10th place, respectively. I reviewed their solutions and spoke with them to summarize their approaches in this article. Both teams have made their solutions public: the 9th-place and 10th-place solutions. Here is a quick review of their solutions.

Nishchay and Nikhil’s team, which secured 9th place, focussed on balancing model complexity with interpretability. They started with a simple base transformer model and made small, incremental changes to the architecture, improving performance while maintaining a clear understanding of what worked. By the end of the competition, they had a high-performing yet efficient system. A key factor in their success was improving input data quality through better normalization, preprocessing, and feature engineering. They focused on maintaining the structure of time-based inputs (Timestep, Number of Features) to ensure consistency in the pipeline.

They introduced architectural improvements to handle over 80 million training samples efficiently, including using Fourier Neural Operator (FNO) blocks, which improved performance and reduced model parameters. They also used the transformer-engine library, speeding up training by 30%, and refined positional encodings by applying multiple encodings in parallel and reducing Multihead attention blocks while maintaining performance. Despite these optimizations, training still took over 24 hours on 4x 4090 GPUs, but this was a significant improvement over earlier timelines.

Ryan’s team, Updraft, which finished in 10th place, employed an ensemble approach by training weights for each model based on specific target predictions, leveraging insights from their exploratory data analysis (EDA). They identified that different models exhibit unique strengths; for example, some are adept at regressing mixing ratios, while others excel in predicting wind velocity. This specialization allowed the team to focus on tailoring their model combinations effectively. They tested various configurations during their experimentation and discovered that an ensemble model without normalization but with bias produced the best results.

They experimented with various architectures, such as Conv-Transformer and U-Net. The team faced challenges with resource limitations but improved their models by focusing on climate-invariant features like Relative Humidity and Heat Flux, which help account for yearly temperature changes. Their final ensemble balanced multiple models and achieved significant accuracy improvements by integrating confidence-aware MSE loss and hard example mining.

Conclusion

Competitions like this are essential for advancing climate science because they promote innovation in machine learning methods and improve our understanding of climate risks, such as extreme weather events and long-term changes. The knowledge gained can give policymakers vital information to address climate change effectively, helping them prepare for future hazards and implement strategies to reduce impacts.