We are living in unprecedented times. Our society and economy are experiencing shocks beyond anything we have seen in living history. Beyond the human cost, there is a data science and machine learning elephant in the room (hopefully 2 meters away): Are your predictive models still doing the job you expect them to do?
The challenge here is greatly complicated by:
What do you do? One option is do nothing, accept higher error rates in your predictive modeling and AI Systems. Curl up in a dark room and hope things return to “normal”. But this is a very risky approach. Just take a look at the media, and many are predicting a ‘new normal ’.
It might be much better to be proactive and work out what we can do and what we can’t. Sure the devastating impact of COVID-19 is unprecedented, but we see similar changes on a smaller scale all the time. Like when Virgin Australia entered (rather than exited) the domestic flying market or more recently with the advent of the neo-banks and fintechs. What can we learn from these and other shocks to the system?
I think we need to look beyond performance on the test set, search ways to rapidly develop AI models, and apply model monitoring techniques at scale. You can check some of H2O Driverless AI’s AutoML capabilities here.
I am interested in hearing others’ thoughts. Are you rebuilding?