Title: "Beyond the average case: machine learning for atypical examples"

Presented By: Tatsunori Hashimoto

Abstract: Although machine learning systems have improved dramatically over the last decade, it has been widely observed that even the best systems fail on atypical examples. For example, prediction models such as image classifiers have low accuracy on images from minority cultures, and generative models such as dialogue systems are often incapable of generating diverse, atypical responses. In this talk, I will discuss two domains where high performance on typical examples is insufficient.

The first is learning prediction models that perform well on minority groups, such as non-native English speakers using a speech recognition system. We demonstrate that models with low average loss can still assign high losses to minority groups, and this gap can amplify over time as minority users that suffer high losses stop using the model. We develop an approach using distributionally robust optimization that learns models that perform well over all groups and mitigate the feedback loop.

The second domain is learning natural language generation (NLG) systems, such as a dialogue system. It has been frequently observed that existing NLG systems which produce high-quality samples rely heavily on typical responses such as "I don't know" and fail to generate the full diversity of atypical but valid human responses.

We carefully quantify this problem through a new evaluation metric based on the optimal classification error between human- and model-generated text and propose a new, edit-based generative model of text whose outputs are both diverse and high-quality.

Biography: Tatsunori (Tatsu) Hashimoto is a 3rd year post-doc in the Statistics and Computer Science departments at Stanford, supervised by Professors Percy Liang and John Duchi. He holds a Ph.D from MIT where he studied random walks and computational biology under Professors Tommi Jaakkola and David Gifford, and a B.S. from Harvard in Statistics and Math. His work has been recognized in NeurIPS 2018 (Oral), ICML 2018 (Best paper runner-up), and NeurIPS 2014 Workshop on Networks (Best student paper).

For more information, contact Eyal deLara.