Research

We publish fundamental machine learning research on methods to help people improve the quality of their datasets and models for messy, real-world applications.

Featured publications by our team

• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

  Curtis Northcutt, Anish Athalye, and Jonas Mueller

  35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks

  Demo, Code, Blog Post, Press [1, 2, 3, 4, 5, 6, 7, 8]

• Confident Learning: Estimating Uncertainty in Dataset Labels

  Curtis Northcutt, Lu Jiang, and Isaac Chuang

  Journal of Artificial Intelligence Research (JAIR), Vol. 70 (2021)

  Code, Blog Post

• Back to the Basics: Revisiting Out-of-Distribution Detection Baselines

  Johnson Kuan and Jonas Mueller

  ICML Workshop on Principles of Distribution Shift, 2022

  Code (to run method), Code (to reproduce results), Blog Post

• Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels

  Curtis Northcutt, Tailin Wu, and Isaac Chuang

  33rd Conference on Uncertainty in Artificial Intelligence (UAI 2017)

• ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

  Hui Wen Goh and Jonas Mueller

  ICLR Workshop on Trustworthy ML, 2023

  Code (to run method), Code (to reproduce results), Blog Post, Press

• CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators

  Hui Wen Goh, Ulyana Tkachenko, and Jonas Mueller

  NeurIPS 2022 Human in the Loop Learning Workshop

  Code (to run method), Code (to reproduce results), Blog Post

• Model-Agnostic Label Quality Scoring to Detect Real-World Label Errors

  Johnson Kuan and Jonas Mueller

  ICML DataPerf Workshop, 2022

  Code

• DataPerf: Benchmarks for Data-Centric AI Development

  Mazumder et al.

  ICML DataPerf Workshop, 2022

  Benchmarks, Blog Post (Google), Blog Post (MLCommons)

• Detecting Label Errors in Token Classification Data

  Wei-Chen (Eric) Wang and Jonas Mueller

  NeurIPS 2022 Workshop on Interactive Learning for Natural Language Processing (InterNLP)

  Code (to run method), Code (to reproduce results), Blog Post

Additional publications by our team

• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

  Anish Athalye, Nicholas Carlini, and David Wagner

  35th International Conference on Machine Learning (ICML 2018)

  Best Paper Award, Code

• EgoCom: A Multi-person Multi-modal Egocentric Communications Dataset

  Curtis Northcutt, Zha Shengxin, Steven Lovegrove, and Richard Newcombe

  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

  Code

• Adaptive Interest for Emphatic Reinforcement Learning

  Martin Klissarov, Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Taesup Kim, and Alex Smola

  Advances in Neural Information Processing Systems (NeurIPS 2022)

• Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features

  Jiuhai Chen, Jonas Mueller, Vassilis Ioannidis, Soji Adeshina, Yangkun Wang, Tom Goldstein, and David Wipf

  International Conference on Learning Representations (ICLR 2022)

• Deep learning for the partially linear Cox model

  Qixian Zhong, Jonas Mueller, and Jane-Ling Wang

  Annals of Statistics, 2022

• Benchmarking Multimodal AutoML for Tabular Data with Text Fields

  Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola

  35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks

  Code

• Overinterpretation reveals image classification model pathologies

  Brandon Carter, Siddhartha Jain, Jonas Mueller, and David Gifford

  Advances in Neural Information Processing Systems (NeurIPS 2021)

  Code

• Deep Extended Hazard Models for Survival Analysis

  Qixian Zhong, Jonas Mueller, and Jane-Ling Wang

  Advances in Neural Information Processing Systems (NeurIPS 2021)

• Continuous Doubly Constrained Batch Reinforcement Learning

  Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Pratik Chaudhari, and Alex Smola

  Advances in Neural Information Processing Systems (NeurIPS 2021)

  Code

• Verifying Hardware Security Modules with Information-Preserving Refinement

  Anish Athalye, M. Frans Kaashoek, and Nickolai Zeldovich

  16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2022)

  Code

• Notary: A Device for Secure Transaction Approval

  Anish Athalye, Adam Belay, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich

  27th ACM Symposium on Operating Systems Principles (SOSP 2019)

  Code

• Synthesizing Robust Adversarial Examples

  Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok

  35th International Conference on Machine Learning (ICML 2018)

  Blog Post

• Black-box Adversarial Attacks with Limited Queries and Information

  Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin

  35th International Conference on Machine Learning (ICML 2018)

  Blog Post, Code

• Identifying Incorrect Annotations in Multi-Label Classification Data

  Aditya Thyagarajan, Elías Snorrason, Curtis Northcutt, and Jonas Mueller

  ICLR Workshop on Trustworthy ML, 2023

  Code (to run method), Code (to reproduce results), Blog Post