Leveraging innovative AI to improve test accuracy
Cleanlab featured in CB Insights GenAI 50, ranking as one of the world’s 50 most innovative Generative AI companies (along with OpenAI, Hugging Face, Cohere, Anthropic, and more).
When fine-tuning OpenAI GPT models in a text classification task (politeness prediction), correcting label errors with Cleanlab Studio improved test accuracy by 37% without any change to the modeling/fine-tuning code (solely the dataset was modified). Read more.
Automatically detect user error
Effortlessly detect errors in reinforcement from human feedback data (RLHF). Here is an example of a human error in the Anthropic RLHF dataset found with Cleanlab Studio, where the human-rejected LLM output (completion) is unequivocally better than the human-chosen LLM output (completion). The human who provided feedback just accidentally made a mistake! Read more.
Flag image issues
Automatically flag low-quality examples for any image dataset. Cleanlab software can report which images are: blurry, under/over-exposed, oddly sized, low information, or (near) duplicates of others. Handling such issues is important in generative AI and computer vision (especially to diagnose spurious correlations). Read more.
Accelerate data labeling for Transformer Models
ActiveLab greatly reduces time and labeling costs to achieve a given model performance compared to standard data annotation. For example, ActiveLab hits 90% model accuracy at only 35% of the label spend as standard training. Read more.
HOW IT WORKS
CLEANLAB CAPABILITIES
Automated data curation for Large Language Models
Careful data curation is crucial for LLMs to go from demo → production, and Cleanlab software makes this systematic/automated. Cleanlab software has been used in LLM applications at Fortune 100 enterprises, tech consulting, and startups that span: LLM fine-tuning on customer service chats, learning tool/API use, online banking, auto compliance-determination, customer simulation, etc. This section lists technical capabilities of Cleanlab software, broken down by LLM use-case.
Better deployed ML models
-
Automatically detect data issues via Python API or no-code Web App such as: label errors, near duplicates, outliers, ambiguous examples, data drift, low-quality text (informal language, not natural language), language that is foreign, toxic, or Personally Identifiable Information.
For text classification, tagging, entity recognition tasks
- Improve LLMs via data curation during:
- Label data efficiently and reliably (produce high-quality data at low cost)
- AI-automated data labeling confidently labels most of your dataset with a few clicks
- Active learning to decide what data is most informative to label or re-label next
- Infer consensus + annotator-quality for multi-annotator data (more accurately than Dawid-Skene/GLAD and other statistical crowdsourcing algorithms)
- Used by BBVA for financial transaction categorization in online banking (reduced data labeling costs 98%, improved model by 28%)
- Auto-train and model deployment
TESTIMONIALS
Practice data curation like the best Generative AI teams
What’s the common thread across leading organizations with the best AI models? Relentless focus on data curation rather than inventing novel models or training algorithms. Cleanlab software helps you effectively curate data without large teams of experts. Easily build your own data engine like those that power leading AI teams!
“I've found that the app corrects mislabelling very well, but I didn't get the results I was looking for when I used your open source library Cleanlab directly in Python. It turns out that your app on the web version works much better than the library in Python!”
Yiwen Jiang, Data Engineer at Orange
“Since training data shapes the capabilities of any learned model, data filtering is a powerful tool for limiting undesirable model capabilities. We prioritized filtering out all of the bad data over leaving in all of the good data. This is because we can always fine-tune our model with more data later to teach it new things, but it’s much harder to make the model forget something that it has already learned.”
OpenAI blog on DALLE-2, describing how they produce one of the best available generative image models.
“I do believe all the great labs are actually pouring huge amounts of energy into cleaning their data.”
Nat Friedman, former CEO of GitHub, investor in hundreds of AI startups
“At Tesla, I spend most of my time just massaging the datasets, and this takes a huge amount of work/effort, and you want to do it extremely well.”
Andrej Karpathy, former Director of AI at Tesla and co-founder of OpenAI at Spark+AI Summit, hosted by Databricks.
“I've found that the app corrects mislabelling very well, but I didn't get the results I was looking for when I used your open source library Cleanlab directly in Python. It turns out that your app on the web version works much better than the library in Python!”
Yiwen Jiang, Data Engineer at Orange
“Since training data shapes the capabilities of any learned model, data filtering is a powerful tool for limiting undesirable model capabilities. We prioritized filtering out all of the bad data over leaving in all of the good data. This is because we can always fine-tune our model with more data later to teach it new things, but it’s much harder to make the model forget something that it has already learned.”
OpenAI blog on DALLE-2, describing how they produce one of the best available generative image models.
