Foundation and Large Language Models
Beyond improving the dataset, Cleanlab Studio allows you to train and deploy foundation models on messy real-world data with a few clicks. The AI that Cleanlab uses to detect issues in your dataset is powered by such models which are automatically fit your data.
Practice data curation like the best generative AI teams
Cleanlab software helps you effectively curate data without large teams of experts. Easily build your own data engine like those that power leading AI teams!
— OpenAI blog on DALLE-2, describing how they produce one of the best available generative image models.
Since training data shapes the capabilities of any learned model, data filtering is a powerful tool for limiting undesirable model capabilities.
We prioritized filtering out all of the bad data over leaving in all of the good data. This is because we can always fine-tune our model with more data later to teach it new things, but it’s much harder to make the model forget something that it has already learned.
— Aidan Gomez (Founder & CEO of Cohere) speaking on the data sensitivity of LLM training on Weights and Biases podcast
“If you teach the model something wrong, it will lock that in and forever believe that wrong thing.”
I was not prepared for how sensitive some of these models are.
— Emad Mostaque, founder and CEO of Stability.ai on Infinite Loops podcast
Data is the new oil but it's got to be clean data.
The valuable data here is the content that is structured in the world that allows you to learn principles as opposed to again, the big data age where it was about as much data as possible to extract patterns
— Nat Friedman, former CEO of GitHub, investor in hundreds of AI startups
I do believe all the great labs are actually pouring huge amounts of energy into cleaning their data
— Andrej Karpathy, former Director of AI at Tesla, co-founder of OpenAI at Spark+AI Summit
At Tesla, I spend most of my time just massaging the datasets, and this takes a huge amount of work/effort and you want to do it extremely well.
Read more about this topic: Data Engine Design by George Pearse, ML Engineer at Binit.AI
Data Annotation & Crowdsourcing
Data Entry, Management, and Curation
CLEANLAB IS BUILT FROM THE GROUND UP TO SUPERCHARGE LLMS
- Cleanlab TLM (Trustworthy Language Model) that quantifies answer uncertainty
- Improve LLM fine-tuning accuracy by 30% using Cleanlab (optionally in Databricks)
- Automatically improve the quality of any:
LLM instruction tuning (ie. alignment, fine-tuning) dataset, RLHF dataset, or labeled dataset
- Improve prompt engineering via more reliable model evaluation and few-shot prompt selection
- Document curation (eg. for Retrieval-Augmented Generation or legal applications)
- Assess synthetic image/text data produced via Generative AI
- Deploy ML models more accurate than fine-tuned OpenAI LLMs
for classifying text like product reviews or legal judgements
When fine-tuning OpenAI GPT models in a text classification task (politeness prediction), correcting label errors with Cleanlab Studio improved test accuracy by 37% without any change to the modeling/fine-tuning code (solely the dataset was modified). Read more.
Effortlessly detect errors in reinforcement from human feedback data (RLHF). Here is an example of a human error in the Anthropic RLHF dataset found with Cleanlab Studio, where the human-rejected LLM output (completion) is unequivocally better than the human-chosen LLM output (completion). The human who provided feedback just accidentally made a mistake! Read more.
Automatically flag low-quality examples for any image dataset. Cleanlab software can report which images are: blurry, under/over-exposed, oddly sized, low information, or (near) duplicates of others. Handling such issues is important in generative AI and computer vision (especially to diagnose spurious correlations). Read more.
Accelerate data labeling for Transformer Models. ActiveLab greatly reduces time and labeling costs to achieve a given model performance compared to standard data annotation. For example, ActiveLab hits 90% model accuracy at only 35% of the label spend as standard training. Read more.
For any text dataset (whether real or LLM-generated)
- Automatically detect data issues via Python API or no-code Web App such as: label errors, near duplicates, outliers, ambiguous examples, data drift, low-quality text (informal language, not natural language), language that is foreign, toxic, or Personally Identifiable Information.
- Auto-fix these detected issues rapidly in large datasets.
For text generation with LLMs
- Assess LLM outputs, both individually and as an overall synthetic dataset
- Automatically find bad (request, response) pairs in fine-tuning (ie. instruction tuning, alignment) datasets
- Get confidence scores for responses from any LLM (and improve its responses)
For text classification, tagging, entity recognition tasks
Improve LLMs via data curation during:
- Fine-tuning - improves OpenAI models by 37% in case study
- Prompt selection or few-shot / in-context learning
- AI-automated data labeling confidently labels most of your dataset with a few clicks
- Active learning to decide what data is most informative to label or re-label next
- Infer consensus + annotator-quality for multi-annotator data (more accurately than Dawid-Skene/GLAD and other statistical crowdsourcing algorithms)
- Used by BBVA for financial transaction categorization in online banking (reduced data labeling costs 98%, improved model by 28%)