What Is Class Imbalance?
- Jul 22
- 36 min read
A fraud model can score 99% accuracy on real transaction data and still catch almost no fraud. That is not a contradiction — it is the accuracy paradox, and it is the single clearest symptom of class imbalance. If 9,900 of 10,000 transactions are legitimate and only 100 are fraudulent, a model that predicts "legitimate" every single time is right 99% of the time and useless 100% of the time. Understanding why that happens, and what to do instead, is the entire subject of this guide.
TL;DR
Class imbalance means one class (the majority class) has far more examples than another (the minority class) in a classification dataset; there is no fixed ratio at which a dataset officially becomes "imbalanced."
Accuracy can be dangerously misleading on imbalanced data because a model can score high while never correctly identifying the class you actually care about.
Metrics such as balanced accuracy, precision-recall curves, PR-AUC, macro F1, and the Matthews correlation coefficient (MCC) usually give a more honest picture than raw accuracy or even ROC-AUC.
The main fix families are data-level methods (oversampling, undersampling, SMOTE), algorithm-level methods (class weights, cost-sensitive learning, focal loss), and decision-level methods (threshold tuning and probability calibration).
Resampling must happen only inside the training data or training folds — never before the train/test split — or your validation numbers will be fiction.
No single method is universally best; the right combination depends on business costs, class overlap, sample size, and how the model will actually be used after deployment.
What Is Class Imbalance?
Class imbalance is a machine-learning classification problem in which one class has substantially more examples than another. It matters because standard accuracy hides poor minority-class performance. Solutions fall into three families: resampling the data (oversampling, undersampling, SMOTE), reweighting the learning objective (class weights, cost-sensitive loss), and adjusting the decision threshold or calibrating probabilities after training.
Table of Contents
A Clear Definition of Class Imbalance
In a machine learning classification task, class imbalance describes a dataset where the classes you are trying to predict do not appear in roughly equal numbers. The majority class is the one with the most examples. The minority class (sometimes called the rare class) has far fewer. Imbalance can happen in binary problems (fraud vs. not fraud), multiclass problems (normal, minor defect, critical defect), and multilabel problems (a product review tagged with several rare complaint categories at once).
There is no universal percentage at which a dataset officially "becomes" imbalanced. A 60:40 split is mild and rarely a real obstacle. A 95:5 split is meaningfully skewed. A 99.9:0.1 split, common in fraud and rare-disease detection, is severe. What actually matters is not the ratio in isolation, but whether that ratio, combined with the difficulty of the problem, causes a model trained on it to underperform on the class the business or research question cares about. A dataset can be 90:10 and still be easy to learn if the two classes are well separated. A dataset can be 55:45 and still be genuinely hard if the classes overlap heavily.
The imbalance ratio (IR) gives a simple way to describe skew:
IR = (number of majority-class examples) / (number of minority-class examples)
Example: a dataset with 9,000 majority-class rows and 300 minority-class rows has an imbalance ratio of 9,000 / 300 = 30. That means the majority class outnumbers the minority class 30 to 1.
An unequal class distribution and a genuinely difficult imbalanced-learning problem are not the same thing. The ratio tells you about the data. Whether that ratio harms your model depends on absolute sample counts, feature quality, class overlap, and — critically — the real-world cost of each type of error.
It is also worth stating plainly: the minority class is not automatically the positive class, and it is not automatically the class that matters most. In manufacturing quality control, the rare "defective" class is usually the one you care about detecting. But in some tasks the rare class is simply rare, not more important — for example, a rare customer segment that happens to be low-value. Always define which class matters, and why, before choosing metrics or fixes.
Analogy: think of a busy hospital emergency room. Most patients who walk in have minor, non-urgent issues. A tiny fraction have a life-threatening condition. If a triage system just assumes "probably not urgent" for everyone, it will be right most of the time — and it will miss the rare cases that matter most. The analogy illustrates the stakes, but it does not replace the technical definition above: what makes the problem hard is the interaction between rarity, overlap, and cost, not rarity alone.
A Simple Worked Example
Picture a fraud-detection dataset of 10,000 credit-card transactions. Historically, 100 of them (1%) are fraudulent, and 9,900 (99%) are legitimate. This is a common order of magnitude in real payments data.
Suppose a classifier takes the laziest possible strategy: predict "legitimate" for every transaction, regardless of the input. Here is the resulting confusion matrix:
Predicted Legitimate | Predicted Fraud | |
Actual Legitimate | 9,900 (TN) | 0 (FP) |
Actual Fraud | 100 (FN) | 0 (TP) |
Accuracy = (TP + TN) / Total = (0 + 9,900) / 10,000 = 99%
Minority-class recall (the fraction of actual fraud correctly caught) = TP / (TP + FN) = 0 / 100 = 0%
In plain English: the model is right 99 times out of 100, and it has never once identified a fraudulent transaction. For its actual purpose — stopping fraud — it is worthless. This is the accuracy paradox in miniature, and it is why accuracy alone should never be trusted as the headline metric on an imbalanced dataset.
Why Class Imbalance Occurs
Imbalance is not a data-collection mistake by itself; it is often a faithful reflection of how the world works. Common causes include:
Naturally rare events — most diseases, most equipment failures, most fraud attempts are rare by nature.
Expensive or difficult data collection — labeling requires expert time (radiologists, security analysts), so rare-event examples accumulate slowly.
Sampling and measurement processes — a survey or sensor network may systematically capture more of one class.
Filtering or eligibility rules — business rules that pre-screen cases before they reach the model.
Temporal changes — a category that was common last year may be rare this year, or vice versa.
Geographic or demographic selection effects — data collected in one region may under-represent conditions common elsewhere.
Data aggregation — combining multiple data sources can dilute a minority class further.
Missing or delayed labels — outcomes that take months to confirm (loan default, churn) leave many recent cases unlabeled, distorting apparent class balance.
Label noise — mislabeling can shrink an already-small minority class or inflate it artificially.
Long-tail distributions — in multiclass problems (product categories, defect types), most classes are individually rare.
Class definitions that are too broad or too narrow — merging categories can balance a dataset artificially; splitting them can create new rare classes.
Class imbalance is frequently confused with related but distinct problems, and these problems can coexist:
Small sample size — a dataset can be small overall without being imbalanced (50 examples split 25/25).
Covariate shift — the input feature distribution changes between training and deployment, independent of class ratio.
Concept drift — the relationship between features and the label changes over time.
Label noise — incorrect labels, regardless of class balance.
Class overlap — classes are hard to separate in feature space, regardless of how many examples exist.
Selection bias — the sample is not representative of the population, again independent of raw class counts.
Why Class Imbalance Can Harm a Model
Most learning algorithms minimize an average loss across all training examples. When one class dominates numerically, the loss-minimization process can achieve a low average loss largely by getting the majority class right, since those examples make up most of the total loss. This can push the model's decision boundary toward favoring the majority class, shrink the influence of minority examples on feature learning, and skew predicted probabilities toward the majority class even when the underlying signal for the minority class is present in the data.
Effects worth tracking explicitly:
Decision boundaries can drift toward the majority class, especially with algorithms that optimize an unweighted loss.
Probability estimates for the minority class can be systematically too low, even when ranking is reasonable.
Default decision thresholds (like 0.5) interact badly with skewed base rates — a threshold appropriate for one prevalence is often wrong for another.
Hyperparameter selection done with the wrong metric (accuracy) can select a model that looks good but ignores the minority class.
Model comparison across candidate models becomes unreliable if accuracy is the deciding metric.
Business decisions built on a model that silently ignores the minority class can be costly or dangerous depending on the domain.
This is the accuracy paradox: overall accuracy rises as imbalance increases, even while usefulness for the minority class falls, purely because the majority class dominates the accuracy calculation.
None of this means imbalance automatically breaks every model. Some imbalanced datasets remain highly learnable, particularly when the classes are well separated in feature space and the minority class, while rare, still has enough absolute examples for the model to learn its pattern.
The Factors That Determine Difficulty
The class ratio alone is an incomplete signal. Two datasets with an identical 50:1 imbalance ratio can have very different levels of real difficulty. What actually determines difficulty:
Absolute minority-class sample count — 500 minority examples behave very differently than 5.
Feature quality and separability — how cleanly classes can be distinguished in feature space.
Class overlap — regions where majority and minority examples look nearly identical.
Label noise — mislabeled examples are especially damaging when the minority class is already small.
Small disjuncts — minority sub-clusters that are themselves rare within the rare class.
Within-class diversity — a minority class with many different "flavors" is harder to learn than a homogeneous one.
Dimensionality — high-dimensional, sparse feature spaces (e.g., text) make minority patterns harder to isolate.
Model capacity and training objective — some models and loss functions are more sensitive to skew than others.
Error costs — how expensive false positives and false negatives are in the real deployment.
Probability calibration needs — whether the application requires trustworthy probabilities, not just correct rankings.
Deployment prevalence and dataset shift — whether the class ratio at deployment time matches the training data.
Scenario | Class Overlap | Minority Sample Count | Typical Difficulty |
Well-separated classes, thousands of minority examples | Low | High | Easy — often needs little beyond a sensible metric and threshold |
Well-separated classes, only dozens of minority examples | Low | Very low | Moderate — variance in evaluation becomes the main issue |
Heavily overlapping classes, thousands of minority examples | High | High | Hard — resampling and weighting help little; better features matter more |
Heavily overlapping classes, only dozens of minority examples | High | Very low | Very hard — reliable generalization may not be achievable without more or better data |
The lesson from this table: before reaching for a sampling technique, diagnose whether the real bottleneck is skew, overlap, sample size, or label quality — they call for different fixes.
How to Detect and Diagnose Class Imbalance
A practical diagnostic workflow, in order:
Inspect raw class counts for every class, not just the two you assume matter.
Calculate percentages and the imbalance ratio described earlier.
Examine absolute minority counts, not just the ratio — 1% of 10,000 is very different from 1% of 100 million.
Visualize class distributions with bar charts or histograms.
Check class distributions across train, validation, and test sets — a mismatch here can silently distort results.
Examine distributions over time and across important subgroups to catch drift or selection effects.
Review label quality, ideally by manually auditing a sample of both classes.
Inspect class overlap and difficult regions using dimensionality reduction or simple pairwise feature plots.
Establish a naive baseline (see the next section) before judging any model.
Evaluate per-class performance, not just an aggregate score.
The test set should reflect the intended deployment environment unless there is a clearly justified reason to do otherwise — the whole point of testing is to estimate how the model will behave once it is live, on the natural, unresampled class distribution it will actually encounter.
Use stratified splitting so that each split (train, validation, test) preserves the original class proportions. For grouped data — multiple rows belonging to the same customer, patient, or device — use group-aware splitting so that the same entity never appears in both train and test, which would otherwise leak information. For temporal data, use time-based splits (train on the past, test on the future) instead of random splits, since random shuffling can leak future information into training and can also hide prevalence changes over time. Ordinary random stratified splitting is inappropriate whenever rows are not independent (grouped) or whenever the deployment scenario is inherently forward-looking (temporal).
Evaluation Metrics for Imbalanced Classification
Every metric below starts from the confusion matrix:
True Positive (TP): model predicts positive, actual is positive.
False Positive (FP): model predicts positive, actual is negative.
True Negative (TN): model predicts negative, actual is negative.
False Negative (FN): model predicts negative, actual is positive.
Accuracy = (TP + TN) / (TP + TN + FP + FN). Interpretation: the overall fraction of correct predictions. Misleading under imbalance because it can be dominated by the majority class, as shown in the worked example above.
Precision = TP / (TP + FP). Interpretation: of everything the model flagged as positive, what fraction was actually positive. Matters when false positives are costly (e.g., unnecessary manual review).
Recall (sensitivity) = TP / (TP + FN). Interpretation: of everything that was actually positive, what fraction did the model catch. Matters when false negatives are costly (e.g., missed fraud, missed disease).
Specificity = TN / (TN + FP). Interpretation: of everything actually negative, what fraction was correctly identified as negative.
F1 score = 2 × (Precision × Recall) / (Precision + Recall). Interpretation: the harmonic mean of precision and recall, useful when you want one number that penalizes ignoring either error type, but it hides which of the two you're actually trading off.
F-beta score = (1 + β²) × (Precision × Recall) / (β² × Precision + Recall). Interpretation: a tunable version of F1; β > 1 weights recall more heavily, β < 1 weights precision more heavily — choose β based on real cost ratios.
Balanced accuracy = ½ × (Recall + Specificity), equivalent to the average of per-class recall. According to scikit-learn's documentation, balanced accuracy reduces to ordinary accuracy when the classifier performs equally well on both classes, and drops toward chance level when accuracy is only high because the classifier exploits an imbalanced test set, calculated as the arithmetic mean of sensitivity and specificity in the binary case. Interpretation: a fairer summary than raw accuracy when classes are skewed, though it can have high variance with very small minority classes and can obscure overall predictive power in some contexts.
Macro averaging computes a metric separately for each class and takes the unweighted mean, treating every class as equally important regardless of size. Micro averaging aggregates contributions across all classes before computing the metric, effectively weighting by class frequency. Weighted averaging takes the per-class mean weighted by class support (the number of true instances per class). In multiclass imbalanced problems, macro averages surface minority-class weaknesses that micro or weighted averages can hide.
ROC-AUC measures the area under the Receiver Operating Characteristic curve (true positive rate vs. false positive rate across thresholds). It can look optimistic under severe imbalance because the false positive rate denominator (all negatives) is huge, so even a large number of false positives moves the rate only slightly.
Precision-Recall curves and PR-AUC (also called average precision) plot precision against recall across thresholds. Because precision is directly sensitive to the number of false positives relative to true positives, and true positives are scarce in an imbalanced dataset, this curve tends to expose weaknesses that ROC curves can mask. Saito and Rehmsmeier's widely cited 2015 study demonstrated that the visual interpretability of ROC plots on imbalanced datasets can be deceptive about classifier reliability, due to an intuitive but incorrect interpretation of specificity, while precision-recall plots more accurately reflect future classification performance because they evaluate the fraction of true positives among positive predictions. Davis and Goadrich's related work formalized the mathematical relationship between the two curve families, showing that a curve dominates in ROC space if and only if it dominates in PR space — but the visual impression of "how good" a classifier looks can differ sharply between the two under imbalance.
Matthews correlation coefficient (MCC) is a single balanced measure computed from all four confusion-matrix cells; it ranges from -1 to +1 and is considered informative even when classes are of very different sizes, since it only produces a high score if the model does well on both classes simultaneously.
Geometric mean of sensitivity and specificity is another single-number summary sometimes used in imbalanced binary classification, penalizing any classifier that does very well on one class and very poorly on the other.
Log loss and Brier score evaluate the quality of predicted probabilities themselves, not just the final class label — essential whenever probabilities feed a downstream decision, cost model, or risk score.
Metric | What It Measures | Best Use | Main Limitation |
Accuracy | Overall correct-prediction rate | Roughly balanced classes | Misleading under imbalance |
Precision | Correctness of positive predictions | High cost of false positives | Ignores false negatives entirely |
Recall | Coverage of actual positives | High cost of false negatives | Ignores false positives entirely |
F1 / F-beta | Balance of precision and recall | Single-number summary with tunable trade-off | Can mask which error type dominates |
Balanced accuracy | Average of per-class recall | Comparing models fairly across skewed classes | High variance with tiny minority classes |
ROC-AUC | Ranking quality across all thresholds | Comparing overall discrimination | Can look optimistic under severe imbalance |
PR-AUC / Average Precision | Ranking quality focused on the positive class | Rare positive-class detection tasks | Less intuitive to explain to non-technical stakeholders |
MCC | Balanced single-number summary from full confusion matrix | Comparing models on both classes at once | Less familiar; requires explanation to stakeholders |
No single metric is always best. Metric selection should follow the real cost of false positives versus false negatives in your specific application, and a strong evaluation almost always reports several complementary metrics — a ranking metric (PR-AUC, ROC-AUC), a threshold-dependent metric (precision, recall, F1 at a chosen operating point), and, when probabilities matter, a calibration metric (Brier score, log loss). In multiclass problems, per-class metrics should always accompany the macro or weighted averages, since averages can bury a single badly performing rare class. Precision also changes when prevalence changes, which is why a precision figure measured on a resampled or historical dataset can be misleading once deployed at a different real-world prevalence.
Establishing a Reliable Baseline
Before trying anything sophisticated, establish what "doing nothing special" looks like:
Use scikit-learn's DummyClassifier (with strategies such as most_frequent or stratified) to see what a trivial rule achieves.
Compare against a simple, interpretable model (e.g., logistic regression) before moving to complex ensembles.
Record per-class metrics for every baseline, not just an aggregate score.
Evaluate the model without any resampling first — sometimes a well-specified model with an appropriate metric and threshold already performs acceptably.
Separate ranking quality (does the model order minority examples above majority examples?) from threshold performance (does the default cutoff produce good decisions?).
Use this baseline as the yardstick for every later intervention — an "improvement" that doesn't beat the honest baseline isn't one.
Data-Level Methods
Random Oversampling
Random oversampling duplicates existing minority-class examples until the classes reach a desired balance. It is simple to implement and requires no assumptions about feature types. The main risk is overfitting: because rows are literally copied, a model can memorize duplicated examples rather than learning generalizable patterns. It tends to work best when the minority class already has reasonable diversity and the model or downstream regularization can tolerate repeated rows.
Random Undersampling
Random undersampling removes examples from the majority class until the desired balance is reached. It reduces training time and can help some algorithms focus on the minority signal, but it risks discarding potentially useful majority-class information, which can hurt overall discrimination — especially when the majority class itself contains meaningful sub-patterns. It suits situations with abundant majority data, where discarding some of it barely affects what the model can still learn.
SMOTE
The Synthetic Minority Over-sampling Technique, introduced by Chawla, Bowyer, Hall, and Kegelmeyer, generates new synthetic examples rather than duplicating existing ones, describing an approach to constructing classifiers from imbalanced datasets where a dataset is imbalanced if the classification categories are not approximately equally represented, and where misclassifying a rare, interesting example is often far more costly than the reverse error (Chawla et al., 2002). SMOTE picks a minority-class example, finds its nearest minority-class neighbors in feature space, and creates a new synthetic point somewhere along the line segment connecting the original point and a randomly chosen neighbor. This means SMOTE is not simply copying rows — it interpolates between real examples to manufacture plausible new ones.
Important parameters include k_neighbors (how many neighbors to consider) and the target sampling_strategy (how much to balance the classes). Potential benefits include reduced overfitting compared with pure duplication and richer coverage of the minority-class feature space. Risks include: generating unrealistic synthetic samples in regions of heavy class overlap; amplifying noise if a noisy or mislabeled minority example is chosen as a seed; poor behavior with purely categorical variables, since interpolating between category codes is not meaningful (SMOTE-NC exists specifically to address this); difficulty in very sparse or very high-dimensional spaces, where the notion of "nearest neighbor" becomes less reliable; and the general risk that synthetic points can end up sitting inside the majority-class region, actively confusing the decision boundary rather than clarifying it.
Other Sampling Approaches
Borderline-SMOTE focuses synthetic generation specifically near the decision boundary, where minority examples are considered "in danger" of being misclassified, rather than across the whole minority class.
ADASYN adaptively generates more synthetic samples for minority examples that are harder to learn, based on their local density relative to the majority class.
SMOTE-NC extends SMOTE to datasets that mix continuous and categorical (nominal) features.
Tomek links identify pairs of very close majority/minority examples and remove the majority member, cleaning up the boundary rather than balancing the whole dataset.
Edited nearest neighbours (ENN) removes examples whose class does not match the majority vote of their nearest neighbors, cleaning noisy regions.
SMOTE-Tomek and SMOTE-ENN combine oversampling with a boundary-cleaning step, generating synthetic examples and then removing points that create ambiguous regions.
Each of these methods addresses a specific weakness of plain random oversampling or undersampling — boundary ambiguity, adaptive difficulty, or categorical data — but none is universally safe. Every synthetic or cleaning method can still create implausible samples or discard genuinely useful data if applied without checking whether it improves validation performance on the untouched distribution.
Algorithm-Level and Cost-Sensitive Methods
Instead of changing the data, algorithm-level methods change the learning objective itself.
Class weighting assigns a higher weight to minority-class errors during training, so that misclassifying a minority example contributes more to the loss than misclassifying a majority example. Most scikit-learn classifiers accept a class_weight="balanced" parameter that sets weights inversely proportional to class frequency.
Sample weighting is a more granular version, assigning weights to individual training examples rather than entire classes — useful when some examples are more reliable or more important than others within a class.
Cost-sensitive learning generalizes this idea to explicit misclassification costs: instead of treating every error the same, you define a cost matrix reflecting the true business or clinical cost of each error type, and the algorithm optimizes against that cost structure.
Custom loss functions let you encode imbalance-awareness directly into the objective a neural network minimizes.
Focal loss, introduced by Lin, Goyal, Girshick, He, and Dollár for dense object detection, reshapes the standard cross-entropy loss so that well-classified, "easy" examples contribute less to the total loss, letting training focus on hard examples, motivated by the discovery that extreme foreground-background class imbalance during training of dense object detectors was the central obstacle to their accuracy (Lin et al., 2017). Though developed for computer vision, the underlying idea — down-weighting easy majority examples so gradient signal concentrates on the hard, often minority, cases — generalizes to other extreme-imbalance classification settings.
Balanced random forests and EasyEnsemble-style methods build ensembles where each base learner is trained on a class-balanced subsample (typically all minority examples plus a random majority subsample), then combine their votes.
Gradient-boosting class-weight parameters (such as scale_pos_weight in gradient-boosting libraries) let you reweight the positive class directly within the boosting objective.
Anomaly-detection or one-class approaches treat the minority class as "anomalous" relative to a majority-class model of normal behavior, which can be more appropriate than standard classification when the minority class is exceptionally rare (fractions of a percent) or too sparse to model directly as a second class.
Class weights should not be chosen mechanically. "Balanced" weighting is a reasonable starting point, but the weights that actually reflect your real error costs may be different, and only validation results — evaluated on the untouched, natural class distribution — tell you whether a given weighting scheme genuinely helps.
Decision-Threshold Tuning
Training a model and deciding how to convert its output scores into a "yes/no" action are two separate decisions. A model's predict_proba output can be excellent while the default 0.5 cutoff is a poor choice for your situation. Scikit-learn's documentation on threshold tuning notes that tuning the decision threshold of a classifier once it has been trained can address the mismatch between a fixed default cutoff and the metric a business actually cares about, using an internal cross-validation to choose the threshold that maximizes a chosen metric, and that in one worked example, a vanilla classifier predicting the positive class above a probability of 0.5 and a tuned classifier predicting it above roughly 0.02 share identical probability outputs and ROC/PR curves, yet produce very different class-label decisions because their thresholds differ.
0.5 is not universally optimal. It is only a sensible default when false positives and false negatives are equally costly and the classes are roughly balanced — rarely true in imbalanced problems. Thresholds should generally be tuned on validation data (or via nested cross-validation), never on the test set, since tuning on test data is a form of leakage that produces an overly optimistic final estimate. Cost-based threshold selection picks the cutoff that minimizes expected cost given your specific false-positive and false-negative costs. Capacity constraints matter too: if a fraud team can only manually review 100 flagged cases per day, the threshold should be set to produce roughly that many alerts, not an abstractly "optimal" number. Group-specific thresholds are sometimes used, but only where legally, ethically, and operationally justified — this is a sensitive design decision, not a default. Finally, deployment prevalence can shift what threshold is appropriate: a threshold tuned when fraud is 1% of transactions may no longer be right if fraud rates change to 3%.
Small numerical example: suppose reviewing a false positive costs $5 in analyst time, and missing actual fraud (a false negative) costs $500 in unrecovered losses. If a proposed threshold produces 200 false positives and 10 false negatives on your validation set, expected cost = (200 × $5) + (10 × $500) = $1,000 + $5,000 = $6,000. Comparing this total across several candidate thresholds — not just comparing accuracy or F1 — is how cost-based threshold selection actually works in practice.
Probability Calibration
Calibration means that when a model says "70% probability," roughly 70% of examples given that score should actually belong to the positive class. This is different from ranking quality: a model can rank examples almost perfectly (high scores for true positives, low scores for true negatives) while still producing probabilities that are systematically too high or too low.
Reliability diagrams plot predicted probability against observed frequency, visually exposing miscalibration. The Brier score and log loss quantify calibration and ranking quality jointly, in a single number.
Scikit-learn's CalibratedClassifierCV supports two long-standing approaches: sigmoid calibration, corresponding to Platt's method — essentially a logistic regression fit on top of the classifier's raw scores — and isotonic calibration, a non-parametric, more flexible approach. As the documentation notes, sigmoid calibration is generally preferable when the calibration curve itself is roughly sigmoid-shaped and calibration data is limited, while isotonic calibration is preferable for non-sigmoid calibration curves and when more calibration data is available, since isotonic regression tends to overfit with very few samples.
Calibration should always be fit using held-out data or appropriate cross-validation, never on the same data used to fit the underlying classifier — using the same data for both would let the calibrator simply memorize training noise. Resampling and class weighting both distort a model's raw output scores relative to true real-world probabilities, so a model trained on SMOTE-resampled or heavily reweighted data usually needs a separate, explicit calibration step before its probabilities can be trusted for a risk-based decision — for example, when a predicted probability directly feeds a monetary risk calculation, an insurance premium, or a clinical risk score. Not every imbalanced model is automatically uncalibrated, however; some models remain reasonably well calibrated even on skewed data, which is exactly why measuring calibration directly (reliability diagrams, Brier score) is more reliable than assuming it either way.
Cross-Validation and Data-Leakage Prevention
Never resample the full dataset before splitting it into train and test partitions.
If you oversample or undersample before splitting, information about examples that end up in your "test" set can influence which synthetic points get created or which majority rows get kept, and — in the case of duplication or interpolation — near-duplicate versions of test examples can leak straight into training. The imbalanced-learn project's documentation on common pitfalls describes exactly this failure: resampling the entire dataset before splitting it into train and test partitions is a common pitfall, because the model ends up tested on a dataset with a class distribution similar to training rather than the natural, deployment-like distribution, and because the resampling procedure can use information from samples that will later be used for testing — the standard definition of data leakage. The practical consequence is well documented: leaked pipelines report cross-validated performance that looks strong, but the same model tested correctly on unresampled data performs noticeably worse.
The correct pattern:
Split off an untouched, representative test set first, using stratification (and grouping or temporal ordering where relevant).
Apply resampling only to the training partition.
During cross-validation, apply resampling separately inside each training fold — never once, globally, before folding.
Use an imbalanced-learn Pipeline (not the plain scikit-learn Pipeline) so that sampling steps are automatically confined to each fold's training data.
Fit any other preprocessing (scaling, encoding) only on training folds, exactly as with the sampler.
Evaluate every candidate model on the original, unresampled validation and test distributions.
Beyond leakage, several cross-validation choices matter specifically for imbalanced data: stratified cross-validation preserves class proportions in every fold; repeated stratified cross-validation runs this process multiple times with different random splits to reduce variance in the estimate, which matters more when the minority class is small; group-aware splitting prevents the same entity from appearing in both train and validation folds; time-series or temporal validation respects chronological order rather than random shuffling; and nested cross-validation is worth the added computation whenever extensive hyperparameter or threshold tuning is being performed, since it separates the tuning process from the final performance estimate.
Practical Python Examples
The following examples use scikit-learn and imbalanced-learn. Exact printed numbers will depend on your library versions, random seed, and dataset, so treat the code — not any specific fabricated output — as the reference.
Example 1: Building an imbalanced dataset and a leakage-safe baseline
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
confusion_matrix, classification_report,
balanced_accuracy_score, roc_auc_score, average_precision_score
)
# Create a synthetic, imbalanced binary classification dataset
X, y = make_classification(
n_samples=10000, n_features=20, n_informative=5,
weights=[0.97, 0.03], # roughly 97:3 imbalance
flip_y=0.01, random_state=42
)
# Split BEFORE any resampling — the test set stays untouched and natural
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42
)
# Baseline: no resampling, no special weighting
baseline = LogisticRegression(max_iter=1000, random_state=42)
baseline.fit(X_train, y_train)
y_pred = baseline.predict(X_test)
y_proba = baseline.predict_proba(X_test)[:, 1]
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))
print("Balanced accuracy:", balanced_accuracy_score(y_test, y_pred))
print("ROC-AUC:", roc_auc_score(y_test, y_proba))
print("Average precision (PR-AUC):", average_precision_score(y_test, y_proba))Example 2: Comparing class_weight="balanced" against a leakage-safe SMOTE pipeline inside cross-validation
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.ensemble import RandomForestClassifier
from imblearn.pipeline import Pipeline as ImbPipeline # imbalanced-learn's Pipeline
from imblearn.over_sampling import SMOTE
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = ["balanced_accuracy", "roc_auc", "average_precision"]
# Treatment A: class weighting, no resampling
model_weighted = RandomForestClassifier(
n_estimators=300, class_weight="balanced", random_state=42
)
# Treatment B: SMOTE applied ONLY inside each training fold via an imblearn Pipeline
model_smote = ImbPipeline([
("smote", SMOTE(random_state=42)), # leakage-safe: fit only on training folds
("clf", RandomForestClassifier(n_estimators=300, random_state=42)),
])
results_weighted = cross_validate(model_weighted, X_train, y_train, cv=cv, scoring=scoring)
results_smote = cross_validate(model_smote, X_train, y_train, cv=cv, scoring=scoring)
for name, res in [("class_weight=balanced", results_weighted), ("SMOTE pipeline", results_smote)]:
print(name)
for metric in scoring:
scores = res[f"test_{metric}"]
print(f" {metric}: mean={scores.mean():.3f}, std={scores.std():.3f}")Inspect the mean and standard deviation for each metric across folds rather than assuming either treatment "wins" — the better choice depends on your data, and neither method is guaranteed to outperform the other.
Example 3 (optional): Tuning the decision threshold on validation data, not the test set
from sklearn.model_selection import TunedThresholdClassifierCV
tuned_model = TunedThresholdClassifierCV(
estimator=RandomForestClassifier(n_estimators=300, class_weight="balanced", random_state=42),
scoring="balanced_accuracy", # or a custom cost-based scorer
cv=5,
)
tuned_model.fit(X_train, y_train) # threshold is tuned internally via cross-validation
# Only now evaluate once, on the held-out test set
y_pred_tuned = tuned_model.predict(X_test)
print(classification_report(y_test, y_pred_tuned, digits=3))Note that the test set is touched exactly once, at the very end, for a single final evaluation — never for choosing the threshold, the sampling method, or the hyperparameters.
End-to-End Workflow
Define the decision being made and the real costs of each error type.
Audit labels and the data-collection process for quality and consistency.
Split the data correctly — stratified, grouped, or temporal as appropriate — before touching anything else.
Establish a naive baseline and record its per-class performance.
Select metrics that reflect the decision's real costs.
Evaluate ranking quality, threshold performance, and calibration separately.
Try the simplest justified intervention first (often class weighting).
Tune using validation data or cross-validation, never the test set.
Compare multiple methods fairly, under identical validation procedures.
Test once, on untouched data that reflects deployment conditions.
Select an operating threshold based on cost or capacity constraints.
Document every assumption made along the way.
Deploy gradually, with a rollback plan.
Monitor prevalence, drift, calibration, and error costs continuously after launch.
This workflow matters more than picking the most fashionable sampling algorithm, because most real failures in imbalanced-learning projects come from skipped steps — untouched test data, wrong metrics, or leaked resampling — not from choosing SMOTE over ADASYN.
Choosing the Right Method
Method | Consider It When | Main Advantage | Main Risk | Validation Requirement |
Collect more minority data | Feasible and time allows | Genuinely new information | Slow, costly | Standard split suffices |
Class weighting | Quick first intervention | Simple, no data duplication | May not fix severe overlap | Cross-validation |
Random oversampling | Small-to-moderate imbalance | Simple, preserves all majority information | Overfitting via duplication | Fold-safe resampling |
Random undersampling | Abundant majority data | Faster training | Discards majority information | Fold-safe resampling |
SMOTE / synthetic sampling | Continuous features, some overlap tolerance | Adds diversity vs. duplication | Unrealistic points in overlapping/categorical/high-dimensional space | Fold-safe pipeline |
Ensemble methods (balanced RF, EasyEnsemble) | Tree-based models, moderate-to-severe imbalance | Robust, less tuning-sensitive | More complex to explain | Cross-validation |
Threshold tuning | Any imbalanced problem | Cheap, no retraining | Wrong if tuned on test data | Validation or nested CV |
Probability calibration | Probabilities feed a cost or risk decision | Trustworthy probabilities | Needs separate held-out data | Held-out calibration set |
Anomaly detection | Exceptionally rare event (<<1%) | Suited to near-absent positive class | Different mental model, may need re-education of stakeholders | Careful, often unsupervised metrics |
Practical guidance by scenario:
Millions of majority samples, still enough minority examples: class weighting or a fold-safe SMOTE pipeline both tend to work; compare empirically.
Very few minority examples: be cautious with SMOTE (few neighbors to interpolate between); consider anomaly detection, simpler models, or focused data collection instead.
Severe class overlap: resampling and weighting have limited effect; invest in better features or reconsider whether the two classes are actually well defined.
Noisy minority labels: clean labels first — resampling amplifies whatever noise already exists.
High-dimensional sparse text: class weighting or algorithm-level methods are often more reliable than distance-based synthetic sampling, since "nearest neighbor" becomes less meaningful in sparse, high-dimensional space.
Mixed numeric and categorical data: use SMOTE-NC or algorithm-level methods rather than plain SMOTE.
Deep-learning tasks: class-weighted loss, focal loss, or oversampled minibatches are common starting points.
Streaming or changing prevalence: favor threshold tuning and calibration monitoring over static resampling ratios fixed at training time.
High false-positive costs: prioritize precision, tune the threshold upward, and treat calibration carefully.
High false-negative costs: prioritize recall, tune the threshold downward, and consider a lower, capacity-constrained operating point.
When Not to Balance the Dataset
Forcing an artificial 50:50 split is sometimes unnecessary or actively harmful. Situations where you should think twice before resampling:
The original prevalence carries important information — for a well-calibrated risk model, the natural base rate is part of the signal, not noise to be erased.
The model already performs well under suitable metrics — if PR-AUC, balanced accuracy, and per-class recall are all acceptable, resampling adds risk without benefit.
Ranking matters more than hard class labels — if the end use is "sort these 10,000 cases by risk," ranking quality (PR-AUC, ROC-AUC) matters more than any specific threshold-based balance.
Class weights or threshold tuning are already sufficient — cheaper, lower-risk interventions that solve the same problem without touching the data.
Synthetic sampling would create implausible observations — especially with categorical, sparse, or highly overlapping data.
The minority labels are too noisy to trust as a basis for interpolation or duplication.
Probability estimates must reflect real prevalence — for instance, actuarial or epidemiological models where the base rate itself is the quantity of interest.
Deployment conditions differ from the training sample — resampling training data does nothing to fix a mismatch with deployment-time prevalence; that's a calibration and monitoring problem, not a resampling problem.
More representative data collection is a better long-term fix than any algorithmic patch.
Balanced training data is a means, not the goal. The goal is a model that produces useful, reliable decisions under the real conditions it will face.
Common Mistakes and Failure Modes
Using accuracy alone. Correction: pair it with balanced accuracy, PR-AUC, or per-class recall.
Resampling before splitting. Correction: split first, resample only inside training folds.
Evaluating on a balanced test set that doesn't represent deployment. Correction: keep the test set's natural class distribution.
Treating 0.5 as a universal threshold. Correction: tune the threshold on validation data using real costs.
Applying SMOTE to raw categorical features without an appropriate variant. Correction: use SMOTE-NC or an algorithm-level method instead.
Oversampling noisy or mislabeled examples. Correction: audit and clean labels before resampling.
Ignoring calibration. Correction: check reliability diagrams and Brier score whenever probabilities drive decisions.
Reporting only aggregate averages in multiclass problems. Correction: always report per-class metrics alongside macro/weighted averages.
Choosing a metric after seeing test results. Correction: fix the evaluation metric before looking at test performance.
Treating all false positives and false negatives as equally costly. Correction: build an explicit cost model where the two error types genuinely differ.
Assuming the minority class is always the positive class. Correction: define which class matters based on the business question, not dataset size.
Forgetting temporal or group leakage. Correction: use time-aware or group-aware splitting when rows aren't independent.
Comparing models evaluated under different validation procedures. Correction: hold the validation protocol constant across every candidate.
Deploying without monitoring prevalence changes. Correction: track class prevalence and model calibration continuously in production.
Real-World Examples
Credit-card or payment fraud: minority/rare event is a fraudulent transaction; false negatives (missed fraud) usually cost far more than false positives (a blocked legitimate transaction), so recall and PR-AUC often take priority; typical treatment blends class weighting, threshold tuning, and post-hoc review capacity.
Medical diagnosis or screening: the rare event is a positive disease finding; missing a disease (false negative) is usually far more costly than a false alarm; sensitivity (recall) at a clinically acceptable specificity is a common target metric, alongside calibration for risk communication.
Manufacturing defects: the rare event is a defective unit; cost trade-offs vary by industry — in safety-critical manufacturing, missed defects (false negatives) can be extremely costly, so recall-oriented thresholds and per-class metrics matter.
Cybersecurity and intrusion detection: the rare event is a genuine intrusion or attack signature; false positives create alert fatigue for security teams, while false negatives allow a breach through, making the precision/recall trade-off and capacity-based thresholding especially important.
Equipment failure prediction: the rare event is an imminent failure; false negatives (missed failures) risk costly unplanned downtime, while false positives trigger unnecessary maintenance; balanced accuracy and PR-AUC are common evaluation choices.
Customer churn: the rare event is a customer who leaves; false negatives mean lost revenue from unaddressed churn risk, while false positives mean wasted retention-offer spend; the trade-off is directly monetizable, making cost-based thresholding a natural fit.
Content moderation: the rare event is genuinely policy-violating content; false positives over-censor legitimate content, while false negatives allow harmful content through; precision and recall are both scrutinized carefully, along with subgroup performance.
Rare-object detection (computer vision): the rare event is the object of interest in a mostly empty or background-dominated image; this is exactly the setting focal loss was designed to address, since easy background regions vastly outnumber the object pixels.
Class Imbalance, Bias, and Fairness
Class imbalance and algorithmic bias are related but distinct concepts, and it's worth keeping them separate. Class imbalance describes the distribution of the target label — how many examples belong to each outcome class. Algorithmic bias and demographic imbalance describe how a model's errors are distributed across protected or sensitive subgroups — a different axis of "class" altogether, namely a demographic group, not a target label.
These two problems can interact in important ways. A model with strong overall minority-class recall can still perform poorly for a specific demographic subgroup if that subgroup is itself under-represented within the minority class — a form of intersectional imbalance that a single aggregate metric will not reveal. Evaluation should therefore often include subgroup-specific error analysis (per-group precision, recall, and calibration), not just an overall minority-class score.
Resampling techniques such as SMOTE or class weighting address the target-label imbalance; they do not, by themselves, guarantee fairness across demographic groups, and they can occasionally worsen subgroup disparities if the resampling process amplifies patterns that were already skewed within a particular group's data. Fairness auditing is a separate, deliberate step, not a side effect of fixing class imbalance. This guide does not offer legal conclusions on fairness requirements; consult qualified compliance or legal expertise for regulated applications.
Production Monitoring
A treatment that worked well at training time can silently degrade after deployment. Monitor:
Class prevalence — is the real-world positive rate still close to what training assumed?
Predicted-score distributions — are scores drifting up or down over time?
Precision and recall once true labels arrive (which may be delayed).
False-positive and false-negative rates in absolute terms, not just ratios.
Calibration — do reliability diagrams still track the diagonal?
Threshold stability — does the chosen threshold still meet the intended cost or capacity target?
Data drift — are input feature distributions changing?
Concept drift — has the relationship between features and label itself changed?
Subgroup performance — is any demographic or operational subgroup degrading disproportionately?
Label delays — are enough confirmed outcomes arriving to support reliable monitoring?
Review capacity — can operations teams still handle the volume of flagged cases?
Business costs — have the real costs of false positives or false negatives shifted?
A model that performed well during training and validation offers no guarantee about its behavior months later, especially in domains — fraud, security, churn — where the adversary or the customer base actively adapts.
Myths and Misconceptions
Myth: "Any dataset that is not 50:50 is imbalanced." Fact: mild skew (like 60:40) is rarely a real obstacle; meaningful imbalance is a matter of degree and consequence, not a fixed cutoff.
Myth: "High accuracy means the model is good." Fact: as the worked fraud example shows, 99% accuracy can coexist with 0% minority-class recall.
Myth: "SMOTE is always the best solution." Fact: SMOTE can underperform class weighting or simple threshold tuning, especially with categorical features, heavy overlap, or very few minority examples to interpolate between.
Myth: "Oversampling automatically creates more information." Fact: random oversampling duplicates existing information; even SMOTE only interpolates within the existing feature space — it cannot invent genuinely new signal the data never contained.
Myth: "ROC-AUC is useless for all imbalanced datasets." Fact: ROC-AUC remains a legitimate ranking metric; it can simply look more optimistic than PR-AUC under severe imbalance, which is a reason to report both, not to discard ROC-AUC entirely.
Myth: "The minority class is always the positive class." Fact: which class is "positive" is a modeling and business choice, independent of which class happens to be numerically rare.
Myth: "Class weights and oversampling are equivalent." Fact: they can produce similar effects on some loss functions, but they behave differently with regularization, with tree-based splitting criteria, and with probability calibration — they are not interchangeable in every algorithm.
Myth: "Balancing the training data guarantees calibrated probabilities." Fact: resampling and reweighting typically distort raw output probabilities relative to real-world prevalence, which is exactly why a dedicated calibration step is usually necessary afterward.
A Concise Decision Framework
Use these six checkpoints, in order, on any new imbalanced-classification problem:
Data — What is the true imbalance ratio and absolute minority count? Is the data clean and representative?
Decision — What real-world decision does this model support, and what does each error type actually cost?
Distribution — Does the training, validation, and deployment class distribution match, or will it shift?
Diagnostics — What does a naive baseline achieve, and where specifically does it fail (recall, precision, calibration)?
Design — Which combination of resampling, weighting, and threshold tuning is justified by validation evidence, not habit?
Deployment — How will prevalence, drift, and calibration be monitored once the model is live?
This framework is meant to be walked through on every project, not memorized as a slogan — each checkpoint forces a specific, falsifiable question rather than a vague reminder to "be careful with imbalanced data."
FAQ
What is class imbalance in simple terms?
Class imbalance means one outcome category in your dataset shows up far more often than another. For example, in a dataset of manufacturing parts, 98% might be "good" and only 2% "defective." Models trained naively on this kind of data can learn to just guess the common category and still look accurate, even though they fail at the task that actually matters.
What is an example of class imbalance?
Credit-card fraud detection is a classic example: out of 10,000 transactions, only around 100 might be fraudulent, giving a roughly 99:1 imbalance. Similar patterns show up in rare-disease screening, equipment-failure prediction, and spam detection, where the "interesting" event is naturally uncommon compared with normal, everyday cases.
How do you know whether a dataset is imbalanced?
Count the examples in each class and compute the imbalance ratio (majority count divided by minority count). Also check the absolute minority-class count, not just the percentage — 1% of 100,000 rows behaves very differently than 1% of 500 rows. Visualizing the distribution and checking it across your train, validation, and test splits completes the picture.
Is a 60:40 dataset imbalanced?
Technically yes, since the classes aren't equal, but a 60:40 split is usually mild and rarely causes serious modeling problems on its own. Severe, practically important imbalance usually starts appearing around 90:10 or beyond, and becomes a major concern at 99:1 or worse — though the real answer always depends on class overlap and error costs, not the ratio alone.
Why is accuracy misleading for imbalanced data?
Accuracy simply counts how many predictions were correct overall. When one class vastly outnumbers the other, a model can rack up a high accuracy score just by defaulting to the majority class, while completely failing to identify the minority class you actually care about — the accuracy paradox described earlier in this guide.
Which metric is best for class imbalance?
There is no single best metric. Balanced accuracy, PR-AUC (average precision), macro F1, and MCC are all commonly used, but the right choice depends on whether you care more about false positives, false negatives, ranking quality, or well-calibrated probabilities. Strong evaluations typically report several of these together rather than relying on one number.
Is F1 score enough for imbalanced data?
F1 is a reasonable single-number summary of the precision/recall trade-off, but it treats both errors as equally important, which may not match your real costs. It also doesn't capture ranking quality across thresholds the way PR-AUC does, so it's usually best used alongside, not instead of, other metrics.
Should I use ROC-AUC or PR-AUC?
For severely imbalanced problems where the positive (often minority) class is what you care most about, PR-AUC tends to give a more realistic picture, since it directly reflects how many of your positive predictions are actually correct. ROC-AUC remains useful as a general ranking metric, and reporting both together, rather than choosing one exclusively, is common good practice.
What is the difference between oversampling and undersampling?
Oversampling adds more minority-class examples (by duplication or synthesis) to balance the dataset, keeping all majority data intact. Undersampling removes majority-class examples instead, which shrinks the overall dataset. Oversampling risks overfitting to repeated or synthetic patterns; undersampling risks discarding useful majority-class information.
What does SMOTE do?
SMOTE generates new synthetic minority-class examples by picking a real minority example, finding its nearest minority neighbors, and creating a new point somewhere along the line between them. It's designed to add variety to the minority class rather than simply copying existing rows, which random oversampling does.
Can SMOTE cause overfitting?
Yes, particularly when the minority class is very small, when synthetic points fall inside heavily overlapping regions with the majority class, or when SMOTE is applied incorrectly across the full dataset before splitting. It can also propagate label noise if a mislabeled example is used as an interpolation seed.
Should SMOTE be applied before or after the train-test split?
Always after splitting off the test set, and only inside the training data (or inside each training fold during cross-validation). Applying SMOTE before splitting is a well-documented data-leakage mistake that produces overly optimistic performance estimates that won't hold up in production.
Are class weights better than SMOTE?
Neither is universally better; they behave differently depending on the algorithm, the amount of class overlap, and whether your features are continuous or categorical. Class weighting is usually simpler, cheaper, and less prone to creating implausible synthetic data, but SMOTE can sometimes help with certain distance-based or tree-based models. Compare both empirically on your own validation data.
Does class imbalance affect regression?
Class imbalance, by definition, applies to classification problems with discrete categories. Regression tasks can have an analogous issue — an imbalanced or skewed distribution of the continuous target — but the diagnostic tools and fixes differ, since there are no discrete classes, confusion matrices, or class-based resampling in the same sense.
How is multiclass imbalance handled?
The same core ideas apply — class weighting, resampling, and threshold or decision-rule tuning — but evaluation becomes more nuanced: use macro-averaged metrics to catch minority-class weaknesses, report per-class precision and recall individually, and be aware that some classes may need very different treatment than others within the same dataset.
Can deep-learning models handle class imbalance?
Deep-learning models are subject to the same imbalance dynamics as any classifier, but they have some additional tools available, including class-weighted loss functions, focal loss (originally designed for extreme foreground-background imbalance in object detection), and oversampled or class-balanced minibatch sampling during training. As with classical ML, no single deep-learning-specific fix is guaranteed to work best in every case.
Key Takeaways
Class imbalance is contextual — the same ratio can be trivial in one problem and severe in another, depending on overlap, sample size, and cost.
Accuracy alone is insufficient and can actively mislead you about a model's real usefulness on the class that matters.
Evaluation choices must reflect the real costs of false positives and false negatives, not a generic default metric.
Resampling belongs strictly inside the training data or training folds — leakage from resampling before splitting produces unreliable, overly optimistic results.
Threshold tuning and probability calibration are separate, essential decisions that happen after training, not automatic side effects of a well-trained model.
No single treatment — not SMOTE, not class weighting, not any other method — is universally the best fix; the right choice depends on your specific data and validation evidence.
Deployment monitoring is necessary because prevalence, drift, and calibration can all shift after launch, degrading a solution that worked well during training.
Actionable Next Steps
Count every class in your dataset and calculate the imbalance ratio and absolute minority count.
Split your data first — stratified, grouped, or temporal as your data requires — before any resampling.
Benchmark a naive baseline (dummy classifier and a simple model) and record its per-class metrics.
Measure balanced accuracy, PR-AUC, and per-class recall alongside accuracy, not instead of it.
Compare at least two treatments (for example, class weighting versus a fold-safe SMOTE pipeline) under identical validation procedures.
Validate every resampling step using an imbalanced-learn pipeline so it stays confined to training folds.
Calibrate probabilities on held-out data if your application relies on trustworthy probability estimates.
Monitor class prevalence, calibration, and per-class error rates continuously after deployment.
Glossary
Accuracy: The proportion of all predictions that are correct.
Average precision: A summary statistic of the precision-recall curve, closely related to PR-AUC.
Balanced accuracy: The average of recall computed separately across each class.
Calibration: How closely a model's predicted probabilities match the true observed frequency of the positive outcome.
Class imbalance: A classification dataset where one class has substantially more examples than another.
Class weight: A multiplier applied to a class's contribution to the training loss, used to counteract imbalance.
Confusion matrix: A table summarizing true positives, false positives, true negatives, and false negatives.
Cost-sensitive learning: Training approaches that explicitly account for different real-world costs for different error types.
False negative: An actual positive case incorrectly predicted as negative.
False positive: An actual negative case incorrectly predicted as positive.
F1 score: The harmonic mean of precision and recall.
Imbalance ratio: The number of majority-class examples divided by the number of minority-class examples.
Majority class: The class with the most examples in a dataset.
Minority class: The class with the fewest examples in a dataset; not necessarily the "positive" class.
Oversampling: Increasing the number of minority-class examples, by duplication or synthesis.
Precision: The proportion of positive predictions that are actually correct.
Precision-recall curve: A plot of precision against recall across different decision thresholds.
PR-AUC: The area under the precision-recall curve; also called average precision.
Recall: The proportion of actual positive cases that a model correctly identifies; also called sensitivity.
ROC-AUC: The area under the Receiver Operating Characteristic curve, plotting true positive rate against false positive rate.
SMOTE: Synthetic Minority Over-sampling Technique; generates new minority-class examples by interpolating between existing ones.
Stratified sampling: Splitting data so that each subset preserves the original class proportions.
Threshold: The cutoff probability or score above which a model predicts the positive class.
Undersampling: Reducing the number of majority-class examples to rebalance a dataset.
Sources & References
Documentation
scikit-learn developers. 3.4. Metrics and scoring: quantifying the quality of predictions. scikit-learn 1.9.0 documentation. https://scikit-learn.org/stable/modules/model_evaluation.html
scikit-learn developers. balanced_accuracy_score. scikit-learn 1.9.0 documentation. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html
scikit-learn developers. 3.3. Tuning the decision threshold for class prediction. scikit-learn 1.9.0 documentation. https://scikit-learn.org/stable/modules/classification_threshold.html
scikit-learn developers. TunedThresholdClassifierCV. scikit-learn 1.9.0 documentation. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TunedThresholdClassifierCV.html
scikit-learn developers. 1.16. Probability calibration. scikit-learn 1.9.0 documentation. https://scikit-learn.org/stable/modules/calibration.html
scikit-learn developers. CalibratedClassifierCV. scikit-learn 1.5.2 documentation. https://scikit-learn.org/1.5/modules/generated/sklearn.calibration.CalibratedClassifierCV.html
scikit-learn developers. 12. Common pitfalls and recommended practices. scikit-learn 1.9.0 documentation. https://scikit-learn.org/stable/common_pitfalls.html
imbalanced-learn developers. 9. Common pitfalls and recommended practices. imbalanced-learn 0.14.2 documentation. https://imbalanced-learn.org/stable/common_pitfalls.html
imbalanced-learn developers. SMOTE. imbalanced-learn 0.14.2 documentation. https://imbalanced-learn.org/stable/references/generated/imblearn.over_sampling.SMOTE.html
imbalanced-learn developers. imbalanced-learn documentation. https://imbalanced-learn.org/
Academic Papers
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321–357. https://doi.org/10.1613/jair.953
He, H., & Garcia, E. A. (2009). Learning from Imbalanced Data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263–1284.
Saito, T., & Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE, 10(3), e0118432. https://doi.org/10.1371/journal.pone.0118432
Davis, J., & Goadrich, M. (2006). The Relationship Between Precision-Recall and ROC Curves. Proceedings of the 23rd International Conference on Machine Learning (ICML), 233–240.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2980–2988. https://doi.org/10.1109/ICCV.2017.324
