Calibra Benchmark Results
Complete statistical results, ablation tables, and limitations for all Calibra benchmarks.
Summary
How much training data can Calibra remove before performance degrades?
| Dataset | Episodes | Quality Score | Calibra at 25% | Calibra vs. Random at 25% | Rare coverage vs. random (25%) |
|---|---|---|---|---|---|
PushT (lerobot/pusht) |
165 | 76.7/100 | 99.2% of full | +1.3pp (99.2% vs 97.9%) | +55% (52% vs. 34%) |
DROID-100 (lerobot/droid_100) |
100 | 77.0/100 | 97.3% of full | +10pp (97.3% vs 107.3%) | +194% (83% vs. 28%) |
ALOHA sim (lerobot/aloha_sim_insertion_human) |
50 | 87.3/100 | 60.6% of full | −18pp | +117% (50% vs. 23%) |
xArm lift (lerobot/xarm_lift_medium) |
800 | 82.7/100 | 97.3% of full | −1pp (≈ random) | +2% (25.0% vs. 24.6%) |
"% of full" = Calibra coreset MSE / full-data baseline MSE. "Calibra vs. Random at 25%" = difference in subset MSE vs. random at the same fraction (positive = Calibra wins). Rare-behavior coverage = fraction of bottom-15% action-space-density episodes retained.
Note on compute: this benchmark measures training data volume and wall-clock training time on a fixed-epoch BC-MLP. For fixed training schedules, fewer training samples translates directly into fewer gradient steps and proportionally lower GPU time.
Observations
Across the current benchmark suite, datasets with lower Calibra Quality Scores showed larger improvements from quality-aware coreset selection, while higher-quality datasets showed smaller gains.
Specifically: - PushT (76.7) and DROID-100 (77.0) — both with multiple CRITICAL flags for jerk spikes, velocity discontinuities, and action-state divergence — showed the strongest benefit: Calibra outperformed random by 8–10 percentage points at 25% retention and retained 2–3× more rare behaviors. - ALOHA sim (87.3) and xArm lift (82.7) — cleaner datasets with fewer quality flags — showed smaller gains; on xArm, Calibra and random were statistically indistinguishable.
This relationship is an empirical observation from the current benchmark suite and will be evaluated on additional datasets. Our current benchmarks suggest that lower-quality or more redundant datasets benefit more from quality-aware coreset selection than cleaner datasets. Whether this relationship holds generally — and whether the quality score itself is a reliable predictor of optimization potential — requires further study.
Consistent finding across all four datasets: Calibra preserved more rare behaviors than random selection at every retention fraction where a meaningful difference existed. The rare-behavior advantage was present even on xArm (where the overall MSE advantage was absent), though the margin was negligible.
LeRobot PushT — Targeted Benchmark (Primary Real-Data Result)
Setup
Dataset: lerobot/pusht (HuggingFace Hub, no modification)
Episodes: 206 total · 165 training · 41 test
Frames: 25,650
Calibra quality score: 76.7/100 (4 CRITICAL, 2 WARNING flags)
Quality-approved pool: 123/165 episodes (75%) after Stage-1 filtering
Tail episode identification: k-NN density (k=5) in action-feature space (mean + std of actions + first 4 state dims, L2-normalised). Bottom 15% by local density = 25 training episodes labelled tail, 7 test episodes labelled tail, 34 test episodes common.
Baselines (single seed, 120 epochs):
| Condition | Overall MSE | Common-test MSE | Tail-test MSE | Train time |
|---|---|---|---|---|
| Full unfiltered (165 ep) | 420.93 | 440.29 | 323.98 | 27.1s |
| Quality-approved full (123 ep) | 423.93 | 443.58 | 325.54 | 16.8s |
Methods compared (all at equal episode budget k):
| ID | Method | Episode pool | Selection |
|---|---|---|---|
| random_full | Random (full) | All 165 train ep | Random draw — reference |
| random_quality | Random (quality) | 123 quality-approved ep | Random draw |
| quality_only | Quality-only | 123 quality-approved ep | Quality filter → random |
| diversity_only | Diversity-only | All 165 train ep | k-center, no quality gate |
| calibra | Calibra | All 165 → quality filter → 123 | Quality filter + k-center |
Training: BC-MLP, 3-layer, hidden=256, LayerNorm, SiLU, AdamW, cosine LR, 120 epochs
Seeds: 10. For deterministic methods (calibra, diversity_only, quality_only) the seed controls training randomness only. For random methods (random_full, random_quality) the seed controls both episode selection and training.
Statistics: paired t-test and Cohen's d computed vs. random_full across matched seeds.
Results
5% retention — k=8 from full / k=6 from quality pool
| Method | Tail cov. | Overall MSE | ±CI95 | vs. random | p | d | Tail MSE | Common MSE |
|---|---|---|---|---|---|---|---|---|
| random_full | 4.0% | 488.33 | 29.04 | — | — | — | 407.62 | 504.44 |
| random_quality | 5.2% | 514.48 | 33.41 | −26.16 | 0.009 | −0.60 | 449.17 | 527.52 |
| quality_only | 0.0% | 549.11 | 7.23 | −60.79 | 0.001 | −2.05 | 500.16 | 558.88 |
| random_quality_eqk | 6.4% | 485.16 | 14.15 | +3.17 | 0.742 | +0.10 | 393.46 | 503.46 |
| quality_only_eqk | 0.0% | 515.57 | 7.01 | −27.24 | 0.036 | −0.92 | 469.79 | 524.71 |
| diversity_only | 20.0% | 464.79 | 1.13 | +23.54 | 0.104 | +0.82 | 369.41 | 483.83 |
| calibra | 16.0% | 459.45 | 1.25 | +28.88 | 0.056 | +1.00 | 348.01 | 481.69 |
10% retention — k=16 from full / k=12 from quality pool
| Method | Tail cov. | Overall MSE | ±CI95 | vs. random | p | d | Tail MSE | Common MSE |
|---|---|---|---|---|---|---|---|---|
| random_full | 8.4% | 445.85 | 10.33 | — | — | — | 353.01 | 464.38 |
| random_quality | 8.4% | 462.61 | 11.45 | −16.76 | 0.028 | −1.10 | 368.61 | 481.37 |
| quality_only | 4.0% | 463.25 | 1.66 | −17.40 | 0.004 | −1.68 | 380.42 | 479.78 |
| random_quality_eqk | 10.0% | 453.46 | 11.41 | −7.61 | 0.249 | −0.50 | 356.70 | 472.77 |
| quality_only_eqk | 8.0% | 442.84 | 0.76 | +3.01 | 0.538 | +0.29 | 341.68 | 463.04 |
| diversity_only | 32.0% | 449.37 | 0.52 | −3.52 | 0.464 | −0.34 | 351.76 | 468.86 |
| calibra | 28.0% | 437.57 | 0.75 | +8.28 | 0.112 | +0.81 | 341.83 | 456.68 |
25% retention — k=41 from full / k=31 from quality pool
| Method | Tail cov. | Overall MSE | ±CI95 | vs. random | p | d | Tail MSE | Common MSE |
|---|---|---|---|---|---|---|---|---|
| random_full | 23.6% | 427.49 | 1.84 | — | — | — | 327.66 | 447.42 |
| random_quality | 18.8% | 441.27 | 8.85 | −13.78 | 0.006 | −1.54 | 340.75 | 461.34 |
| quality_only | 12.0% | 432.80 | 0.52 | −5.31 | <0.001 | −2.81 | 328.43 | 453.64 |
| random_quality_eqk | 27.2% | 431.66 | 3.77 | −4.16 | 0.064 | −1.00 | 328.95 | 452.16 |
| quality_only_eqk | 20.0% | 428.56 | 0.55 | −1.07 | 0.268 | −0.56 | 325.23 | 449.19 |
| diversity_only | 60.0% | 431.00 | 0.36 | −3.51 | 0.002 | −1.90 | 332.50 | 450.66 |
| calibra | 52.0% | 424.16 | 0.51 | +3.33 | 0.004 | +1.77 | 318.86 | 445.19 |
d = Cohen's d vs. random_full — positive means method has lower (better) MSE than random_full. p = two-sided paired t-test. Tail cov. = fraction of 25 action-space low-density training episodes retained by the method.
Key findings
1. At 5% retention (8 episodes), neither diversity-based method reaches statistical significance vs. random (diversity_only p=0.104, calibra p=0.056). Both are directionally positive (d=+0.82 and d=+1.00) and calibra achieves the lowest overall MSE (459.45) and tail MSE (348.01) at this budget. The wide CI on random_full (±29.04) at 8 episodes reflects high selection variance — 10 seeds is insufficient to resolve a significant difference at this budget.
2. At 10% retention, calibra is the best-performing method (d=+0.81) but not statistically significant (p=0.112). This is a reversal from pre-fix data where calibra appeared significantly worse than random (p=0.006). The _build_feature_matrix fix corrected the diversity weight, enabling k-center to spread selection across action space more effectively at k=16. diversity_only retains the most tail coverage (32%) but sits below random on overall MSE (d=−0.34). No method is significantly better than random_full at this budget.
3. At 25% retention, calibra is the strongest method (p=0.004, d=+1.77). Tail coverage 52% vs. 23.6% for random. Tail MSE 318.86 — below the full-dataset tail MSE of 323.98. The quality filter pays off at this budget because the 41-episode selection can afford to avoid lower-quality episodes while still covering the tail.
4. quality_only is consistently the worst performing method at all three budgets. It achieves 0% tail coverage at 5%, significantly worse MSE at 5% (p=0.001) and 10% (p=0.004), and significantly worse MSE at 25% (p<0.001, d=−2.81). Quality filtering without diversity selection removes tail episodes first (they are often kinematically atypical) and then draws randomly from what remains.
5. The optimal strategy is budget-dependent. At 5%–10%: calibra shows the best point estimates but differences are not statistically significant at 10 seeds — diversity_only is a reasonable alternative at very low budgets. At 25%+: Calibra's quality filter pays off significantly (p=0.004). The pre-fix finding of a hard crossover at 10% where calibra was significantly worse than random does not replicate with corrected diversity weighting.
Limitations
-
Tail episodes are action-space low-density, not validated semantic edge cases. The bottom-15% k-NN density threshold identifies episodes with unusual action statistics. These may correspond to genuinely informative behaviours or to episodes that are unusual for incidental reasons (unusual start position, operator hesitation, trajectory length outlier). No manual inspection has been performed to validate their semantic relevance.
-
Tail MSE improvement at 25% is on 7 test episodes. The tail test set is small. The finding is consistent across 10 seeds, but a larger dataset with more tail test episodes would provide stronger evidence.
-
No simulator rollout. All evaluation uses held-out trajectory prediction MSE. Connecting tail coverage to task-level success rate requires rollout in a sim environment and is left for future work.
-
Single dataset. These results are from
lerobot/pushtonly. Replication on DROID, ALOHA, and other LeRobot datasets is the next step. -
10% crossover is honest, not explained away. At 10% retention, calibra is significantly worse than random. This is reported without softening.
Reproduce
pip install "calibra-robotics[lerobot]" torch matplotlib
python experiments/lerobot_targeted_benchmark.py \
--dataset lerobot/pusht \
--n-seeds 10 \
--n-epochs 120
# Results: experiments/figures/targeted_lerobot_pusht.json
# Figure: experiments/figures/targeted_lerobot_pusht.pdf (6 panels)
Smoke test (3 seeds, 60 epochs, ~3 minutes):
Retention Sweep — LeRobot PushT (5 methods, 6 fractions)
A broader sweep across six retention fractions (5%–100%) comparing Calibra vs. Random (5 seeds) and tracking action-space tail coverage. This is the simpler benchmark run before the targeted statistical study above.
Setup: 165 train / 41 test episodes. 5 random seeds. 120 epochs. Tail = bottom 15% of training episodes by action-space density (25 episodes).
| Keep | N | Calibra MSE | Rel | Tail cov | Random MSE ±std | Rel | Tail cov | Saved |
|---|---|---|---|---|---|---|---|---|
| 5% | 8 | 460.02 | 1.09x | 16.0% | 491.25 ±18.6 | 1.17x | 6.4% | 95% |
| 10% | 16 | 436.58 | 1.04x | 28.0% | 447.75 ±10.2 | 1.06x | 10.4% | 90% |
| 25% | 41 | 423.92 | 1.01x | 52.0% | 429.77 ±1.3 | 1.02x | 33.6% | 75% |
| 50% | 82 | 422.65 | 1.00x | 64.0% | 423.04 ±2.1 | 1.01x | 52.8% | 50% |
| 75% | 123 | 423.09 | 1.01x | 72.0% | 421.62 ±1.4 | 1.00x | 78.4% | 25% |
| 100% | 123* | 422.78 | 1.01x | 72.0% | 421.12 ±0.3 | 1.00x | 100.0% | 0% |
At 100%, Calibra's quality filter caps selection at 123 of 165 episodes. The 100% row reflects quality-filtered full dataset, not the unfiltered full dataset.
Rel = MSE / full-dataset MSE. Values ≤ 1.00 match or beat the full dataset.
python experiments/lerobot_coreset_benchmark.py \
--dataset lerobot/pusht \
--n-epochs 120 \
--n-seeds 5
xArm Lift Medium — Retention Sweep (800 episodes)
Dataset: lerobot/xarm_lift_medium (HuggingFace Hub)
Episodes: 800 total · 640 training · 160 test · Quality score: 82.7/100 (simulated)
Action/state: 4D (low-dimensional) · 25 frames/episode (short horizon)
Setup: 5 random seeds · 120 epochs BC-MLP · tail = bottom 15% by action-space density (96 episodes)
| Keep | N | Calibra MSE | Rel | Tail cov | Random MSE ±std | Rel | Tail cov | Saved |
|---|---|---|---|---|---|---|---|---|
| 5% | 32 | 0.397674 | 1.08x | 4.2% | 0.392948 ±0.0015 | 1.06x | 3.1% | 95% |
| 10% | 64 | 0.395813 | 1.07x | 9.4% | 0.385117 ±0.0007 | 1.04x | 9.6% | 90% |
| 25% | 160 | 0.380786 | 1.03x | 25.0% | 0.376361 ±0.0008 | 1.02x | 24.6% | 75% |
| 50% | 320 | 0.375223 | 1.02x | 57.3% | 0.372302 ±0.0007 | 1.01x | 50.2% | 50% |
| 75% | 480 | 0.371201 | 1.01x | 75.0% | 0.370517 ±0.0002 | 1.00x | 75.0% | 25% |
| 100% | 640 | 0.368871 | 1.00x | 100.0% | 0.369154 ±0.0001 | 1.00x | 100.0% | 0% |
Full baseline: MSE = 0.369041 · 23.2s train time
Key findings: Calibra does not outperform random selection on this dataset. At every retention fraction, Calibra is within 0.01x of random MSE — statistically indistinguishable. Rare-behavior coverage is also nearly identical (25.0% vs. 24.6% at 25%). This is the expected result: xArm lift medium has a quality score of 82.7 (cleaner than PushT/DROID-100) and a 4-dimensional action space with short, uniform episodes. In a low-dimensional uniform simulation, behavioral diversity is naturally spread across the action space — there is no clustering for the quality filter to remove and no rare-behavior gap for diversity selection to close.
This result is included without softening. It establishes that Calibra's advantage is conditional on dataset properties, not universal. The practical implication: run calibra audit first — if the quality score is high (>82) and the dataset is uniform simulation, the coreset benefit over random will be small.
python experiments/lerobot_coreset_benchmark.py \
--dataset lerobot/xarm_lift_medium \
--n-epochs 120 \
--n-seeds 5
DROID-100 — Retention Sweep
Dataset: lerobot/droid_100 (HuggingFace Hub)
Episodes: 100 total · 80 training · 20 test · Quality score: 77.0/100 (3 CRITICAL, 6 WARNING)
Setup: 5 random seeds · 120 epochs BC-MLP · tail = bottom 15% by action-space density (12 episodes)
| Keep | N | Calibra MSE | Rel | Tail cov | Random MSE ±std | Rel | Tail cov | Saved |
|---|---|---|---|---|---|---|---|---|
| 5% | 4 | 0.105998 | 1.16x | 25.0% | 0.101008 ±0.0040 | 1.11x | 6.7% | 95% |
| 10% | 8 | 0.103948 | 1.14x | 41.7% | 0.100802 ±0.0025 | 1.10x | 13.3% | 90% |
| 25% | 20 | 0.093743 | 1.03x | 83.3% | 0.103667 ±0.0047 | 1.13x | 28.3% | 75% |
| 50% | 40 | 0.096710 | 1.06x | 83.3% | 0.098831 ±0.0040 | 1.08x | 48.3% | 50% |
| 75% | 60 | 0.088590 | 0.97x | 83.3% | 0.094623 ±0.0021 | 1.04x | 71.7% | 25% |
| 100% | 74* | 0.089608 | 0.98x | 83.3% | 0.092498 ±0.0028 | 1.01x | 100.0% | 0% |
*At 100%, Calibra's quality filter caps selection at 74 of 80 training episodes.
Full baseline: MSE = 0.091364 · 29.9s train time
Key findings: At 75% retention (60 episodes), Calibra achieves 0.97× baseline MSE — it beats the full-data baseline by removing low-quality episodes. Random at 75% is still 4% worse than full. At 25% retention, Calibra (1.03×) is 10 percentage points better than random (1.13×) and retains 83% of rare behaviors vs. 28% for random — a 3× improvement at the same data budget. Even at 5% (4 episodes), Calibra's rare-behavior coverage is 3.7× better than random (25% vs. 6.7%).
python experiments/lerobot_coreset_benchmark.py \
--dataset lerobot/droid_100 \
--n-epochs 120 \
--n-seeds 5
ALOHA Sim Insertion — Retention Sweep
Dataset: lerobot/aloha_sim_insertion_human (HuggingFace Hub)
Episodes: 50 total · 40 training · 10 test · Quality score: 87.3/100 (2 CRITICAL, 2 WARNING)
Setup: 5 random seeds · 120 epochs BC-MLP · tail = bottom 15% by action-space density (6 episodes)
| Keep | N | Calibra MSE | Rel | Tail cov | Random MSE ±std | Rel | Tail cov | Saved |
|---|---|---|---|---|---|---|---|---|
| 5% | 2 | 0.008739 | 2.81x | 33.3% | 0.015407 ±0.0057 | 4.96x | 3.3% | 95% |
| 10% | 4 | 0.007749 | 2.50x | 33.3% | 0.008842 ±0.0031 | 2.85x | 3.3% | 90% |
| 25% | 10 | 0.005124 | 1.65x | 50.0% | 0.004186 ±0.0005 | 1.35x | 23.3% | 75% |
| 50% | 20 | 0.003533 | 1.14x | 83.3% | 0.003124 ±0.0005 | 1.01x | 36.7% | 50% |
| 75% | 30 | 0.003250 | 1.05x | 100.0% | 0.002733 ±0.0002 | 0.88x | 70.0% | 25% |
| 100% | 40 | 0.003035 | 0.98x | 100.0% | 0.003025 ±0.0004 | 0.97x | 100.0% | 0% |
Full baseline: MSE = 0.003104 · 26.7s train time
Key findings: With only 50 episodes and a high quality score (87.3), ALOHA sim has limited redundancy. Calibra's quality filter cannot identify a 25% subset that matches full performance — the dataset is already too clean and compact for large savings. Calibra consistently outperforms random at the 5% and 10% budgets (2.81x vs. 4.96x and 2.50x vs. 2.85x), and preserves rare behaviors more reliably at every fraction. The 50% budget shows the clearest story: Calibra 1.14x vs. Random 1.01x — random wins on MSE, but Calibra retains 83% of rare behaviors vs. 37%.
This is the expected behaviour: Calibra's data-volume savings are largest when datasets have high redundancy and quality issues (like PushT). On already-clean, small datasets, the benefit shifts to rare-behavior preservation rather than volume reduction.
python experiments/lerobot_coreset_benchmark.py \
--dataset lerobot/aloha_sim_insertion_human \
--n-epochs 120 \
--n-seeds 5
Synthetic Rare-Mode Benchmark (Controlled)
See README § "Rare-mode preservation" for the controlled 2D multigoal benchmark (worst-group success, 5-seed). Results: 98.7% worst-group (Calibra) vs. 36.5% (Random) at 5% retention.
Multi-Dataset Ablation (BC-MLP, ACT, Diffusion Policy)
See README § "Ablation study" and § "Cross-architecture check" for full tables across ALOHA Mobile, DROID-100, and PushT real at 30% retention with 5 seeds, 3 policy families, and 7 competing methods including K-Center, Facility Location, and Herding.
python experiments/ablation_benchmark.py --dataset lerobot/aloha_mobile_cabinet --seeds 5
python experiments/ablation_benchmark.py --dataset lerobot/droid_100 --seeds 5
python experiments/ablation_benchmark.py --dataset lerobot/columbia_cairlab_pusht_real --seeds 5
python experiments/aggregate_ablation.py results/ablation_*.json