Trajlens: a validator for LeRobotDataset, audited 100 Hub datasets

🤗Datasets
1

Posting here because this is directly about Hub data quality.

LeRobotDataset is now the largest dataset category on the Hub. I built trajlens, an open-source linter for it, and ran it against 100 random public lerobot-tagged datasets to measure real-world quality.

Results: 19 PASS, 13 FAIL, 47 load errors (mostly the known v2.x streaming limitation, not trajlens bugs), 21 timeout.

Of the datasets that linted successfully, two named upstream bugs account for most of the failures:

lerobot issue 2401 (v2.1 to v3.0 episode-frame corruption): roughly 19% prevalence
lerobot issue 3177 (timestamp float drift): roughly 3% prevalence

If you maintain a dataset on the Hub, running trajlens lint your-org/your-dataset takes under a minute and will tell you if either of these applies to your data.

pip install trajlens

Repo: GitHub - Kunal-Somani/trajlens: The quality and synthesis layer for the open robot-learning data ecosystem · GitHub
Apache 2.0, pluggable check registry, contributions welcome.

trajlens_linkedin_card
trajlens_linkedin_card1600×900 52.8 KB

2

19% prevalence of the episode-frame corruption bug across 100 random datasets is a striking number for something that is silent until someone runs an integrity check like this.

Question on scope: does trajlens check for instruction-string consistency, the task or language field matching what the dataset’s actual episode is showing, or is it purely structural? Asking because I have seen an instruction-corruption bug in a different robot benchmark, the LIBERO-Plus issue #64, that a purely structural linter would pass cleanly, since the frames are fine and only the language field is corrupted from source metadata leaking into it.

If that is out of scope today, it might be worth a check-registry entry given how easy it is to miss by eye and how much it can skew a reported success rate.