Status: this is not a dataset yet
The archive behind this page is engineering test data produced while the capture system was being built — recorder bring-up, bench runs, and automatic boot recordings that nobody intended to keep. Roughly half of the sensor files are empty stubs, the largest single file is a recorder that was left running for hours, and only a handful of takes carry a campaign label. Nothing here should be quoted as a collection result. What this page establishes is the catalogue and the quality gate that the real campaign will be measured against.
What the dataset will be
Human work motion captured as a by-product of real work. A worker wears the FRT exoskeleton and performs their normal task; the exoskeleton records its own joint encoders, body-mounted IMUs, foot-pressure insoles and chest/belt RGB-D cameras. No robot is occupied, no dedicated demonstrator is hired, and no separate capture studio is needed — which is what separates this from teleoperation-based demonstration collection.
The recordings feed a retargeting pipeline (human pose → humanoid joint trajectories) and are packed into LeRobot datasets for imitation-learning policies (ACT / diffusion / GR00T fine-tuning).
Unit of organisation: the episode
One episode is one recording take. Physically it is one sensor CSV plus zero or more camera clips
(_rgb.mp4 / _depth.mp4, one pair per camera). The camera clip does not carry the same
timestamp as the CSV — the camera starts when it is ready, and since the wall-clock retiming fix its stamp can
drift further — so clips are matched to a CSV by nearest start stamp within 180 s, with the same activity
label preferred over a merely closer stamp. Clips that match no CSV are listed as video only episodes
rather than being hidden.
Sensor schema
Three CSV schemas exist. All of them start with the same first 41 columns, so a legacy parser reads a developer-mode file without changes.
| Schema | Columns | When |
|---|---|---|
| legacy | 41 | Default recording mode. Encoders + IMU accel/gyro + FSR. |
| developer | 57 | Recorded with “use developer options” ticked: adds 16 fused-quaternion
columns after src_ts, plus raw 16-bit depth PNGs and per-frame sync sidecars.
Pipeline data must be recorded in this mode. |
| refined | 44 | Post-processed output of refine_session.py: resampled to a
fixed 30 Hz, quaternions converted to per-IMU relative Euler angles. |
| Column | Unit | Meaning |
|---|---|---|
| time | ISO 8601 | Host wall-clock of the sample. |
| elapsed_s | s | Seconds since recording start. Duration and rate are derived from this. |
| src_ts | epoch s | Source timestamp — the clock shared with the camera sidecar
_frames.csv, i.e. the column to use when aligning video to sensors. |
| label | text | Task phase chosen by the recorder (see taxonomy below). |
| weight_kg | kg | Payload entered by the operator for that take (0 if not entered). |
| pos_l / pos_r | rad | Absolute hip joint encoder position, left / right. |
| torque_l / torque_r | N·m | Actuator torque, left / right. |
| imu1..4_ax/ay/az | m/s² | Linear acceleration per IMU. Mounting: imu1 = upper trunk (back), imu2 = lower spine, imu3 = left, imu4 = right. |
| imu1..4_gx/gy/gz | rad/s | Angular rate per IMU. |
| imu1..4_qi/qj/qk/qr | — | Fused quaternion (developer mode only), Adafruit i,j,k,real order. All-zero means that IMU was never read. |
| fsr_l1..l4 / fsr_r1..r4 | raw ADC | Foot-pressure insole zones. Raw scale differs between feet by roughly 3× — normalise per foot before comparing. |
Not every unit carries every sensor. Units differ in installed IMU count and in whether insoles are fitted, so a column being all-zero is only a fault if that unit normally produces data on it. The quality scan below learns each unit's own expected sensor set from its own recordings instead of assuming a fixed layout.
Camera streams
Each camera writes <stamp>_<serial>_rgb.mp4 and …_depth.mp4. Serial
numbers are mapped to stable “Camera 1 / Camera 2” names by sorting all serials seen, the same order the device
dashboard uses. The depth .mp4 is a colourised preview and is not metric — for metric depth use
the raw 16-bit PNG folder <base>_depth_raw/ produced in developer mode. Those PNGs, the
_frames.csv sync sidecar and _meta.json (depth scale, colour intrinsics) are kept on the
device and are not part of this cloud archive.
Label taxonomy
Labels are task phases, not object classes. The campaign set is:
| Label | Covers |
|---|---|
| grabbing | Approach and pick, including the steps taken to reach the object. |
| moving | Loaded transport of the object. |
| releasing | Place and let go, including the steps to the drop point. |
| walking | Unloaded locomotion. |
| standing | Stationary, used for neutral-pose calibration takes. |
| unlabeled | No label was chosen — mostly automatic boot recordings. Not campaign data. |
Quality assurance (DQA)
The quality scan reads each episode's CSV in the browser and runs the checks below. Nothing is computed server-side and no file is modified; grades are cached in this browser only.
| Check | Verdict |
|---|---|
| sensor present | fail if every expected IMU is silent. warn if an individual expected channel group is all-zero (dead) or never changes (stuck). |
| sample rate | fail below 10 Hz, warn below 25 Hz. Computed from
elapsed_s deltas. |
| timing gaps | warn if any gap between consecutive samples exceeds 0.5 s. |
| duration | fail under 2 s or under 30 rows — these are aborted or boot-artefact takes. |
| completeness | fail if the sensor CSV is missing entirely; warn if no camera clip was paired with it. |
| time-sync (video) | Checked when an episode is opened: the browser reads the clip's real duration and compares it to the CSV duration. warn above 1.5 s difference. |
| playable video | warn if a clip fails to decode — interrupted recordings can be written
without their moov atom. |
Usable ratio = episodes graded pass ÷ episodes graded. It is reported over the scanned subset only, never extrapolated.
Access
This page is read-only and reads the archive directly with a scoped, read-and-list-only signature. To pull the same files from a terminal:
loading…
Privacy and de-identification
Recordings contain a worker's body motion and on-site video. Published and shared copies are to be the de-identified versions (faces blurred, original and de-identified copies stored separately). The archive behind this page is the working store and is access-controlled; it is not the publication channel.
Known limitations
- Camera extrinsics are not yet calibrated, so camera-frame data cannot be fused metrically with body frames.
- No gripper open/close signal is recorded — hand state must be inferred from video.
- Fused quaternions are only present in developer-mode recordings; some units report a hard-coded identity quaternion because they carry a 6-axis IMU with no on-board fusion.
- Recordings made while the device clock was unset appear with a 1970 timestamp.
- Insole hardware is not fitted on every unit; where it is absent the FSR columns are legitimately zero.