Technical Report · September 2026
Figure 1. Output of our production setup on three workplace recordings (packing, two sessions; ironing, one session). Top: the camera view with the estimated hand joints. Bottom: the recovered 3D hands on a 50 mm grid. The production setup differs from the configurations measured in Section 4; these recordings have no ground truth and are not part of the measurements.
Recording human hands in 3D is a bottleneck for learning manipulation from human demonstrations. We present BYND-3D, a system that recovers both hands in metric 3D, 21 joints per hand (also exported in Meta's 25-joint hand layout), from video captured by a single head-mounted camera. We compare it with three recent methods, ACE-Ego-Hand, HandFlow and EgoForce, which we ran from their released code and weights and scored with the same code, on two test sets built from the public HOT3D dataset: EgoForce's released validation protocol (Project Aria glasses), and a Quest 3 protocol we defined on four participants held out from all of our development and scored once. On the validation protocol BYND-3D's mean 3D joint error is 25.6 mm, against 31.3, 35.0 and 47.2 mm. On the Quest 3 protocol it is 44.0 mm, against 117.2, 70.2 and 58.5 mm. The two results use two BYND-3D configurations, each fixed before its test was scored. Under the comparison rules fixed before scoring, all six of these differences are statistically clear over recording sequences; the smallest, 5.7 mm against ACE-Ego-Hand on the validation protocol, is not clear after correcting for our three scorings of that data (Section 5). The Quest 3 configuration is also clearly lower than all three methods on the validation protocol (26.8 mm), although its 4.5 mm margin over ACE-Ego-Hand is likewise not clear after that correction; the validation configuration scores 53.9 mm on Quest 3, lower than EgoForce but not clearly. A multi-camera version of BYND-3D is in development.
Robot learning from human video needs the 3D position of each hand, in metric units, at every frame. A single head-mounted camera is the cheapest and least intrusive way to record such video, but it leaves depth ambiguous, sees hands at the distorted edge of wide-angle lenses, and loses hands when they leave the view or are hidden by objects.
BYND-3D is built for this setting. This report measures it against three recent research methods on two test sets built from the public HOT3D dataset [1], under rules chosen to limit selective reporting: every scored hand counts, a missed hand is charged as an error, all methods are scored by the same code, and the comparison rule and the wording of each possible outcome were fixed before the final scoring of each test.
BYND-3D takes video from one head-mounted camera, including wide-angle lenses, and outputs, for each frame, the 3D positions of 21 joints for each hand it finds, in millimetres in the camera's coordinate frame, together with a hand mesh. It also exports each hand in Meta's 25-joint layout: the same 21 joints plus four metacarpal joints, which are derived from the other 21. All measurements in this report use the 21 standard joints. It is offline: its estimate for a frame also uses later frames of the same video. This report describes results, not the method.
Table 1. Mean 3D joint error (mm, lower is better), all scored hands, misses charged. Headline results in bold; the other BYND-3D cell of each column is that test's pre-registered secondary.
| Method | EgoForce HOT3D val | Our Quest 3 protocol |
|---|---|---|
| BYND-3D (A) | 25.6 | 53.9 |
| BYND-3D (B) | 26.8 | 44.0 |
| ACE-Ego-Hand [3] | 31.3 | 117.2 |
| HandFlow [4] | 35.0 | 70.2 |
| EgoForce [2] | 47.2 | 58.5 |
Figure 2. Mean 3D joint error on the two test sets. Of the four methods we ran, BYND-3D has the lowest error on both. The Quest 3 protocol is our own, and EgoForce receives hand boxes derived from the ground truth.
On the validation protocol configuration A's error is 25.6 mm: 5.7 mm lower than ACE-Ego-Hand, 9.4 mm lower than HandFlow and 21.6 mm lower than EgoForce (Table 2, Figure 3). It is lower on 23, 23 and 30 of the 31 sequences, and each difference stays below zero when any one of the four participants is left out. Configuration B scores 26.8 mm there and is also clearly lower than all three methods.
On our Quest 3 protocol every method has a higher error than on validation; the camera, lens, participants and scored hands all differ, so the two columns are not directly comparable. Configuration B's error is 44.0 mm: 14.5 mm lower than EgoForce, 26.2 mm lower than HandFlow and 73.2 mm lower than ACE-Ego-Hand, lower on 47, 52 and 56 of the 57 sequences, and each difference holds with any one participant left out. Much of ACE-Ego-Hand's gap is depth scale: it places Quest 3 hands too far from the camera (by about 30% on our development participants and, in a check made after scoring, about 21% on the test). Configuration A scores 53.9 mm on Quest 3; its difference to EgoForce, −4.6 mm [−11.8, +2.8], is not statistically clear.
Table 2. BYND-3D's error minus each method's (mm), with 95% paired bootstrap intervals over sequences and the number of sequences where BYND-3D is lower. Negative favours BYND-3D. Every interval shown lies below zero. Configuration A on Quest 3, that test's pre-registered secondary, is not shown: against EgoForce its difference is −4.59 [−11.83, +2.84] · 36/57, not clear; its differences to ACE-Ego-Hand and HandFlow were not pre-registered.
| vs | A on EgoForce HOT3D val (headline) | B on EgoForce HOT3D val (secondary) | B on our Quest 3 protocol (headline) |
|---|---|---|---|
| ACE-Ego-Hand | −5.73 [−10.98, −0.94] · 23/31 | −4.53 [−8.96, −0.15] · 21/31 | −73.16 [−82.73, −64.00] · 56/57 |
| HandFlow | −9.37 [−12.99, −5.81] · 23/31 | −8.18 [−12.23, −4.15] · 20/31 | −26.22 [−34.20, −19.22] · 52/57 |
| EgoForce | −21.61 [−27.08, −16.33] · 30/31 | −20.41 [−25.47, −15.41] · 28/31 | −14.54 [−20.70, −8.23] · 47/57 |
Figure 3. How much lower BYND-3D's error is than each method's, for the headline configuration of each test, with 95% paired bootstrap intervals over recording sequences. Every interval lies above zero, the pre-registered criterion for a clear difference. Against ACE-Ego-Hand on the validation protocol the interval reaches zero after correcting for our three scorings of that data ([−0.03, 12.3] mm) and when participants are resampled (Section 5). The two panels use different scales.
Figure 4. Mean 3D joint error per participant, for the headline configuration of each test. BYND-3D is not the lowest for every person: on validation HandFlow is lower for P0002 and ACE-Ego-Hand for P0009, and BYND-3D and ACE-Ego-Hand are level for P0002 (23.2 and 23.4 mm); on Quest 3, BYND-3D and EgoForce are close for P0013 (52.3 and 54.5 mm). P0001 contributes 16 of the 31 validation sequences.
A single view can only estimate depth. We are extending BYND-3D to head-mounted rigs with six or more cameras and will evaluate that version on new multi-camera data.