Can I trust the numbers on my watch?
Heart rate mostly (within about ±3%); resting HRV agreed with ECG mainly for Polar and chest-strap devices; VO2max, calories and sleep are rough estimates. Only 11% of devices validated.
Doherty and colleagues set out to keep a running ("living") summary of how accurately consumer wearables measure health outcomes, to be updated every six months.
A search on 5 June 2024 found 24 systematic reviews (8 with a meta-analysis, published 2013-2024) that compared consumer devices with an accepted reference; after removing overlap they held 249 validation studies of consumer devices (430,465 participants, 43% women). They ranked the reviews by risk of bias and took each outcome's numbers from the best available review.
Heart rate: a Fitbit meta-analysis found a mean bias of −3.39 bpm (more under-reading in cycling), and a narrative review of 29 studies found errors of about ±3% whatever the brand. HRV agreed closely with ECG at rest (ICC 0.98-1.00 for Polar; 0.85-0.99 across devices) and less during exercise or movement.
Watch VO2max came from the one VO2max review (REF-1253): bias +2.17 mL/kg/min with limits of agreement −13.07 to 17.41 for resting-based estimates, −0.09 with limits −9.92 to 9.74 for exercise-based ones.
Energy expenditure was under-read by about 3 kcal/min (limits of agreement −13 to 7 kcal/min, Fitbit), and the broadest review's authors were "hesitant to consider any device sufficiently accurate". SpO2 differed from a pulse oximeter by up to 2.0% (Apple Watch Series 6 only). Devices over-read total sleep and under-read wake after sleep onset.
Only 34 of 310 consumer devices released since 2003 had been validated for any outcome, and only 3.5% of the device-outcome pairs. The authors conclude that accuracy "remains inherently indeterminate" and that composite scores (readiness, stress, "body battery") have never been formally validated.
Our reading: the most useful map the base has of which wearable readings agree with a reference and which do not — a reading the sensor measures at rest agrees best, an algorithm's estimate worst — but its numbers are borrowed from reviews of very different quality, several are misreported in the abstract, and none describes a 2026 device.
Protocol
Design: living umbrella review, reported to PRISMA; PROSPERO CRD42023402703 registered 18 March 2023; search updates every 6 months, versions posted on OSF.
Search: MEDLINE (PubMed), Embase, CINAHL and SPORTDiscus on 5 June 2024, no filters; strategies built with a librarian.
Eligibility: systematic reviews or meta-analyses of consumer wearables (any body-worn device available for public purchase, including discontinued ones) against an accepted reference standard; reviews of research-grade devices or of device-against-device agreement excluded.
Selection: two authors screened and extracted independently in Covidence, a third decided disagreements. 904 records, 92 duplicates, 812 screened, 41 full texts, 24 reviews included.
Evidence base: the 24 reviews held 653 studies; de-duplicated, 391 primary studies (888,033 participants), of which 249 were validation studies of consumer devices (n = 430,465; 243,068 men and 181,064 women; 80 studies did not report sex). One arrhythmia review alone counted 424,371 participants; the heart-rate evidence is 165 studies of 5816 participants.
Device census: 310 consumer devices released since 2003 by 20 manufacturers (among them Garmin 72, Fitbit 32, Polar 24, Huawei 24, Samsung 24, Apple 14, Suunto 13, Coros 8, Whoop 4, Oura 3); the most validated makers were Fitbit (31 studies), Apple (12) and Polar (6).
Review quality (Drucker tool, six domains): 2 high (8%), 10 moderate (42%), 12 low (50%), none critically low. The two high-quality reviews are a Fitbit-only meta-analysis and the VO2max meta-analysis (REF-1253).
Synthesis: statistics extracted were MAPE, pooled bias, ICC and mean absolute difference, with confidence intervals where given; when reviews overlapped, the higher-quality review took precedence. A meta-analysis was not attempted: of the 249 studies only 166 used an accepted gold-standard criterion and only 75 of those used appropriate statistics.
Funding and interests: Science Foundation Ireland National Challenge Fund (22/NCF/FD/10949); open-access funding through the IReL consortium; the authors declare no conflicts of interest. Not declared in this paper but printed in REF-1253's byline and funding statement: two authors (Argent, Caulfield) co-wrote that VO2max review, which the umbrella rated high quality and quotes, and which was partly funded by Huawei.
Key Results
| Outcome | Reviews (quality) and evidence | What the best available review found |
|---|---|---|
| Heart rate | 6 reviews (1 high, Fitbit only; 2 moderate; 3 low); 165 studies, 5816 participants | Fitbit meta-analysis of 52 studies: mean bias −3.39 bpm, more under-reading in cycling than in daily living, treadmill or overground walking; narrative review of 29 studies: errors of about ±3% regardless of device (Apple, Fitbit, Garmin) or criterion; Fitbit Charge 2: MAPE < 10% in 5 of 8 studies |
| HRV | 2 reviews (moderate, low), no meta-analysis; 22 studies, 714 participants | Polar devices at rest: ICC 0.98-1.00 ("near perfect validity"); a range of devices: ICC 0.85-0.99 at rest, falling to 0.85 with exercise or movement |
| Arrhythmia | 6 reviews (2 moderate, 4 low); 85 studies | Smartwatches (mostly Apple, Samsung) in 424,371 people: sensitivity 100% (95% CI 0.99-1.00), specificity 95% (0.93-0.97); mostly people already diagnosed |
| VO2max | 1 review (high); 14 studies, 403 participants | Resting-based: bias +2.17 mL/kg/min (95% CI 0.28-4.07), limits of agreement −13.07 to 17.41; exercise-based: bias −0.09 (−1.66 to 1.48), limits −9.92 to 9.74 |
| SpO2 | 1 review (low); 5 studies, 973 participants, Apple Watch Series 6 only | r 0.76-0.89 with pulse oximeters; mean absolute difference up to 2.0%; limits of agreement as wide as −5.8% to +5.9% in some studies |
| Steps | 8 reviews (1 high, Fitbit only; 3 moderate; 4 low); 184 studies, 6197 participants | Fitbit −3.11 steps/min (range −13 to 7); wrist devices mean −9% (median −2%); Fitbit Charge 2 MAPE 12% |
| Activity intensity | 4 reviews (1 moderate, 3 low), no meta-analysis; 126 studies, 4087 participants | Over 80% of Fitbit comparisons erred by more than 10% for light-to-vigorous time; moderate-to-vigorous minutes over-counted by up to 89.8 min/day; other devices MAPE 29-80% |
| Energy expenditure | 8 reviews (1 high, 4 moderate, 3 low; 3 meta-analyses); 218 studies, 7734 participants | Fitbit under-read about 3 kcal/min (limits of agreement −13 to 7 kcal/min) or −3%; 64-study meta-analysis across brands: effect size −0.23 (95% CI −0.44 to −0.03), primary-study error −21.27 to 14.76% |
| Sleep | 4 reviews (2 moderate, 2 low); 95 studies, 2985 participants | Total sleep over-read and wake after sleep onset under-read (mean absolute difference about 22.0 min/day, ICC 0.85, 4 studies); Fitbit over-read total sleep and sleep efficiency, often by more than 10% (3 studies); latency and wake errors 12-180% |
Validation coverage: 34 of 310 devices (11%) validated for at least one outcome; assuming five outcomes per device, 54 of 1550 possible device-outcome validations (3.5%) exist.
Composite scores: the authors state that readiness, stress and "body battery" scores from Fitbit, Garmin, Oura and Whoop have not undergone formal validation.
Numbers that do not add up (our reading):
The abstract says wearables "significantly overestimated VO2max by ± 15.24% during resting tests and ± 9.83% during exercise tests". The Results give ±15.24 and ±9.83 as half-widths of limits of agreement in mL/kg/min, and the exercise-based bias was −0.09 (not an overestimate). A 15% figure for watch VO2max should not be quoted from this paper.
The abstract's energy-expenditure "mean bias = − 3 kcal per minute, or − 3%" joins two reviews (a kcal/min figure from one Fitbit review, a percentage from another). At a typical exercise expenditure of about 10 kcal/min, −3 kcal/min would be about −30%, not −3% (our arithmetic).
The abstract gives step-count "mean absolute percentage errors ranging from − 9 to 12%"; −9% is a mean signed error and 12% a MAPE from another review — a MAPE cannot be negative.
The abstract's sleep "mean absolute percentage error typically > 10%" is, in the body, "often by more than 10%" from a Fitbit review of three studies.
The Discussion says 71% of primary studies used a gold-standard criterion and 40% of those used appropriate statistics; the Results' counts give 166 of 249 (67%) and 75 of 166 (45%) (our arithmetic).
Mechanisms
Measured: agreement depended on the outcome and the condition — heart rate was under-read more during cycling than walking (Fitbit subgroups), HRV agreement fell from rest to movement, and activity-intensity error depended on the intensity.
Authors' interpretation: error comes from differing proprietary algorithms, user factors (age, body size, skin tone), placement and wear time, temperature and humidity, and above all continual hardware, firmware and algorithm updates, so any validation is a snapshot that ages within a product cycle.
Inference: the pattern fits the hierarchy the base already holds as a narrative claim (REF-397, C+): a quantity the sensor measures directly at rest (resting heart rate, resting HRV) agrees best; quantities an algorithm infers (VO2max, energy expenditure, activity intensity, sleep versus quiet wake) agree worst; composites built from those estimates are untested. This review gives that hierarchy a systematic, if heterogeneous, footing.
Inference: a mean bias near zero (heart rate ±3%, exercise-based VO2max −0.09) says little about one athlete's reading; where limits of agreement were reported (VO2max ±10 to ±15 mL/kg/min, energy expenditure −13 to +7 kcal/min), the individual error is large relative to the changes athletes look for.
Practical Application
Can I trust the numbers on my watch? Heart rate mostly (within about ±3%); resting HRV agreed with ECG mainly for Polar and chest-strap devices; VO2max, calories and sleep are rough estimates. Only 11% of devices validated.
Rank a reading before acting on it — measured at rest, then measured during exercise, then an algorithm's estimate, then a composite score (Not from this paper; the base's own practice, consistent with its findings and REF-397). Label the reading measured or estimated, as SYNTH-38 already asks for heart-rate anchors.
Watch VO2max is a range, not a number: limits of agreement of about ±10 mL/kg/min for exercise-based and −13 to +17 for resting-based estimates (REF-1253, A-; restated here). This gives SYNTH-78's line "The number on a watch or head unit is an estimate of the same kind and deserves the same caution" its first source.
Do not do fuelling arithmetic from watch calories: the best-quality review found Fitbit under-reading by about 3 kcal/min with limits of −13 to +7 kcal/min, and the broadest one would not call any device accurate enough; REF-1254 found an Apple Watch Series 6 and a Garmin Fenix 6 over-reading walking and running by about 17-29% on average. This addresses SYNTH-87's stated gap ("Nothing here tests how accurate app or watch estimates of exercise expenditure are").
Accuracy of one model is not the brand's: the evidence here ends with devices studied by 2022; check the model and, where possible, the year of the validation before quoting a figure.
Limitations
No pooling. Every number is a single included review's number, chosen by that review's quality, so each carries its source's strength: heart rate rests on a Fitbit-only meta-analysis and a narrative review, SpO2 on a low-quality review of one Apple Watch model.
Review and study quality. Half the reviews were rated low quality; of the 249 primary studies only 166 used a gold-standard criterion and only 75 of those used appropriate agreement statistics.
Out of date by design. The most recent primary study was published in August 2022; the authors note three new Apple Watch iterations since then and that every device analysed has been retired or superseded.
Who. Mostly healthy adults and patients; athletes, intense exercise and sport-specific conditions are not analysed separately; Fitbit dominates three outcomes.
Reporting. The abstract misstates the VO2max limits as percentages and as overestimation, merges two energy-expenditure figures, and calls a signed error a MAPE; Discussion percentages do not match Results counts.
Interests. Science Foundation Ireland funding and no declared conflicts; but two authors co-wrote the VO2max review that this umbrella rated as one of its two high-quality reviews, and that review was partly funded by Huawei (REF-1253). Rating one's own review is a soft spot in the quality grading (our reading).
Not read by us: the supplementary files with device-level data and per-review risk-of-bias tables.
How this entry was checked and changed
Why this level
an umbrella review, which the rubric places at A+, held at A-: the authors could pool nothing ("pervasive heterogeneity" in protocols, criteria, devices, firmware and outcomes; fewer than five comparable studies for any device and outcome), only 2 of the 24 reviews were rated high quality and 12 low, each headline number is one included review's number (the SpO2 figure comes from one low-quality review of a single watch), and the primary studies are small single-device validations, most recent August 2022. Read from the full text (open access, CC BY 4.0); the supplementary device tables were not read
Full text (checked 2026-10-27)
24 systematic reviews met our inclusion criteria; these systematic reviews included 249 non-duplicate validation studies of consumer wearable devices involving 430,465 participants (43% female) (full text)
Of these 310 devices, 34 (11%) were validated for at least one biometric outcome; the 54 validity studies conducted represent only 3.5% of the 1550 potential validations (full text)
Only two reviews (8%) showed a high quality of evaluation; 10 (42%) were of moderate quality, and a further 12 (50%) were deemed to be of low quality (full text)
Heart rate: 165 non-duplicate studies of 5816 participants; Fitbit pooled mean bias of − 3.39 bpm; wrist-worn wearables typically exhibit measurement errors of approximately ± 3% (full text)
HRV: "near perfect validity" (ICC 0.98–1.00) for Polar devices at rest; ICCs ranging from 0.85 to 0.99 at rest, decreasing to 0.85 as exercise and/or motion increased (full text)
VO2max: resting test bias = 2.17 ml kg−1 min−1 (95% CI 0.28–4.07), limits − 13.07 to 17.41 (i.e. ± 15.24); exercise tests bias = − 0.09 (95% CI − 1.66 to 1.48), limits − 9.92 to 9.74 (i.e. ± 9.83) (full text)
Energy expenditure: underestimate by approximately 3 kcal per min (limits of agreement − 13 to 7 kcal per min); across devices ES − 0.23, 95% CI − 0.44 to − 0.03, error ranging from − 21.27 to 14.76% (full text)
SpO2 (Apple Watch Series 6 only): mean absolute differences up to 2.0%; some studies showing limits of agreement of − 5.8% to + 5.9% (full text)
Sleep: overestimate TST while underestimating wakefulness after sleep onset (mean absolute difference of approximately 22.0 min/day and ICC 0.85) (full text)
Of the 249 studies, only 166 used an accepted gold standard criterion, and only 75 used appropriate statistical analysis (full text)
Funded by Science Foundation Ireland's National Challenge Fund (Grant ID: 22/NCF/FD/10949); the authors have no conflicts of interest with the content of this article (full text)