Application of Open-Source Step Counting Algorithms on Publicly-Available Data

Statistical methods in digital health research

Lily Koffman

JHU Biostatistics

3/9/24

Background

• Steps: easily translatable metric of physical activity
• Goal: estimate steps with wearable accelerometery
• Transition from hip/waist accelerometers \(\rightarrow\) wrist-worn devices
  • Better adherence¹
  • Harder to estimate steps

The landscape

• Proprietary algorithms (Apple, FitBit, AcitGraph, Garmin)
• Open-source algorithms
• Lack of publicly-available data with ground truth step counts; especially free-living data
• Mean absolute percent error reported \(>20\%\) in free-living²
• Step counts differ widely between devices on same individual³

Existing open-source algorithms

• Adaptive empirical pattern transformation (ADEPT)⁴
  • Pattern matching with pre-specified stride templates
• Oak⁵
  • Continuous wavelet transform
• \(\texttt{stepcount}\)⁶
  • Hybrid machine-learning and peak finding
  • SSL and RF
• Step Detection Threshold (SDT)⁷
  • Peak finding
• Verisense⁸
  • Peak finding with constraints
• ActiLife\(^*\)⁹

Publicly available datasets with ground truth step counts

• Clemson Pedometer Evaluation Project¹⁰
  • 30 participants; 15 Hz
• Movement Analysis in Real-world Environments using Accelerometers (MAREA) Gait Database¹¹
  • 20 participants; 128 Hz
• OxWalk¹²
  • 39 participants; 25 and 100 Hz

Data

• Clemson
• MAREA
• OxWalk

Methods

• Resample all data to 30 Hz (for fair comparison)
• Fit all algorithms on raw data
• Evaluate:
  • Walking recognition (F1 score, precision, recall)
  • Step count (absolute percent error, bias)
  • Sensitivity to sampling rate
• All code for this analysis at https://github.com/lilykoff/step_algorithms
• https://github.com/muschellij2/walking and https://github.com/jhuwit/stepcount R packages implement all algorithms

Results: walking recognition

Mean (SD) F1 Score
Algorithm	Clemson	MAREA	OxWalk	Overall
ActiLife	0.82 (0.03)	0.96 (0.03)	0.56 (0.23)	0.73 (0.22)
ADEPT	0.68 (0.08)	0.75 (0.18)	0.44 (0.23)	0.59 (0.22)
Oak	0.81 (0.11)	0.69 (0.25)	0.66 (0.21)	0.72 (0.20)
stepcount RF	0.91 (0.02)	0.99 (0.01)	0.75 (0.20)	0.86 (0.17)
stepcount SSL	0.91 (0.02)	0.99 (0.01)	0.83 (0.14)	0.89 (0.11)
SDT	0.78 (0.04)	0.99 (0.01)	0.54 (0.25)	0.72 (0.24)
Verisense original	0.86 (0.04)	0.92 (0.05)	0.64 (0.22)	0.78 (0.20)
Verisense revised	0.87 (0.04)	0.97 (0.03)	0.66 (0.22)	0.80 (0.20)

Results: walking recognition

Results: step counts

Algorithm	Clemson	MAREA	OxWalk	Overall
Absolute percent error mean (SD)
ActiLife	20.2 (11)	39.0 (13)	51.5 (77)	38.1 (53)
ADEPT	43.4 (9)	36.2 (23)	61.4 (16)	49.8 (19)
Oak	14.6 (15)	42.5 (28)	31.8 (46)	28.3 (36)
stepcount RF	5.9 (6)	12.9 (10)	29.9 (63)	18.0 (43)
stepcount SSL	5.5 (6)	11.4 (10)	9.6 (10)	8.6 (9)
SDT	38.5 (12)	16.1 (11)	228.6 (252)	117.9 (194)
Verisense original	21.3 (10)	30.6 (15)	28.5 (42)	26.5 (29)
Verisense revised	20.4 (10)	19.2 (11)	28.0 (38)	23.5 (27)
Bias (predicted steps - true steps) mean (SD)
ActiLife	-379 (222)	-633 (514)	-222 (653)	-364 (531)
ADEPT	-819 (173)	-652 (614)	-831 (777)	-788 (597)
Oak	-262 (310)	-742 (730)	-42 (336)	-268 (514)
stepcount RF	-32 (155)	-232 (277)	116 (263)	-10 (269)
stepcount SSL	7 (151)	-210 (262)	-45 (90)	-63 (179)
SDT	720 (205)	-257 (273)	1110 (618)	682 (687)
Verisense original	-408 (195)	-491 (445)	-229 (413)	-346 (374)
Verisense revised	-388 (194)	-333 (324)	-248 (388)	-314 (323)

Results: step counts

• Clemson
• MAREA
• OxWalk

Sensitivity to sample rate

• stepcount
• Verisense

Sensitivity to speed

Takeaways

• Strength of machine-learning based algorithms
• Need more free-living data with ground truth step counts from more diverse set of individuals
  
• Rethinking “gold standard”: simultaneous wrist and hip/thigh accelerometry?

Acknowledgments

• Drs. Khandelwal and Wickstrom at Intelligent Systems Lab, Halmstad Unversity for sharing the MAREA Gait Database
• Ryan Mattfeld, Elliot Jesch, and Adam Hoover of the Clemson University Holcombe Department of Electrical and Computer Engineering for making the Clemson Ped-Eval data publicly available
• Scott Small, Aidan Acquah, Sara Khalid, and Andrew Price from University of Oxford Nuffield Department of Population Health and Lennart von Fritsch from Heidelberg University for making the OxWalk data publicly available
• Authors of ADEPT, Oak, SDT, \(\texttt{stepcount}\), and Verisense for making their algorithms open source

Footnotes

1. Richard P. Troiano, James J. McClain, Robert J. Brychta, and Kong Y. Chen. Evolution of accelerometer methods for physical activity research. British Journal of Sports Medicine, 48(13):1019–1023, July 2014

2. Ivar Holm, Jonatan Fridolfsson, Mats Börjesson, and Daniel Arvidsson. Fourteen days free-living evaluation of an opensource algorithm for counting steps in healthy adults with a large variation in physical activity level. BMC Biomedical Engineering, 5(1), April 2023.

3. Lindsay P. Toth, Susan Park, Cary M. Springer, Mckenzie D. Feyerabend, Jeremy A. Steeves, and David R. Bassett. Video-Recorded Validation of Wearable Step Counters under Free-living Conditions. Medicine & Science in Sports & Exercise, 50(6):1315, June 2018

4. Karas, Marta, Straczkiewicz, Marcin, Fadel, William, Harezlak, Jaroslaw, Crainiceanu, Ciprian, Urbanek, and Jacek. Adaptive empirical pattern transformation (ADEPT) with application to walking stride segmentation. \(\textit{Biostatistics}\),2019.

5. Marcin Straczkiewicz, Emily J Huang, and Jukka-Pekka Onnela. A ``one-size-fits-most’’ walking recognition method for smartphones, smartwatches, and wearable accelerometers. \(\textit{NPJ Digital Medicine}\), 6(1):29, 2023

6. Scott R. Small, Shing Chan, Rosemary Walmsley, Lennart von Fritsch, Aidan Acquah, Gert Mertes, Benjamin G. Feakins, Andrew Creagh, Adam Strange, Charles E. Matthews, David A. Clifton, Andrew J. Price, Sara Khalid, Derrick Bennett, and Aiden Doherty. Development and Validation of a Machine Learning Wrist-worn Step Detection Algorithm with Deployment in the UK Biobank, February 2023.

7. Scott W. Ducharme, Jongil Lim, Michael A. Busa, Elroy J. Aguiar, Christopher C. Moore, John M. Schuna, Tiago V. Barreira, John Staudenmayer, Stuart R. Chipkin, and Catrine Tudor-Locke. A Transparent Method for Step Detection using an Acceleration Threshold. \(\textit{Journal for the Measurement of Physical Behaviour}\), 4(4):311–320, December 2021.

8. Benjamin D. Maylor, Charlotte L. Edwardson, Paddy C. Dempsey, Matthew R. Patterson, Tatiana Plekhanova, Tom Yates, and Alex V. Rowlands. Stepping towards More Intuitive Physical Activity Metrics with Wrist-Worn Accelerometry: Validity of an Open-Source Step-Count Algorithm. \(\textit{Sensors}\), 22(24):9984, January 2022

9. ActiGraph’s proprietary algorithm, *not open-source

10. https://cecas.clemson.edu/~ahoover/pedometer/

11. https://wiki.hh.se/caisr/index.php/Gait_database

12. https://ora.ox.ac.uk/objects/uuid:19d3cb34-e2b3-4177-91b6-1bad0e0163e7