Real-time Human Pose Estimation for Gym Equipments

The following video consists of two parts of my work on human pose estimation:

• First, a 2D version for treadmill: keypoint detection of the lower body in real-time.
• Second, a 3D version for gym mirror: pose estimation with a RGB camera on the gym mirror.

2D Human Keypoint Detection

Recently, the lite versions of some state-of-the-arts have been proposed for real-time 2D human keypoint detection, e.g., OpenPose, HRNet and so on. To meet the requirments of the treadmill product, I made efforts in deploying my lite model on the development board which is a ARM-based platform with the Android OS. My work can be listed as follows:

• Performance: As the detection of foot keypoints are imperative but difficult, I design a self-attention connected module (SCM) which serves as a plug-and-play module. With encoding the connected intensities among keypoint features, SCM can promote the features of different keypoint coordinates to be more distinguishable. As a result, the detection of foot keypoints is more accurate.
• Speed: Including the cost of rendering time on Android, the improved lite model can execute in real-time on the development board with a 1.2TOPS NPU. As to the instability caused by pruning, I further take the Kalman filter as the post-processing for stabilizing the outputs.
• Auxiliary Functions: The lite model has been integrated in the treadmill product (G21), and some auxiliary functions, such as unbalanced pose detection and statistical analysis of running data, are also developed. The above functions have TWO related patents.

3D Human Pose Estimation

Owing to the potential application of gym mirror (i.e., fitness mirror), I extend the lite model from a half-body prediction to a whole-body prediction (19 keypoints) to obtain a set of 2D human keypoints. Moreover, I apply the GCNs on the 2D keypoints to lifting the dimension from 2D to 3D. More details of the 3D performance can be referred in the above video, which includes three distinct fitness actions.