Full Body Pose Estimation for Sports Analysis - Skeleton Calibration

Full Body Pose Estimation for Sports Analysis on NVIDIA ®Jetson™
About
Camera Abstraction
Camera Calibration AprilTag Camera Calibration Chessboard Camera Calibration 3D Pose Estimation Pose Representation Skeleton Calibration Temporal Filter Kinematic Fitting Pose Optimization Pose Alignment Pose Correction Performance
Introduction Required Hardware How to Get the Code How to Install the Library How to Use the Library
Getting Started
Contact Us

Introduction

The skeleton calibration module is intended to obtain the lengths of the limbs of the person whose pose is going to be estimated. This process is necessary in order to apply the kinematic fitting to a calibrated skeleton, which will adjust the pose prediction to the user body.

Figure 1: Skeleton calibration module location in general workflow.

Calibration Process

Figure 2: Flow diagram of calibration process.

The calibration process is computed using the 3D positions of the skeleton joints. We can get the limb lengths by using the joints positions and the child-to-parent relationships that are described in the skeleton structure. These lengths are obtained by computing the Euclidean distance between the two 3D positions. Once we have all the limb measures for a given sample, a mean of them is calculated and compared with the one from the last sample.

When the calibration is ready, the computed lengths will be assigned to the skeleton structure provided. The calibrator allows to set different pairs of limbs as symmetric, which means that they will be assigned the same length. This is done by computing the mean length of the two limbs and then assigning the result to both of them.

Stop Conditions

The calibration process will be ready once one of these two conditions is met:

• The difference between the current and last sample's lengths mean is less than a threshold [math]\displaystyle{ \epsilon }[/math].
• The maximum number of samples allowed for the calibrator has been reached.

Calibration Parameters

• Skeleton structure: a structure that will provide the relationships between joints in order to compute the lengths of the limbs.
• Threshold [math]\displaystyle{ \epsilon }[/math]: the real positive number that will define if the calibration process is ready or not.
• Maximum number of samples: natural number that will define the maximum number of samples allowed, in case there is no convergence to the threshold.
• Minimum number of samples: natural number that will define the minimum number of samples allowed, in order to prevent an early convergence to the threshold with few samples.
• Symmetric limbs: a list of the different pairs of limbs that need to be set as symmetric.

How to Use the Module

Here we will explain how the calibration process is done through an example using the parameters described above in order to have a better understanding of the module usage.

Let's say that we have an array of joint positions that were taken on a sequence of frames, then each frame has a set of 15 3D positions. However, the positions of the joints in each frame suggest that the length of the skeleton's limbs is changing. In order to have a fixed skeleton, we will analyze the positions until there is just a little variation in the lengths of all the skeleton limbs.

• First we need to import the calibration module from the pose estimation library.

 from pose_estimation.skeleton_calibration import *

• Now we define the calibration numeric parameters. These are the minimum and a maximum number of samples, and the threshold for the calibration process.

 #Threshold in cm
 THRESHOLD = 1
 MIN_SAMPLES = 10
 MAX_SAMPLES = 100

• Next we have to define the skeleton structure that will provide the skeleton joints relationships. This skeleton will also contain the resulting lengths when the calibration process is done. The skeleton requires a parameter the path to a JSON file describing its joints. The structure of this file is shown here.

 JSON_PATH = "/path/to/the/json/file"  
 skeleton = Skeleton(JSON_PATH)

• Now we can initialize our calibrator object.

 calibrator = SkeletonCalibrator(skeleton, THRESHOLD, MIN_SAMPLES, MAX_SAMPLES)

• Next step is to start the calibration process. For this step, we will assume that the joints positions that were taken on a sequence of frames are already loaded in an array.

 for frame_positions in frame_positions_array: 
    #Calibrate function
    calibrator.calibrate(frame_positions)
    #Check if the stop conditions are met
    if (calibrator.is_calibrated()):
        break

• Finally, when the calibration process is done we can obtain our calibrated skeleton. As mentioned above, we can tell the calibrator if we want some limbs to be symmetric. In order to make limbs symmetric, we need to provide a list of identifiers that define the pairs of symmetric limbs. Once again, it is recommended to check this in order to understand how limbs are defined. If you don't want any symmetric limbs, just don't provide any argument to the function.

In this example, the pairs provided represent the id of the shoulders, elbows, wrists, hips, knees, and ankles for a human skeleton, but remember that this may change according to the skeleton you define.

 SYMMETRIC_LIMBS = [[3,9],[4,10],[5,11],[6,12],[7,13],[8,14]]
 skeleton = calibrator.get_calibrated_skeleton(SYMMETRIC_LIMBS)