Full Body Pose Estimation for Sports Analysis - About

Full Body Pose Estimation for Sports Analysis on NVIDIA ®Jetson™
About
Camera Abstraction
Camera Calibration AprilTag Camera Calibration Chessboard Camera Calibration 3D Pose Estimation Pose Representation Skeleton Calibration Temporal Filter Kinematic Fitting Pose Optimization Pose Alignment Pose Correction Performance
Introduction Required Hardware How to Get the Code How to Install the Library How to Use the Library
Getting Started
Contact Us

Introduction

The full-body pose estimation library is a set of Python modules intended to carry out the whole pose estimation process. This process includes the camera calibration, video capturing, the pose estimation itself, and the estimation refinement. The main objective of this library is to further use it as a starting point for human motion analysis in sports. However, the majority of modules are designed in a way that they can be reused in other applications, besides this one.

A general view of the whole system can be seen in Figure 1.

Figure 1. General block diagram.

Workflow Description

In the diagram, it can be seen that there is a flow between some of the modules. This is the workflow of the functional units of the pose estimation process, however, we have other modules in the corner that are not part of the general workflow, but play an important role in the whole process. Next, we will describe briefly the role of each one of the modules above:

• Camera Abstraction:

This module is in charge of providing an interface for instancing and using of different camera types. It is related to the frame capture in the workflow since it provides the camera configuration according to the type of camera to be used. This is one of the modules that may be reusable in other applications that require camera interaction.

• Camera Calibration:

This module provides a way to relate a camera frame with what it is being actually captured in the real world. A camera is considered calibrated when its calibration parameters are known. Therefore, the camera calibration process is a procedure that allows the determination of the camera's parameters. This is also related to the frame capture in the workflow since it is a process that should be done before starting the recording stage, in order to have accurate pose predictions. This is also a reusable module for any other application that has camera interaction.

• 3D Pose Estimation:

This is the module that provides the estimation of the three-dimensional human pose from the input frames, using a deep neural network. The output of this module may include some noise due to the network's accuracy or the recording process. In order to overcome that noise, its output is passed to the following modules for refinement. In this case, the deep neural network is already trained with human poses, which limits this module to work only with human skeleton types. However, it can also be reused in other related to human pose estimation applications.

• Temporal Filtering:

In this module, the sequence of frames is smoothed through a low pass adaptive filter, with the purpose of reducing the constantly jumping positions of the joints due to the previous stage's accuracy. This is another module that can be reused in other applications to filter the 3D positions of any object since it is not attached to work only with skeletons.

• Pose Representation:

This is another module out of the general workflow, but important as well. It is intended to model the movement of any type of skeleton by using a relative representation. By using its relative representation it is possible to adjust the predicted pose to a fixed skeleton using rotation angles. This module is necessary for the skeleton calibration and kinematic fitting since both of them need to know with which type of skeleton they are working with. The pose representation module was created to design any type of skeleton besides humans, which provides high flexibility for its use in other applications.

• Skeleton Calibration:

This other module is intended to obtain the limb lengths of the skeleton whose pose is going to be estimated. This process is necessary in order to apply the kinematic fitting to a calibrated skeleton, which will adjust the pose prediction to the subject body, and will ignore any noise related to limbs' lengths variations. This process is meant to be executed in the first iterations of the workflow until the skeleton is considered calibrated. Once the process finishes, it will output the skeleton measures to the kinematic fitting stage and will stop calibrating, which means that in following iterations the flow will go directly from the filter to the kinematic fitting. This module is also reusable with any other type of skeleton, and it is meant to be used along with the pose representation module.

• Kinematic Fitting:

The kinematic fitting module is intended to receive the filtered 3D positions from the pose estimator and adjust them to the user limb lengths while considering the anatomic constraints of the human body. This process is done through a deep network model that applies kinematic chains that model the skeleton pose with three-dimensional rotations and translations. The output of this module will be the pose with all the refinements applied and the possibility of transferring its angles to any other human skeleton. Since this module uses a deep network model it is bounded to use human skeletons, but to fix this, we also provide the model training steps in order to make it reusable with any other type of skeletons, along with the pose representation module.