Full Body Pose Estimation for Sports Analysis - Pose Alignment
 | WORK IN PROGRESS. Please Contact RidgeRun OR email to support@ridgerun.com if you have any questions. |
Contents
Introduction
The pose alignment module is intended to receive the 2D or 3D positions from the pose estimation of the videos of 2 different people performing the same type of exercise in each one. From these 2 videos it is assumed that one of them is performing the exercise correctly (the trainer) and the other one is doing the same exercise but might require corrections (the trainee). The idea of this module is to align the trainer and trainee movements in order to decide which corrections needs to do the trainee in order to perform the exercise correctly.
How it Works
This module is able to work with both 2D and 3D dimensions, thus it needs a different kind of processing for each one of them. The steps done in this module are divided into 2 main stages, the preprocessing stage and the alignment stage. These stages and their subprocessing steps are shown in the next flow diagram.
Preprocessing
This step is meant to prepare the poses data according to their dimensions in order to be aligned in the next stage. This process is applied in the same way for both the trainer and trainee videos. In the next subsections we will explain how every module shown in the flow diagram works.
2D: Front and Side Views Generation
The front and side views of the performed movements are used as a way of obtaining that extra perspective or depth that 2D images lack of. To obtain these views we use the 2D Motion Retargeting model that allows to combine the motion, skeleton and viewpoint of 3 different videos into 1 sequence that contains the motion of the person in the first video, the skeleton dimensions of the second one and the viewpoint of the third one. The front and side views are generated by using this model and combining the trainer motion and the trainer skeleton with a predetermined video of a person standing in a front and side position from the camera viewpoint, since we only want to modify the viewpoint and not the movements or body. This same process is then applied also to the trainee videos. From this step we generate a front and side view for both the trainer and the trainee, and further alignment processing steps are performed over the front and side views of each one. The next gif show a representation of these views.
2D: Hips Alignment with Horizontal Image Axis
As part of the normalization steps, we align the 2D vector that goes from the left hip to right one with the X axis, in order to correct diagonal orientations of the camera when the videos were recorded. This is done by computing the angle between the hips vector and the X axis and then rotating all the joints positions with than angle.
2D: Translation of Poses to the Image Center
The other normalization step is about moving the skeleton in the image to the center of this one. For this step we translate the skeleton pelvis from any position in the image to the center, which in our case is (320,240) since our images are of 640x480, and then move the other joints as well to maintain the distances to the pelvis before it was moved.
2D: Average of the Estimation
Since we allow videos that were recorded with multiple synchronized cameras, we need a way to reduce the estimations for all of them into one. To do these we average the joints positions in a same frame for every side and front view, assuming that we have a front and side view for every camera that was used to record the same video. This step is only done when multiple cameras were used to record a video.
3D: Rotation Angles Computing
This is the only step required when inputs have 3 dimensions, since the pose estimator already ensures that they are normalized. In this case all joint positions are relative to the pelvis because it is the center of the coordinated system. However, since coordinates depend on skeleton dimensions we prefer to use the angles representing the pose for the alignment process. For obtaining these angles for every pose we use the kinematic fitting module that allows to map a coordinates representation to an angles representation.
Alignment
