Android GStreamer audio video playback optimization

Overview

Audio/video playback on Android devices can often be optimized by taking advantage of the device's hardware accelerators. GStreamer supports plug-ins that make it easy to use those hardware accelerators. An iMX6 based hardware platform is used to demonstrate how to create an APK that supports hardware accelerated audio/video playback.

Note: These steps were run in ubuntu 14.04

Porting GStreamer 1.6.0 for Android

Cross-compiling GStreamer 1.6.0 core using Cerbero

Cerbero is a multi-platform build system for Open Source projects that builds and creates native packages for different platforms, architectures and distributions. Projects are defined using recipes files (.recipe), which provides a description of the project being built such as name, version, licenses, source packages and instruction to build the project. Recipes also provide listing of files, which are later used for the packaging. Moreover, the built projects can be combined into packages for distribution. These packages are, depending on the target platform, Windows or OS X installers or Linux packages.

The following steps cross-compile the GStreamer 1.6.0 static libraries using cerbero.

Clone Cerbero repository

cd $HOME
git clone git://anongit.freedesktop.org/gstreamer/cerbero cerbero
cd cerbero
git checkout 1.6.0

Set distribution version

Edit the ~/cerbero/config/cross-android-armv7.cbc file and set the target_distro_version like this.

target_distro_version = DistroVersion.ANDROID_JELLY_BEAN

Fetch all necessary dependent packages

./cerbero-uninstalled -c config/cross-android-armv7.cbc bootstrap

Build the GStreamer 1.6.0 package

./cerbero-uninstalled -c config/cross-android-armv7.cbc package gstreamer-1.0

list available packages

./cerbero-uninstalled -c config/cross-android-armv7.cbc list-packages

Set up and build Android Jelly Bean for Nitrogen6x

A Boundary Devices Nitrogen6x board is used for the target hardware.

Setting up your host machine to build the AOSP correctly

To get a successful compilation of the Android Open Source Project, make sure you have installed the following packages in your host computer.

sudo apt-get install bison g++-multilib git gperf libxml2-utils make python-networkx zlib1g-dev:i386 zip
sudo apt-get install libswitch-perl

Register and install SSH key

To get Android source code for the Boundary Devices boards, you need to register at the Boundary Devices website and then upload your public SSH key so you can get access to the repository. See this links for instructions:

• Register at Boundary Devices website
• Upload public SSH key

Download repo tool

mkdir -p ~/bin
curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo

Create directory to hold Boundary Devices repository

mkdir myandroid
cd myandroid

Fetch Android SDK repository

~/bin/repo init -u git://github.com/boundarydevices/imx-android-r13.4-ga.git -b boundary-imx_jb4.3_1.0.0-ga
~/bin/repo sync

Configure environment

source build/envsetup.sh
lunch

For example, for Nitrogen you can use: 15. nitrogen6x-eng

Build the AOSP

make -j2

Cross-compiling GStreamer-i.MX using Cerbero

In order to build 'gstreamer-imx' and the 'imxeglvivsink' element, 3 dependencies are needed:

• libimxvpuapi: an alternative to Freescale's VPU wrapper. Both the wrapper and this library are layered on top of imx-vpu , the low-level iMX6 VPU interface.
• libvpu: which is included with the Android distribution for the iMX6 under external/linux-lib/vpu.
• libfslvpuwrapper: v 1.0.45 or higher, provided by Freescale.

For this steps is assumed that the GStreamer core is already built as described above.

Extract kernel headers from your Android build

export KERNEL_HEADERS=/tmp/kernel_headers/usr/
cd $DEVDIR/myandroid/jb-4.3/kernel_imx
make headers_install ARCH=arm INSTALL_HDR_PATH=$KERNEL_HEADERS

Copy headers and library to the cerbero

cp myandroid/jb-4.3/external/linux-lib/vpu/vpu_lib.h ~/cerbero/dist/android_armv7/include
cp myandroid/jb-4.3/external/linux-lib/vpu/vpu_io.h ~/cerbero/dist/android_armv7/include
cp myandroid/jb-4.3/out/target/product/nitrogen6x/obj/lib/libvpu.so ~/cerbero/dist/android_armv7/lib

Create and update libimxvpuapi recipe

./cerbero-uninstalled add-recipe libimxvpuapi 0.10.0

Update ~/cerbero/recipes/libimxvpuapi.recipe by replacing the contents with the following recipe.

# -*- Mode: Python -*- vi:si:et:sw=4:sts=4:ts=4:syntax=python
from cerbero.build.build import MakefilesBase

# Static
class Recipe(recipe.Recipe):

    name = 'libimxvpuapi'
    version = '0.10.0'
    licenses = [License.LGPL]
    commit = '0.10.0'
    remotes = {'origin': 'https://github.com/Freescale/libimxvpuapi.git'}
    config_sh = './waf configure'
    make = './waf build'
    make_install = './waf install'
 
    def configure(self):
        MakefilesBase.configure(self)

Build libimxvpuapi

./cerbero-uninstalled -c config/cross-android-armv7.cbc build libimxvpuapi

Create libimxvpuapi and libgstimxveglvivsink pkg-config files

Store the following information in the ~/cerbero/dist/android_armv7/lib/pkgconfig/libimxvpuapi.pc file:

prefix=/home/fquiros/cerbero/dist/android_armv7
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: libimxvpuapi
Description: interface library for i.MX6 VPU devices
Version: 0.10.0
Libs: -L${libdir} -limxvpuapi
Cflags: -I${includedir}

Store the following information in the ~/cerbero/dist/android_armv7/lib/pkgconfig/libgstimxveglvivsink.pc file:

prefix=/home/fquiros/cerbero/dist/android_armv7
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: imxeglvivsink
Description: custom OpenGL ES 2.x based video sink by using the Vivante direct textures, which allow for smooth playback.
Version: 0.12.0
Libs: -L${libdir} -landroid
Cflags: -I${includedir}

Create libfslvpuwrap recipe

./cerbero-uninstalled add-recipe libfslvpuwrap 1.0.46

Update ~/cerbero/recipes/libfslvpuwrap.recipe by replacing the contents with the following recipe.

# -*- Mode: Python -*- vi:si:et:sw=4:sts=4:ts=4:syntax=python

class Recipe(recipe.Recipe):
    name = 'libfslvpuwrap'
    version = '1.0.46'
    licenses = [License.LGPL]
    stype = SourceType.TARBALL
    url = 'http://downloads.ridgerun.com/packages/libfslvpuwrap-1.0.46.tar.gz'

Build libfslvpuwrap

PLATFORM=IMX6Q DEST_DIR=~/cerbero/dist/android_armv7/ ./cerbero-uninstalled -c config/cross-android-armv7.cbc build libfslvpuwrap

Create and update gstreamer-imx recipe

./cerbero-uninstalled add-recipe gstreamer-imx 0.10.1

Update ~/cerbero/recipes/gstreamer-imx.recipe by replacing the contents with the following recipe.

# -*- Mode: Python -*- vi:si:et:sw=4:sts=4:ts=4:syntax=python
from cerbero.build.build import MakefilesBase

# Static
class Recipe(recipe.Recipe):

    name = 'gstreamer-imx'
    version = '0.12.0'
    licenses = [License.LGPL]
    commit = '0.12.0'
    remotes = {'origin': 'https://github.com/Freescale/gstreamer-imx.git'}
    config_sh = './waf configure'
    configure_options = '--build-for-android --kernel-headers=$KERNEL_HEADERS --egl-platform=android'
    make = './waf build'
    make_install = './waf install'
    
    def configure(self):
        MakefilesBase.configure(self)

Fetch and build gstreamer-imx package

mkdir recipes/gstreamer-imx
wget -O recipes/gstreamer-imx/gstreamer-imx-0.10.1-patchset.tar.gz http://downloads.ridgerun.com/packages/gstreamer-imx-0.10.1-patchset.tar.gz
./cerbero-uninstalled -c config/cross-android-armv7.cbc build gstreamer-imx

Fetch helper files

Ah, so here is the magic.

wget http://downloads.ridgerun.com/packages/gstreamer-imx_android_required_files.tar.gz
tar -C ~/cerbero/dist/android_armv7/lib/pkgconfig -xzf gstreamer-imx_android_required_files.tar.gz

If you changed where files are getting installed, then adjust the prefix value in the *.pc files.

Building GStreamer plugins using Cerbero

Create and update gst1.0-fsl-plugins recipe

./cerbero-uninstalled add-recipe gst1.0-fsl-plugins 4.0.4

Update ~/cerbero/recipes/gst1.0-fsl-plugins.recipe by replacing the contents with the following recipe.

# -*- Mode: Python -*- vi:si:et:sw=4:sts=4:ts=4:syntax=python

# For Static (4.0.4)
class Recipe(recipe.Recipe):
    name = 'gst1.0-fsl-plugins'
    version = '4.0.4'
    licenses = [License.LGPL]
    stype = SourceType.TARBALL
    url = 'http://www.freescale.com/lgfiles/NMG/MAD/YOCTO/gst1.0-fsl-plugins-4.0.4.tar.gz'
    configure_options = '--enable-static-plugins --enable-static --disable-shared  PLATFORM=MX6 CFLAGS="-O0 -ggdb3 -I -DGST_PLUGIN_BUILD_STATIC -funwind-tables -ffunction-sections -funwind-tables -fstack-protector -no-canonical-prefixes -fPIC -mthumb -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -DANDROID -DPIC -D__ARM_ARCH_7A__ -Wa,--noexecstack"'
    patches = ['gst1.0-fsl-plugins/0001-fixing-plugins-name.patch',
			   'gst1.0-fsl-plugins/0002-allow-set-bitrate-on-the-fly.patch',
			   'gst1.0-fsl-plugins/0006-avoid-tools-build-4.0.4.patch',
			   'gst1.0-fsl-plugins/0007-vpuenc-gst-imx-compatibility.patch',
			   'gst1.0-fsl-plugins/0008-v4l2-add-capture-type.patch']

Patch build and install gst1.0-fsl-plugins

mkdir recipes/gst1.0-fsl-plugins
wget http://downloads.ridgerun.com/packages/gst1.0-fsl-plugins-4.0.3-patchset.tar.gz
tar -C recipes/gst1.0-fsl-plugins -xzf gst1.0-fsl-plugins-4.0.3-patchset.tar.gz
./cerbero-uninstalled -c config/cross-android-armv7.cbc build gst1.0-fsl-plugins
cp ~/cerbero/dist/android_armv7/lib/gstreamer-1.0/libgstvpu.a ~/cerbero/dist/android_armv7/lib/gstreamer-1.0/static
cp ~/cerbero/dist/android_armv7/lib/gstreamer-1.0/libgstvpu.la ~/cerbero/dist/android_armv7/lib/gstreamer-1.0/static

Improving the GStreamer 1.6.0 audio/video pipeline

Assuming that the previous steps were completed successfully, follow these additional instructions in order to get a better CPU performance and smooth playback when the respective GStreamer 1.6.0 Android application. In this section, the approach used consists in a pipeline that uses the 'imxeglvivsink' element, where you need to add the following linking libraries, GStreamer plugins and dependences in the Android.mk file of your application.

LOCAL_LDLIBS              := -llog -landroid -lGLESv2 -lEGL

GSTREAMER_PLUGINS         := $(GSTREAMER_PLUGINS_CORE) $(GSTREAMER_PLUGINS_SYS) $(GSTREAMER_PLUGINS_EFFECTS) $(GSTREAMER_PLUGINS_PLAYBACK) $(GSTREAMER_PLUGINS_CODECS) $(GSTREAMER_PLUGINS_CODECS_RESTRICTED) vpu imxvpu avi isomp4 imxeglvivsink

GSTREAMER_EXTRA_DEPS      := gstreamer-video-1.0 libgstvpu libgstimxvpu libfslvpuwrap libgstimxcommon libimxvpuapi libgstimxeglvivsink 

GStreamer 1.6.0 audio/video playback pipeline

This section shows both the pipeline used in the source code of the application, where the 'imxeglvivsink' elements was used in order to display the audio and video playback in an Android window.

• 'imxeglvivsink' pipeline

'C' SOURCE CODE PIPELINE

MOVIE=/tmp/bigbuckbunny.mp4
gst_parse_launch("playbin uri=file://$MOVIE video-sink='imxeglvivsink'", &error);

CPU Performance

Run the playback by hitting 'Play' on the Android interface, where the performance measurements recorded were the following and they demonstrate that the performance improves and the video is playing smoother.

• 'imxeglvivsink' pipeline, using 'vpudec'

HIGHEST CPU LOAD

User 16%, System 10%, IOW 0%, IRQ 0%
User 69 + Nice 0 + Sys 44 + Idle 307 + IOW 0 + IRQ 0 + SIRQ 0 = 420

PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
3383  2  19% S    25 952436K  45372K  fg u0_a78   com.gst_sdk_tutorials.tutorial_3

LOWEST CPU LOAD

User 4%, System 10%, IOW 0%, IRQ 0%
User 16 + Nice 0 + Sys 39 + Idle 329 + IOW 0 + IRQ 0 + SIRQ 0 = 384

PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
3383  2   8% S    25 952604K  45576K  fg u0_a78   com.gst_sdk_tutorials.tutorial_3

• 'imxeglvivsink' pipeline, using 'imxvpudec'

HIGHEST CPU LOAD

User 19%, System 6%, IOW 0%, IRQ 0%
User 83 + Nice 0 + Sys 29 + Idle 310 + IOW 0 + IRQ 0 + SIRQ 1 = 423

PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
3312  0  19% S    25 956436K  45668K  fg u0_a48   com.gst_sdk_tutorials.tutorial_3

LOWEST CPU LOAD

User 3%, System 15%, IOW 0%, IRQ 0%
User 15 + Nice 0 + Sys 62 + Idle 312 + IOW 0 + IRQ 0 + SIRQ 0 = 389

PID PR CPU% S  #THR     VSS     RSS PCY UID      Name
3312  0   9% S    25 964352K  45744K  fg u0_a48   com.gst_sdk_tutorials.tutorial_3