Stereo Vision with Kinect (xbox360 device) on i.MX6 processor

After a long time with no posts, here it goes one more !

It is known on Image processing based projects that works as: classifiers, object tracker, and etc., that the a bad segmentation procedure could fail the entire project.

Based on the gesture recognition project already presented on the lasts posts, we can clearly see if we want to the application works fine, with every person, independent of skin color, then the user needs to wear a glove or another accessory that will differentiate from the background, also, in a gesture recognition demo, only movements on X and Y axis is detected efficiently.

It is because in images provided by a single camera, we need to do segmentation based on colors and then check the contours for the desired pattern. If the background has a color most likely with the skin color or the glove color, then we have a serious problem for the segmentation par. A lot of noise will appear on the final image and the classification/recognition will not work properly.

Unless you are working on a project that will classify/track/detect some object in a controlled environment, one camera is sufficient, but if you are aiming a project that could work on every place, so we need a different approach.

The solution for it is quite simple, we just need to add another camera !

With 2 cameras and the correct setup (alignment and distance between them), we have 2 images with different perspectives from the same point, it is, a 3D image.

There are some algorithms on the web that correlates the images from the 2 cameras and returns the disparity map, that is, the depth image.

With a depth image, the color segmentation on a project can be avoided and a segmentation based on distance can be used instead, and also, gestures more complex could be recognized.

Unfortuntely, the disparty map calculation is a heavy to the cpu do it alone, so we have the following options to improve the performance:

1 - use OpenCL (share the tasks with the GPU)
2 - use a device that already release depth images from 2 different sensors.

As we are working on a embedded system, the OpenCL port for a embedded system is not so complete, so we could face some issues and spent a lot of time on it.

On the other hand we could use a device which gives us the desired depth image. The chosen device for this task was the Microsoft Kinect. Yes ! that one used for gaming on Xbox360. Of course there is another device you could use, for example: http://www.tyzx.com/products/DeepSeaG3.html

As I already have a kinect to test (only for work, because for playing I still prefer my PS3 heheheheheh), as a demo we are going to use the kinect.

The library used for this project was the OpenKinect and you can get more information here: www.openkinect.org

Since I use Linux on my projects (LTIB), I´m going to show how to build the OpenKinect library for this platform:


NOTE: As the i.MX6 is supported on the i.MX53 BSP.

1 - Freescale´s linux BSP: https://www.freescale.com/webapp/Download?colCode=L2.6.35_11.03_ER_SOURCE&prodCode=LEIMX&appType=license&location=null&fpsp=1&Parent_nodeId=1276810298241720831102&Parent_pageType=product

2 - follow the documentation on how to install the ltib on your host linux

3 - once you have the linux running on your board we can now start building the openkinect driver.

4 - clone the openkinect git to your host: git clone git://github.com/OpenKinect/libfreenect.git

5 - it will create a freenect folder, cd to it: cd freenect

6 - create a build folder: mk build

7 - get into the ltib shell using the ltib -m command: home/ltib> ./ltib -m shell

8 - as openkinect is intended for x86, the sample apps are based on OpenGL with GLUT, so we need to avoid building any sample code, just edit the file: ../freenect/CMakeLists.txt and at line 51~63 turn the build options to off.


9 - once in the ltib shell (LTIB>) go to the build folder and type: cmake ..

10 - if every thing goes fine the log output should look like:

LTIB> cmake ..
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /opt/freescale/ltib/usr/spoof/gcc
-- Check for working C compiler: /opt/freescale/ltib/usr/spoof/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /opt/freescale/ltib/usr/spoof/g++
-- Check for working CXX compiler: /opt/freescale/ltib/usr/spoof/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Operating system is Linux
-- Got System Processor i686
-- libfreenect will be installed to /usr/local
-- Headers will be installed to /usr/local/include/libfreenect
-- Libraries will be installed to /usr/local/lib
-- Found libusb-1.0:
-- - Includes: /usr/include
-- - Libraries: /usr/lib/libusb-1.0.so
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for include files CMAKE_HAVE_PTHREAD_H
-- Looking for include files CMAKE_HAVE_PTHREAD_H - found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - not found
-- Found X11: /usr/lib/libX11.so
-- Configuring done
-- Generating done
-- Build files have been written to:
/home/andre/bsps/git_openkinect/libfreenect/build

11 - make

LTIB> make
Scanning dependencies of target freenect Scanning dependencies of target freenectstatic [ 10%] [ 20%] Building C object src/CMakeFiles/freenect.dir/core.c.o
[ 30%] Building C object src/CMakeFiles/freenect.dir/tilt.c.o
[ 40%] Building C object src/CMakeFiles/freenect.dir/cameras.c.o
[ 50%] [ 60%] Building C object src/CMakeFiles/freenectstatic.dir/core.c.o
Building C object src/CMakeFiles/freenectstatic.dir/tilt.c.o
Building C object src/CMakeFiles/freenect.dir/usb_libusb10.c.o
[ 80%] [ 80%] Building C object
src/CMakeFiles/freenect.dir/registration.c.o
Building C object src/CMakeFiles/freenectstatic.dir/cameras.c.o
[ 90%] /home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: In function 'freenect_fetch_zero_plane_info':
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:914: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:915: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:916: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:917: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:918: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:919: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:920: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:921: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: At top level:
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:749: warning:
'read_register' defined but not used
[100%] Building C object src/CMakeFiles/freenectstatic.dir/registration.c.o
Building C object src/CMakeFiles/freenectstatic.dir/usb_libusb10.c.o
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: In function
'freenect_fetch_zero_plane_info':
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:914: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:915: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:916: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:917: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:918: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:919: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:920: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:921: warning:
dereferencing type-punned pointer will break strict-aliasing rules
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c: At top level:
/home/andre/bsps/git_openkinect/libfreenect/src/cameras.c:749: warning:
'read_register' defined but not used
Linking C shared library ../lib/libfreenect.so [100%] Built target freenect Linking C static library ../lib/libfreenect.a [100%] Built target freenectstatic
LTIB>

12 - copy the lib and header files to your rootfs:

cp ../build/lib/* /ltib/rootfs/usr/lib
cp ../include/* /ltib/rootfs/usr/include

with this procedure, the openkinect is correctly installed on your rootfs. If you have any problems, for example, with the libusb, you should update the libusb package on ltib, it is easily done by copying the new downloaded package to /opt/freescale/package and changing the correspondent spec file at: ../ltib/dist/lfs-5.1/package_name/package_name.spec. Every package can be updated using this approach, after that you can rebuild the package using ltib commands:

./ltib -p package_name.spec -m prep
./ltib -p package_name.spec -m scbuild
./ltib -p package_name.spec -m scdeploy

Now as simple application, I used the last video to texture project, but now displaying the images from kinect device instead:

this demo application was wrote based on the sample applications that comes with the openkinect library. In the future posts I´ll give more details about the code.

Source Code: git@github.com:andreluizeng/Kinect-hack.git

EOF !

Gesture Recognition on i.MX6

Hi There, just testing the power of this processor, here is the port of the Gesture Recognition project presented on this blog with some modification, adding the code described on the previews post.

Now we can control a plane that with a video stream texture:

Only 2 gestures was enabled on this application:

1 - opened hand (enables control)
2 - closed hand (disables control)

the other gestures already learned by the ANN is already there, ready to use.

Source Code: git@github.com:andreluizeng/gesture-recognition.git

EOF !

Gesture Recognition Project - step 04 - i.MX53 processor

New achievement ! the ANN was trained to recognizes 8 gestures, but only 7 are showed in this post (the 8th one was censored =D).

Here is the Mean square error evolution during the ANN training:

(MSE value x Training Epochs)

The main problem on this step is the patterns used to train the ANN, they are too similar and sometimes the ANN can get confused or understand an input data as a corrupted data and then it does the approximation erroneously. See the results below:

From this point, some investigations will be done on:

1) improve the hand detection, using glove ?!?! covering the arms ?!?!?
2) use new gestures to apply in an useful use-case
3) detect movements and recognizes motion gestures

EOF!

Gesture Recognition Project - step 03 - i.MX53 processor

Alright, one more video regarding one more step on this project.
So, What is new ?

In this step some modifications on the ANN were done, as follows:

1 - changed the output method, more neurons in the output layer
2 - changed the ativation function to tangent hyperbolic
3 - added the MOMENTUM to the Backpropagation algorithm

The ANN was trained now with three different patterns, the result can see below:

Notice that the little delay on the video is due to a lot of images being displayed at the same time and the dirty code wrote during the development.

For the next steps I will clean up the code (if I got time =)).

EOF!

Gesture Recognition Project - step 02 - i.MX53 processor

In this second step follows:

1 - pre-processing (color conversions, mathematical morphology and filters)
2 - skin color detection (segmentation)
3 - convexity hull for fingers detection (not used anymore - besides it is still being calculated on the video)
4 - Artificial Neural Network for Pattern Classification (gesture recognition)

Procedure Used for the recognition module:

Artificial Neural Network (Multilayer Perceptron Architecture) trained with the Backpropagation Algorithm. The input data is a downsized binary image of the detected contours on the pre-processed image. The sample and test data to train the ANN was acquired previously using the same application, saving the pixels values (binary image) in a TXT file.

Some issues appeared during the training, since the values of the pixels were 0 for black and 1 for white. And as this ANN is relatively big (50x50 image + limiar value = 2501 inputs) the neurons were getting saturated on the very first training epoch. With the help of my PhD supervisor, Professor Ivan Nunes da Silva (University of São Paulo - EESC) the values of the pixels were normalized to smaller values, -0.1 for black pixels and 0.1 for white pixels. With this approach the ANN was able to converge and learn the different patterns.

The ANN was trained for this second step with only 2 gestures, opened hand and closed hand. And for a quick test, I let only 30 neurons on the hidden layer and it took only 5 minutes to get the ANN trained.

here is the Result:




The images displayed on the LCD are: HSV segmented image, Grayscale segmented Image, Convexity hull (not used for the recognition), the ANN input data (50x50 binary image) and the result of detection.

UPDATE
------

The saturation of the neurons can be also avoided using lower values for the net weights. Before the training, random values must be generated to the net weights (initial random values), make sure that these values are small.

EOF!