Onboard Camera - V4L wrapper for use with OpenCV

Many people have been asking how to use the onbard camera (CSI and MIPI that comes with the Freesacale's development board) with OpenCV.

Unfortunately the mxc driver for this camera is not compatible with OpenCV and there is no way to use it directly, instead, we can use wrapper functions to access it. In this post I will share with you guys some utility functions I created (based on an Ipu demo app from Rogerio Pimentel - Freescaler) that uses V4L to access the camera.

When using these functions we will get frames on UYVY format (YUV422) and to use with OpenCV it must be converted to RGB24 (24bpp), all these are included in the source code.

The code can be downloaded from here, it also comes with a sample application, you just need to export the ROOTFS envioroment variable and check if the toolchain you are going to use is the same as defined in the Makefile, otherwise you must change it too.

The demo application (tested on i.MX6 sabresd board) performs the Canny edge detector and displays it using cvShowImage, you could display it directly in the framebuffer using the utility function, just uncomment the functions in the code and also add the conversion from RGB888toYUV422.

to run this application, make sure to have these drivers installed:

modprobe ov5642_camera
modprobe ov5640_camera_mipi
modprobe mxc_v4l2_capture

here is the result:

ps:  There is a "small" issue in these wrapper functions, as it uses multiple buffers for capturing and output (if you use it instead cvShowImage), you need to at least use the  V4LWrapper_QueryFrame function 3 times in a row to fill all buffers and the display it, so, the best scenario would be using this function like cvQueryFrame, when I got a fix for this I will update this post. Meanwhile just throw this function inside a loop function and you are good to go =)

EOF !

Building OpenCV-2.4.X for Freescale's i.MX6 BSP (Yocto)

Lately a lot of people are working with the Yocto Project and many of them migrated from LTIB (like me). Yocto uses a different conception when adding new packages/applications to the system, now everything is based on RECIPES. As it is being highly used, the amount of packages (recipes) already included is very big and it keep increasing continuously. For our lucky the recipe for OpenCV is already there, we just need to configure the system in order to add it to us.

In order to get everything up running we will divide de taks in steps:


Step #1 - Installing Yocto
--------------------------

As our focus is to install OpenCV, the Yocto install procedure we can use this very good tutorial created by Daiane: https://community.freescale.com/docs/DOC-94849


Step #2 - Enabling OpenCV
----------------------------

As we already have the OpenCV recipe in our Yocto release, we just need to add what packages we want in our local.conf file, located at /yocto/fsl-community-bsp/build/conf. With some modification (opencv package), it should look like this:

    MACHINE ??= 'imx6qsabresd'
    DISTRO ?= 'poky'
    PACKAGE_CLASSES ?= "package_rpm"
    EXTRA_IMAGE_FEATURES = "debug-tweaks dev-pkgs"
    USER_CLASSES ?= "buildstats image-mklibs image-prelink"
    PATCHRESOLVE = "noop"
    BB_DISKMON_DIRS = "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K" 
    PACKAGECONFIG_pn-qemu-native = "sdl"
    ASSUME_PROVIDED += "libsdl-native"
    CONF_VERSION = "1"

    BB_NUMBER_THREADS = '4'
    PARALLEL_MAKE = '-j 4'

    DL_DIR ?= "${BSPDIR}/downloads/"
    ACCEPT_FSL_EULA = ""

    CORE_IMAGE_EXTRA_INSTALL += "gpu-viv-bin-mx6q gpu-viv-bin-mx6q-dev"
    CORE_IMAGE_EXTRA_INSTALL += "libopencv-core-dev libopencv-highgui-dev
libopencv-imgproc-dev libopencv-objdetect-dev libopencv-ml-dev"

    LICENSE_FLAGS_WHITELIST = "commercial"


Note that we included the "-dev" packages, this is necessary if you always want to have the OpenCV headers/libraries included in the rootfs, Yocto is smart if you don´t add a "-dev" package and the libraries are just included any application that uses it needs to be built. As we always want our OpenCV stuff to build our applications so we use it this way.


Step #3 - Building OpenCV
----------------------------

Now the easy part:

/yocto/fsl-community-bsp/build$./bitbake core-image-x11

after build is finished you can check the images generated by the bitbake command at:

/build/tmp/deploy/images/imx6qsabresd/

and after extracting the rootfs: core-image-x11-imx6qsabresd.tar.bz2, you can find the opencv libraries in the /usr/lib folder:

andre@b22958:~/bsps/yocto/rootfs$ ls usr/lib/libopen*
usr/lib/libopencv_calib3d.so           usr/lib/libopencv_ml.so
usr/lib/libopencv_calib3d.so.2.4       usr/lib/libopencv_ml.so.2.4
usr/lib/libopencv_calib3d.so.2.4.7     usr/lib/libopencv_ml.so.2.4.7
usr/lib/libopencv_contrib.so           usr/lib/libopencv_nonfree.so
usr/lib/libopencv_contrib.so.2.4       usr/lib/libopencv_nonfree.so.2.4
usr/lib/libopencv_contrib.so.2.4.7     usr/lib/libopencv_nonfree.so.2.4.7
usr/lib/libopencv_core.so              usr/lib/libopencv_objdetect.so
usr/lib/libopencv_core.so.2.4          usr/lib/libopencv_objdetect.so.2.4
usr/lib/libopencv_core.so.2.4.7        usr/lib/libopencv_objdetect.so.2.4.7
usr/lib/libopencv_features2d.so        usr/lib/libopencv_ocl.so
usr/lib/libopencv_features2d.so.2.4    usr/lib/libopencv_ocl.so.2.4
usr/lib/libopencv_features2d.so.2.4.7  usr/lib/libopencv_ocl.so.2.4.7
usr/lib/libopencv_flann.so             usr/lib/libopencv_photo.so
usr/lib/libopencv_flann.so.2.4         usr/lib/libopencv_photo.so.2.4
usr/lib/libopencv_flann.so.2.4.7       usr/lib/libopencv_photo.so.2.4.7
usr/lib/libopencv_gpu.so               usr/lib/libopencv_stitching.so
usr/lib/libopencv_gpu.so.2.4           usr/lib/libopencv_stitching.so.2.4
usr/lib/libopencv_gpu.so.2.4.7         usr/lib/libopencv_stitching.so.2.4.7
usr/lib/libopencv_highgui.so           usr/lib/libopencv_superres.so
usr/lib/libopencv_highgui.so.2.4       usr/lib/libopencv_superres.so.2.4
usr/lib/libopencv_highgui.so.2.4.7     usr/lib/libopencv_superres.so.2.4.7
usr/lib/libopencv_imgproc.so           usr/lib/libopencv_video.so
usr/lib/libopencv_imgproc.so.2.4       usr/lib/libopencv_video.so.2.4
usr/lib/libopencv_imgproc.so.2.4.7     usr/lib/libopencv_video.so.2.4.7
usr/lib/libopencv_legacy.so            usr/lib/libopencv_videostab.so
usr/lib/libopencv_legacy.so.2.4        usr/lib/libopencv_videostab.so.2.4
usr/lib/libopencv_legacy.so.2.4.7      usr/lib/libopencv_videostab.so.2.4.7
andre@b22958:~/bsps/yocto/rootfs$

ps: don´t forget to flash the card with the image created at /tmp/deploy/images/imx6qsabresd/core-image-x11-imx6qsabresd.sdcard

$ sudo dd if= /build/tmp/deploy/images/imx6qsabresd/core-image-x11-imx6qsabresd.sdcard of=/dev/sdb

----------------------------------------

After those 3 steps above you should be able to find all the OpenCV headers/libraries needed by mostly of your application, but in any case you need more dev packages, you can look at: /tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/opencv/2.4.6+gitAUTOINC+1253c2101b-r0/packages-split

Now that you have the OpenCV headers/libraries we need the toolchain to build our sample application, just re-do the bitbake command now adding the "-c populate" option in the command line:

/yocto/fsl-community-bsp/build$./bitbake core-image-x11 -c populate_sdk

and then run the install script created at: /yocto/fsl-community-bsp/build/tmp/deploy/sdk to install it.

With that you will be able to see the toolchain installed at: /opt/poky

Now we are able to test our sample code, just a camera test and you can find the source code here: camera_test_sample

To build this application you need a new terminal window (all environment variables will be reset), then run the setup environment:

$ cd /opt/poky/1.5+snapshot/
$ . ./environment-setup-cortexa9hf-vfp-neon-poky-linux-gnueabi

and then go to the camera_test_yocto folder and type make. The binary will be placed in the bin folder.

Once flashed your card with the Yocto image (opencv included), mount the sd card in your host computer and then copy the binary to your rootfs.

To test it, run the application with the following command:

$ DISPLAY =:0 ./camera_test

EOF !

Building OpenCV-2.4.X for Freescale's i.MX6 BSP (LTIB)

I was working on this post for a long time, and I didn't share any information before because I Was facing a big problem to get the highgui library successfully built to our embedded system. And as you may know, the highgui is very important when you need to open a video stream, create a camera device and even work with windows in OpenCV.

Last week I managed to build it successfully using the latest Freescale's BSP (kernel 3.0.35) and I'm going to show you how to do that in the following lines.

First of all,  you can download the Freescale's i.MX6 BSP here !


Unfortunately getting the latest BSP doesn't mean you are going to get all the latest packages. The only things that are really update in this BSP and probably all others are the kernel and specific drivers for the current hardware you are going to work with. The most commons packages like GTK, GLIB2, PANGO, CAIRO and etc... are not updated, and in order to build the new OpenCV 2.4.X we will need to upgrade these packages and install a new one, the v4l-utils.

So let's stop with this BLAH BLAH BLAH and get into what is really interesting  !


Step 1 - Building the dependencies
-----------------------------------------------------------
 1) Assuming you already have the BSP installed (min profile) in your host machine, you will need to upgrade the GLIB2 package and install the v2l-utils one.

2) Get the new spec files at: https://github.com/andreluizeng/ltib-pkgs-upgrade

3) the new GLIB2 now has a dependency, the LIBFFI. (you can also get the spec file of this lib in the link above.

4) Build & install the libffi as follows:
     mkdir ../ltib/dist/lfs-5.1/libffi
     cp ../git/ltib-pkgs-upgrade/libffi.spec ../ltib/dist/lfs-5.1/libffi
     cp ../git/ltib-pkgs-upgrade/libffi-3.0.12-includedir-1.patch /opt/freescale/pkgs
     cp ../downloads/libffi.tar.gz /opt/freescale/pkgs
* you can find the link that you can get the libffi from.
      cd ../ltib
     ./ltib -p libffi.spec -m prep
     ./ltib -p libffi.spec -m scbuild
     ./ltib -p libffi.spec -m scdeploy

5) Build & install the new glib2
    cd ../ltib
    cp ../git/ltib-pkgs-upgrade/glib2.spec ../ltib/dist/lfs-5.1/glib2
    cp ../downloads/glib-2.35.9.tar.xz /opt/freescale/pkgs
    ./ltib -p glib2.spec -m prep
    ./ltib -p glib2.spec -m scbuild
    ./ltib -p glib2.spec -m scdeploy

6) Build & install the v4l-utils packages
    mkdir ../ltib/dist/lfs-5.1/v4l-utils
    cp ../git/ltib-pkgs-upgrade/v4l-utils.spec ../ltib/dist/lfs-5.1/v4l-utils
    cp ../downloads/v4l-utils-0.9.3.tar.bz2 /opt/freescale/pkgs
    cd ../ltib

    ./ltib -p v4l-utils.spec -m prep
    ./ltib -p v4l-utils.spec -m scbuild
    ./ltib -p v4l-utils.spec -m scdeploy


After all these steps you should have the necessary dependencies built for now building the OpenCV 2.4.X.


Step 2 - Cross-Compiling OpenCV-2.4.X
-----------------------------------------------------------------

When Cross-compiling using cmake you need a special file containing the information about the toolchain you are going to use. In our case we can create this file as:


1) touch toolchain.cmake & vi toolchain.cmake

2) add the following information:

               # this one is important
               set( CMAKE_SYSTEM_NAME Linux )

               #this one not so much
               set( CMAKE_SYSTEM_PROCESSOR arm )

               # specify the cross compiler
                set( CMAKE_C_COMPILER /opt/freescale/usr/local/gcc-4.6.2-glibc-    2.13-linaro-multilib-2011.12/fsl-linaro-toolchain/bin/arm-none-linux-gnueabi-gcc )
               set( CMAKE_CXX_COMPILER /opt/freescale/usr/local/gcc-4.6.2-glibc-2.13-linaro-multilib-2011.12/fsl-linaro-toolchain/bin/arm-none-linux-gnueabi-g++ )

              # where is the target environment - point to your rootfs here
              set( CMAKE_FIND_ROOT_PATH  /home/andre/bsps/imx6x/1301/ltib/rootfs )

              # search for programs in the build host directories
              set( CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER )

             # for libraries and headers in the target directories
             set( CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY )
             set( CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY )

             # point to your rootfs path here 
             set( CMAKE_CXX_FLAGS "-L/home/andre/bsps/imx6x/1301/ltib/rootfs/usr/lib" )

3) download OpenCV-2.4.X here or here.

4) untar the package:
    tar -xvf OpenCV-2.4.2.tar.bz2

5) cd OpenCV-2.4.2
    mkdir build
    cd build
    cp ../../toolchain.cmake .
    cmake -DCMAKE_TOOLCHAIN_FILE=toolchain.cmake ../

after these steps you should have the necessary files to be built in your build directory, you now just need to configure which extra packages you want to add or remove for your OpenCV profile.

In your build folder:

6) ccmake .

7) you can let your configuration similar to mine:

BUILD_DOCS                          OFF
 BUILD_EXAMPLES                OFF
BUILD_JASPER                       OFF
BUILD_JPEG                           OFF
 BUILD_PACKAGE                  ON
BUILD_PERF_TESTS              ON
 BUILD_PNG                           OFF
 BUILD_SHARED_LIBS           ON
 BUILD_TESTS                        ON
 BUILD_TIFF                           OFF
 BUILD_WITH_DEBUG_INFO ON
 BUILD_ZLIB                          OFF
 BUILD_opencv_calib3d        ON
 BUILD_opencv_contrib        ON
 BUILD_opencv_core             ON
 BUILD_opencv_features2d  ON
 BUILD_opencv_flann            ON
 BUILD_opencv_gpu              ON
 BUILD_opencv_highgui        ON
 BUILD_opencv_imgproc       ON
 BUILD_opencv_legacy          ON
 BUILD_opencv_ml                 ON
 BUILD_opencv_nonfree        ON
 BUILD_opencv_objdetect     ON
 BUILD_opencv_photo           ON
 BUILD_opencv_stitching      ON
 BUILD_opencv_ts                  ON
 BUILD_opencv_video            ON
 BUILD_opencv_videostab     ON
 CMAKE_BUILD_TYPE
 CMAKE_CONFIGURATION_TYPES  Debug;Release
 CMAKE_INSTALL_PREFIX             /home/andre/imx_applications/OpenCV-2.4.2/build/install
 CMAKE_TOOLCHAIN_FILE             /home/andre/imx_applications/OpenCV-2.4.2/toolchain.cmake
 CMAKE_VERBOSE                   OFF
 CUDA_BUILD_CUBIN               OFF
 CUDA_BUILD_EMULATION     OFF
 CUDA_SDK_ROOT_DIR             CUDA_SDK_ROOT_DIR-NOTFOUND          
 CUDA_TOOLKIT_ROOT_DIR     CUDA_TOOLKIT_ROOT_DIR-NOTFOUND   
 CUDA_VERBOSE_BUILD          OFF
 EIGEN_INCLUDE_PATH            EIGEN_INCLUDE_PATH-NOTFOUND
ENABLE_NOISY_WARNINGS                OFF
 ENABLE_OMIT_FRAME_POINTER      ON
 ENABLE_PRECOMPILED_HEADERS  ON
 ENABLE_PROFILING                           OFF
 ENABLE_SOLUTION_FOLDERS          OFF
 EXECUTABLE_OUTPUT_PATH           /home/andre/imx_applications/OpenCV-2.4.2/build/bin
 INSTALL_C_EXAMPLES                      OFF
 INSTALL_PYTHON_EXAMPLES          OFF
 INSTALL_TO_MANGLED_PATHS        OFF                 
 LIBRARY_OUTPUT_PATH_ROOT         /home/andre/imx_applications/OpenCV-2.4.2/build                                                                                              
 OPENCV_CAN_BREAK_BINARY_COMPAT   OFF
 OPENCV_CONFIG_FILE_INCLUDE_DIR   /home/andre/imx_applications/OpenCV-2.4.2/build                                                                                              
 OPENCV_WARNINGS_ARE_ERRORS
OFF OPENEXR_INCLUDE_PATH                  OPENEXR_INCLUDE_PATH-NOTFOUND PVAPI_INCLUDE_PATH            PVAPI_INCLUDE_PATH-NOTFOUND PYTHON_PACKAGES_PATH     lib/python2.7/site-packages WITH_1394         ON
 WITH_CUBLAS                   OFF
 WITH_CUDA                       OFF
 WITH_CUFFT                     ON
WITH_EIGEN                       ON
WITH_FFMPEG                   ON
 WITH_GSTREAMER           ON
 WITH_GTK                          ON
 WITH_JASPER                    OFF
 WITH_JPEG                        ON
 WITH_OPENEXR                ON
 WITH_OPENGL                  OFF
 WITH_OPENNI                   OFF
 WITH_PNG                         ON
 WITH_PVAPI                       ON
 WITH_QT                           OFF
 WITH_TBB                         OFF
 WITH_TIFF                        ON
 WITH_UNICAP                   OFF
 WITH_V4L                         ON
 WITH_XIMEA                     OFF
 WITH_XINE                        OFF
   
you basically removed the gpu acceleration (we don't support full profile OpenCL), we enabled the v4l library and also disabled some unnecessary ones, you can try your on configuration if you will.

8) make -j

9) make install


If you reached the step 9 with no issues you should have all  your cross built OpenCV in the install folder. Just copy it to your rootfs:

10) sudo cp -a build/install ../ltib/rootfs/usr

and you know have a OpenCV-2.4.X installation in your embedded system.

Now the result using a simple code to open a camera device and display the image:

EOF !

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

                                                            UPDATE

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

I Got some issues building GTK when following the procedure above, so here is the deal to avoid any headache:

1. Install X11        (package list)
2. Install GTK+    (package list)
3. add the LIBFFI and V4L-UTILS like described earlier.
4. Upgrade the GLIB2 package like described earlier.
5. Cross Compile the OpenCV Library.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

EOF² !