Onboard Camera - V4L wrapper with Yocto Project

Based on my last post, there are a lot of people having problems to use this wrapper in Yocto, since it uses a kernel header mxcfb.h and it differs a bit from LTIB, so this post is to help solving this issue.

We have a nice tool on linux that most people don't use it (myself included) and can do the job of dealing with dependencies like a charm ! This tool is called AUTOTOOLS, yes, that one which create the pratical and easy ./configure ./make and ./make install system. Below I will demonstrate how we can use that to solve our issue:

STEP#1 - YOCTO BUILD CHANGES

We now need to add the kernel-dev to our local.conf (in build/conf) file and it will look like:

     BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
     PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
     MACHINE ??= 'imx6qsabresd'
     DISTRO ?= 'poky'
     PACKAGE_CLASSES ?= "package_rpm"
     EXTRA_IMAGE_FEATURES = "debug-tweaks"
     USER_CLASSES ?= "buildstats image-mklibs image-prelink"
     PATCHRESOLVE = "noop"
     BB_DISKMON_DIRS = "\
         STOPTASKS,${TMPDIR},1G,100K \
         STOPTASKS,${DL_DIR},1G,100K \
         STOPTASKS,${SSTATE_DIR},1G,100K \
         ABORT,${TMPDIR},100M,1K \
         ABORT,${DL_DIR},100M,1K \
         ABORT,${SSTATE_DIR},100M,1K" 
     PACKAGECONFIG_pn-qemu-native = "sdl"
     ASSUME_PROVIDED += "libsdl-native"
     CONF_VERSION = "1"

     BB_NUMBER_THREADS = '4'
     PARALLEL_MAKE = '-j 4'

     DL_DIR ?= "${BSPDIR}/downloads/"
    ACCEPT_FSL_EULA = ""

     DISTRO_FEATURES_remove = "x11 wayland"

     CORE_IMAGE_EXTRA_INSTALL += "gpu-viv-bin-mx6q gpu-viv-bin-mx6q-dev kernel-dev"


STEP #2 - REBUILDING YOCTO

     $ bitbake core-image-base


STEP#3 - POSSIBLE ISSUE

     if the mxcfb.h file doens't appear in your sysroot (/opt/poky/1.6.1/sysroots/cortexa9hf-vfp-neon-poky-linux-gnueabi/) then you need to build your toolchain and then reinstall it as well:

     rebuild with -c populate:

          $ cd fsl-community-bsp/build

          $ bitbake core-image-base -c populate_sdk

     and then install:

          $ cd /fsl-community-bsp/build/tmp/deploy/sdk
          $ ./poky-eglibc-x86_64-core-image-base-cortexa9hf-vfp-neon-toolchain-1.6.1.sh

STEP #4 - MODIFYING THE APP TO SUPPORT AUTOTOOLS

     Now is the interesting part, we will need 2 different files (pretty simple ones actually), they are: Makefile.am and configure.ac.

     Based on my projects configuration, I like to have bin, include and src folders, so the sample below will be on those:


     Project Tree:
    
             Project folder
                        |
a)                     ----- Makefile.am
                        |
b)                     ----- configure.ac
                        |
                        ----- bin /
                        |
                        ----- src /
                        |         |
c)                     |         ------ Makefile.am
                        |         |
                        |         ------ camera_test.c
                        |         |
                        |         ------ v4l_wrapper.c
                        |
                        ----- include /
                                  |
                                  ------ v4l_wrapper.h

    a) Makefile.am

             AUTOMAKE_OPTIONS = foreign
        SUBDIRS = src          # add just those ones that will have a Makefile inside.

    b) configure.ac
         
             AC_INIT([camera_test], [0.1], [andre.silva@freescale.com])
             AM_INIT_AUTOMAKE([-Wall -Werror foreign])
             AC_PROG_CC
             AC_CONFIG_HEADERS([config.h])
             AC_CHECK_HEADERS([fcntl.h stdint.h stdlib.h string.h sys/ioctl.h unistd.h])

             # Checks for library functions.
             AC_FUNC_MMAP
             AC_CHECK_FUNCS([gettimeofday memset munmap])

             ##########################################################################
             # Checks for programs needed for tests
             ##########################################################################

             AC_CONFIG_FILES([Makefile
             src/Makefile])

             AC_OUTPUT


    c) Makefile.am

             bin_PROGRAMS = camera_test

             camera_test_SOURCES = camera_test.c ../include/v4l_wrapper.h
             nodist_camera_test_SOURCES = ../src/v4l_wrapper.c

             ROOTFS_DIR = $(SDKTARGETSYSROOT)

             TARGET_PATH_LIB = $(ROOTFS_DIR)/usr/lib
             TARGET_PATH_INCLUDE = $(ROOTFS_DIR)/usr/include

             AM_CPPFLAGS = -I $(prefix)/usr/src/kernel/include/uapi -I $(prefix)/usr/src/kernel/include/ -I $(TARGET_PATH_INCLUDE) -I ../include

             AM_LDFLAGS = -Wl,--library-path=$(TARGET_PATH_LIB),-rpath-link=$(TARGET_PATH_LIB) -lm  -L $(prefix)/usr/lib 
     
             # workaround to get the binaries copied to local /bin folder
             all:
                       mv ../src/$(bin_PROGRAMS) ../bin

             clean-local: 
                        rm -rf ../bin/*
                  rm -rf ../src/*.o


     * if you are not going to use any additional source you can only let the camera_test_SOURCES = camera_test.c


STEP #5 - GENERATING THE BUILD FILES

     In a clean terminal enter the following commands to generate the files:

          $ autoheader

          $ autoreconf --install

     it will create all the necessary files to build your application, including dealing with dependencies, even if they are from kernel.

STEP #6 - BUILDING THE APPLICATION

     For building any application using yocto (unless you have a recipe for it), you have to export the toolchain environment variables, and once it is done, the terminal you have used is now dirty and if you want to do all the steps above again, you will need another one (cleaned).

          $ cd /opt/poky/1.6.1/
          $ .  ./environment-setup-cortexa9hf-vfp-neon-poky-linux-gnueabi
          $ cd /home/usr/v4l_wrapper_yocto/
          $ ./configure --host=arm-poky-linux-gnueabi --prefix=$SDKTARGETSYSROOT
          $ make

     At this point you will have your binary placed in bin/ folder, if you enter the make install command it will install (copy) the application binary into your rootfs/bin, where the rootfs is defined by the --ṕrefix in ./configure line.


THE RESULT

The sample code can be found here.

Thanks to Rogério Pimentel who helped me with the autotools and how to solve the mxcfb.h dependency issue and Daiane Angolini with YOCTO.

EOF !

Onboard Camera - V4L wrapper for use with OpenCV

Many people have been asking how to use the onbard camera (CSI and MIPI that comes with the Freesacale's development board) with OpenCV.

Unfortunately the mxc driver for this camera is not compatible with OpenCV and there is no way to use it directly, instead, we can use wrapper functions to access it. In this post I will share with you guys some utility functions I created (based on an Ipu demo app from Rogerio Pimentel - Freescaler) that uses V4L to access the camera.

When using these functions we will get frames on UYVY format (YUV422) and to use with OpenCV it must be converted to RGB24 (24bpp), all these are included in the source code.

The code can be downloaded from here, it also comes with a sample application, you just need to export the ROOTFS envioroment variable and check if the toolchain you are going to use is the same as defined in the Makefile, otherwise you must change it too.

The demo application (tested on i.MX6 sabresd board) performs the Canny edge detector and displays it using cvShowImage, you could display it directly in the framebuffer using the utility function, just uncomment the functions in the code and also add the conversion from RGB888toYUV422.

to run this application, make sure to have these drivers installed:

modprobe ov5642_camera
modprobe ov5640_camera_mipi
modprobe mxc_v4l2_capture

here is the result:

ps:  There is a "small" issue in these wrapper functions, as it uses multiple buffers for capturing and output (if you use it instead cvShowImage), you need to at least use the  V4LWrapper_QueryFrame function 3 times in a row to fill all buffers and the display it, so, the best scenario would be using this function like cvQueryFrame, when I got a fix for this I will update this post. Meanwhile just throw this function inside a loop function and you are good to go =)

EOF !

Building OpenCV-2.4.X for Freescale's i.MX6 BSP (Yocto)

Lately a lot of people are working with the Yocto Project and many of them migrated from LTIB (like me). Yocto uses a different conception when adding new packages/applications to the system, now everything is based on RECIPES. As it is being highly used, the amount of packages (recipes) already included is very big and it keep increasing continuously. For our lucky the recipe for OpenCV is already there, we just need to configure the system in order to add it to us.

In order to get everything up running we will divide de taks in steps:


Step #1 - Installing Yocto
--------------------------

As our focus is to install OpenCV, the Yocto install procedure we can use this very good tutorial created by Daiane: https://community.freescale.com/docs/DOC-94849


Step #2 - Enabling OpenCV
----------------------------

As we already have the OpenCV recipe in our Yocto release, we just need to add what packages we want in our local.conf file, located at /yocto/fsl-community-bsp/build/conf. With some modification (opencv package), it should look like this:

    MACHINE ??= 'imx6qsabresd'
    DISTRO ?= 'poky'
    PACKAGE_CLASSES ?= "package_rpm"
    EXTRA_IMAGE_FEATURES = "debug-tweaks dev-pkgs"
    USER_CLASSES ?= "buildstats image-mklibs image-prelink"
    PATCHRESOLVE = "noop"
    BB_DISKMON_DIRS = "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K" 
    PACKAGECONFIG_pn-qemu-native = "sdl"
    ASSUME_PROVIDED += "libsdl-native"
    CONF_VERSION = "1"

    BB_NUMBER_THREADS = '4'
    PARALLEL_MAKE = '-j 4'

    DL_DIR ?= "${BSPDIR}/downloads/"
    ACCEPT_FSL_EULA = ""

    CORE_IMAGE_EXTRA_INSTALL += "gpu-viv-bin-mx6q gpu-viv-bin-mx6q-dev"
    CORE_IMAGE_EXTRA_INSTALL += "libopencv-core-dev libopencv-highgui-dev
libopencv-imgproc-dev libopencv-objdetect-dev libopencv-ml-dev"

    LICENSE_FLAGS_WHITELIST = "commercial"


Note that we included the "-dev" packages, this is necessary if you always want to have the OpenCV headers/libraries included in the rootfs, Yocto is smart if you don´t add a "-dev" package and the libraries are just included any application that uses it needs to be built. As we always want our OpenCV stuff to build our applications so we use it this way.


Step #3 - Building OpenCV
----------------------------

Now the easy part:

/yocto/fsl-community-bsp/build$./bitbake core-image-x11

after build is finished you can check the images generated by the bitbake command at:

/build/tmp/deploy/images/imx6qsabresd/

and after extracting the rootfs: core-image-x11-imx6qsabresd.tar.bz2, you can find the opencv libraries in the /usr/lib folder:

andre@b22958:~/bsps/yocto/rootfs$ ls usr/lib/libopen*
usr/lib/libopencv_calib3d.so           usr/lib/libopencv_ml.so
usr/lib/libopencv_calib3d.so.2.4       usr/lib/libopencv_ml.so.2.4
usr/lib/libopencv_calib3d.so.2.4.7     usr/lib/libopencv_ml.so.2.4.7
usr/lib/libopencv_contrib.so           usr/lib/libopencv_nonfree.so
usr/lib/libopencv_contrib.so.2.4       usr/lib/libopencv_nonfree.so.2.4
usr/lib/libopencv_contrib.so.2.4.7     usr/lib/libopencv_nonfree.so.2.4.7
usr/lib/libopencv_core.so              usr/lib/libopencv_objdetect.so
usr/lib/libopencv_core.so.2.4          usr/lib/libopencv_objdetect.so.2.4
usr/lib/libopencv_core.so.2.4.7        usr/lib/libopencv_objdetect.so.2.4.7
usr/lib/libopencv_features2d.so        usr/lib/libopencv_ocl.so
usr/lib/libopencv_features2d.so.2.4    usr/lib/libopencv_ocl.so.2.4
usr/lib/libopencv_features2d.so.2.4.7  usr/lib/libopencv_ocl.so.2.4.7
usr/lib/libopencv_flann.so             usr/lib/libopencv_photo.so
usr/lib/libopencv_flann.so.2.4         usr/lib/libopencv_photo.so.2.4
usr/lib/libopencv_flann.so.2.4.7       usr/lib/libopencv_photo.so.2.4.7
usr/lib/libopencv_gpu.so               usr/lib/libopencv_stitching.so
usr/lib/libopencv_gpu.so.2.4           usr/lib/libopencv_stitching.so.2.4
usr/lib/libopencv_gpu.so.2.4.7         usr/lib/libopencv_stitching.so.2.4.7
usr/lib/libopencv_highgui.so           usr/lib/libopencv_superres.so
usr/lib/libopencv_highgui.so.2.4       usr/lib/libopencv_superres.so.2.4
usr/lib/libopencv_highgui.so.2.4.7     usr/lib/libopencv_superres.so.2.4.7
usr/lib/libopencv_imgproc.so           usr/lib/libopencv_video.so
usr/lib/libopencv_imgproc.so.2.4       usr/lib/libopencv_video.so.2.4
usr/lib/libopencv_imgproc.so.2.4.7     usr/lib/libopencv_video.so.2.4.7
usr/lib/libopencv_legacy.so            usr/lib/libopencv_videostab.so
usr/lib/libopencv_legacy.so.2.4        usr/lib/libopencv_videostab.so.2.4
usr/lib/libopencv_legacy.so.2.4.7      usr/lib/libopencv_videostab.so.2.4.7
andre@b22958:~/bsps/yocto/rootfs$

ps: don´t forget to flash the card with the image created at /tmp/deploy/images/imx6qsabresd/core-image-x11-imx6qsabresd.sdcard

$ sudo dd if= /build/tmp/deploy/images/imx6qsabresd/core-image-x11-imx6qsabresd.sdcard of=/dev/sdb

----------------------------------------

After those 3 steps above you should be able to find all the OpenCV headers/libraries needed by mostly of your application, but in any case you need more dev packages, you can look at: /tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/opencv/2.4.6+gitAUTOINC+1253c2101b-r0/packages-split

Now that you have the OpenCV headers/libraries we need the toolchain to build our sample application, just re-do the bitbake command now adding the "-c populate" option in the command line:

/yocto/fsl-community-bsp/build$./bitbake core-image-x11 -c populate_sdk

and then run the install script created at: /yocto/fsl-community-bsp/build/tmp/deploy/sdk to install it.

With that you will be able to see the toolchain installed at: /opt/poky

Now we are able to test our sample code, just a camera test and you can find the source code here: camera_test_sample

To build this application you need a new terminal window (all environment variables will be reset), then run the setup environment:

$ cd /opt/poky/1.5+snapshot/
$ . ./environment-setup-cortexa9hf-vfp-neon-poky-linux-gnueabi

and then go to the camera_test_yocto folder and type make. The binary will be placed in the bin folder.

Once flashed your card with the Yocto image (opencv included), mount the sd card in your host computer and then copy the binary to your rootfs.

To test it, run the application with the following command:

$ DISPLAY =:0 ./camera_test

EOF !