Skip to content

Home

Jump to bottom
Jean-Michel Gonet edited this page Jan 7, 2019 · 17 revisions

I assume that you know about coding but you are not familiar with C++, Gtk or OpenCV. It was my case when I started this little project, and I spent a huge amount of time discovering the specifics of this rich language and its dependencies. As this is not a C++ tutorial, I will only name the concepts and provide links to the explanations.

Step 1 - A single-window application

In this first step we build a very simple single-window application using CMake to configure the project and Gtk to display windows. Source code is available as master branch at: https://github.com/cpp-tutorial/raspberry-cpp-gtk-opencv. To retrieve it:

Configure the project with CMake

This is the CMakeLists.txt that goes on the src folder:

# src/CMakeLists.txt
cmake_minimum_required(VERSION 3.3 FATAL_ERROR)

# Project name and current version
project(rascam VERSION 0.1 LANGUAGES CXX)

# Enable general warnings
# See http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")

# Use 2014 C++ standard.
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

# Must use GNUInstallDirs to install libraries into correct locations on all platforms:
include(GNUInstallDirs)

# Pkg_config is used to locate headers and files for dependency libraries:
find_package(PkgConfig)

# Defines variables GTKMM_INCLUDE_DIRS, GTKMM_LIBRARY_DIRS and GTKMM_LIBRARIES.
pkg_check_modules(GTKMM gtkmm-3.0)

# Adds GTKMM_INCLUDE_DIRS, GTKMM_LIBRARY_DIRS to the list of folders the
# compiler and linker will use to look for dependencies:
link_directories( ${GTKMM_LIBRARY_DIRS} )
include_directories( ${GTKMM_INCLUDE_DIRS} )

# Declares a new executable target called rascapp
# and lists the files to compile for it:
add_executable(rascapp
    cpp/main.cpp    
    cpp/main-window.cpp
)

# Adds the folder with all headers:
target_include_directories(rascapp PRIVATE hpp)

# Link files:
target_link_libraries(rascapp
   ${GTKMM_LIBRARIES}  
)

Let's review the commands one by one:

  • cmake_minimum_required requires a minimum version of CMake. This allows you do confidently use the features existing in the specified version, or show an explicit error message if installed version is too old.
  • project declares the name of the global project, rascam, specifies the current version and list the languages that we are using, namely C++. Historically, the extension for C++ was *.c++, but in some context there are problems with «+», so they switched to cpp or cxx. In CMake, CXX means C++.
  • set(CMAKE_CXX...) sets a few flags that are appropriate for modern C++. In particular CMAKE_CXX_STANDARD configures which C++ standard we're going to use in the project. You could argue that 2014 is not that recent, and the latest C++ standard is from 2017. Sadly, the C++ compiler in Mac OS X is provided by XCode tools and of today (January 2019) they don't yet support the 2017 standard. In consequence, to keep the project cross platform compatible we have to keep on the 2014 standard.
  • find_package(PkgConfig) links to the pkg-config command, so we can use it later on.
  • pkg_check_modules(GTKMM gtkmm-3.0) does the same as executing pkg-config gtkmm-3.0 in the terminal (try it, if you wish), and then copies each section of the answer into variables prefixed with the specified GTKMM. Therefore, after this command we have three variables named GTKMM_LIBRARY_DIRS, GTKMM_INCLUDE_DIRS and GTKMM_LIBRARIES.
  • Then we declare a target. One project can have multiple targets. In general, targets comprise executables or libraries which are defined by calling add_executable or add_library and which can have many properties set (see distinction between a “target” and a “command”). As this is a single target project, it is perfectly convenient to define the target directly in the main CMakeLists.txt file. We list the source files to compile to create the target.
  • target_include_directories sets the list of folders where to look for the files referred in sources as include "...".
  • target_link_libraries sets the list of libraries the linker (see about what do linkers do) has to link to complete the build.

In case you want to read more about compilation steps:

The Gtk startup

All C++ executables need a main method as entry point. To launch a GtK application, you need to specify a main window, which will act as a base for all other user interaction, including to open more windows.

#include "main-window.hpp"

int main(int argc, char* argv[]) {
    auto app
        = Gtk::Application::create(argc,
                                   argv,
                                   "raspberry-cpp-gtk-opencv");

    MainWindow mainWindow(300, 300);
    return app->run(mainWindow);
}

Declaring a variable as auto specifies that the type of the variable that is being declared will be automatically deduced from its initialiser. This is called type inference.

The call to Gtk::Application::create initialises a new GtK execution context, passes the Program Arguments, and sets the application id (that can be anything, but convention says that it should contain your domain name in reverse, followed by its name, like in ch.agl-developpement.cpp-tutorial.raspberry-cpp-gtk-opencv)

Once you have a context ready you can use it to open a window. The window that can be an instance of any class that extends Gtk::Window. In this case our main window is called quite explicitly mainWindow and it is an instance of class MainWindow, that we will define later.

The way we define (see the difference between define and declare) the mainWindow variable makes it an automatic variable (not same as auto type inference modifier seen before). Automatic variables are initialised and allocated in memory at the point of definition, and uninitialised and deallocated automatically when their scope is ended (read more about scope). On a local variable, this happens at the closing '}' of their code block. We don't need to care about deleting automatic variables (see, in contrast, Dynamic memory allocation with new and delete, or also dynamic memory allocation here at geeks for geeks).

Whenever the type of variable is a class (as opposed to a basic type like int), the class constructor is called with the construction arguments and returns the initialised instance. In this case, construction arguments are the initial size of the window.

The call to run will return when the user closes the window. This gives to Gtk execution context the opportunity to control the exit status.

The basic GtK Window

In Gtk, all buttons, labels, checkboxes and elements that displays or interacts with the user are called Widgets. Here is a gallery of Widgets.

Then, gtkmm adds its own sauce:

gtkmm arranges widgets hierarchically, using containers. A Container widget contains other widgets. Most gtkmm widgets are containers. Windows, Notebook tabs, and Buttons are all container widgets. There are two flavours of containers: single-child containers, which are all descendants of Gtk::Bin, and multiple-child containers, which are descendants of Gtk::Container. Most widgets in gtkmm are descendants of Gtk::Bin, including Gtk::Window

See Gtkmm official documentation.

MainWindow class declaration - the header

class MainWindow : public Gtk::Window declares MainWindow as a class, and as descendant of, or inheriting from, Gtk::Window. The public modifier makes all public members of the ancestor to be also public members of the descendant. Public inheritance is the most usual way of extending a class (in contrast to private inheritance). We are providing MainWindow to the Gtk execution context, which is expecting all functionalities of a Gtk::Window to be available, so we avoid restricting their access.

As a descendant of Gtk::Window, MainWindow is a single-child container that can only contain one Gtk::Widget. I choose it to be a Gtk::Box, which is a Gtk::Container so, in turn, it is able to contain several widgets. In this occasion I want them to be one Gtk::Button and two Gtk::Label, to illustrate the Packing mechanism that GtK uses to stack together a bunch of controls. For MainWindow to respond to clicks on the button, we declare a buttonClick method (could be any other name, but keep them meaningful) -- I'm explaining later how to connect the 'click' signal to it.

#ifndef MAIN_WINDOW_H
#define MAIN_WINDOW_H

#include <gtkmm.h>

class MainWindow : public Gtk::Window {
public:
    MainWindow(int width, int height);
    virtual ~MainWindow() = default;

private:
    void buttonClick();
    Gtk::Button m_button;
    Gtk::Box m_box;
    Gtk::Label m_label1, m_label2;
};

#endif

The two first methods are the constructor and the destructor. As said before, constructor is called every time we declare a new instance of this class. This constructor accepts initial width and height of the window as parameters.

Whenever an instance is destroyed, the class destructor is called to perform all needed clean up tasks. This destructor has two important modifiers:

  • virtual means that, in case of inheritance, the method called will be that of the dynamic type. Read more about what is dynamic type of object, virtual functions, and why most of the time it is a good idea to have virtual destructors.
  • default means that we define this destructor to have a default implementation. The default implementation of a destructor is able to deallocate all defined members of the class, which are m_box, m_button, m_label1 and m_label2 (code>buttonClick is a method therefore it does not need allocation or deallocation). By specifying the default implementation, we spare ourselves the need to define the destructor.

The memory model of C++ has no garbage collector. Reserving and freeing memory is relies on the presence of class constructors and destructors and on a strategy called Resource Acquisition Is Initialisation, or RAII.

MainWindow class definition - the source

Declaration of MainWindow requires two further definitions - the class constructor and the buttonClick method. Here is the code:

#include "main-window.hpp"
#include <iostream>

MainWindow::MainWindow(int witdh, int height): 
m_button("Hello World"), 
m_box(Gtk::ORIENTATION_VERTICAL), 
m_label1("First Label"), 
m_label2("Second Label") {
    // Configure this window:
    this->set_default_size(witdh, height);

    // Connect the 'click' signal and make the button visible:
    m_button.signal_clicked().connect(
        sigc::mem_fun(*this, &MainWindow::buttonClick));
    m_button.show();

    // Make the first label visible:
    m_label1.show();

    // Make the second label visible:
    m_label2.show();

    // Pack all elements in the box:
    m_box.pack_start(m_button, Gtk::PACK_SHRINK);
    m_box.pack_start(m_label1, Gtk::PACK_SHRINK);
    m_box.pack_start(m_label2, Gtk::PACK_EXPAND_WIDGET);

    // Add the box in this window:
    add(m_box);

    // Make the box visible:
    m_box.show();
}

void MainWindow::buttonClick() {
    std::cout << "Hello World" << std::endl;
}

The full name of the constructor is MainWindow::MainWindow, meaning it is the function called MainWindow (the name of the constructor is the same as the class) that belongs to the class MainWindow. It takes width and height as parameters, and starts by calling the constructors of all declared members (read more about constructor member initialiser lists).

Then we set the window's default size by calling [this->set_default_size](https://developer.gnome.org/gtk3/stable/GtkWindow.html#gtk-window-set-default-size). The this is a pointer to the current class instance. As it is a pointer, we use -> to access its members. As class MainWindow extends Gtk::Window, this instance contain all public or protected members of this class and those of the the parent class. This is called inheritance. The use of this pointer is actually optional, and the following snippet would also be valid:.

// Configure this window (now, without the 'this'):
set_default_size(witdh, height);

Next we want to react to the m_button being clicked. One way to achieve this is to connect a function to the click signal of the m_button instance (to see all available signals of a widget, you can check the official documentation, for example, the Gtk::Button has only the clicked signal). Connecting functions to widget's signals allows to process the event from anywhere in the code, as long as you have access to the widget and you can provide the address of the function.

m_button.signal_clicked().connect([address of the method to call]);

The address of the function could be as simple as placing the function's name (see about the address of a function). Our case is more complex because the function we want to specify is a member of a class. Member functions need to be provided with the this pointer, so it can access other members of the same instance. To build the pointer to a method of a specific instance, we use sigc::mem_fun:

sigc::mem_fun(                    // Convert member function to function pointer.
   *this,                         // The address of the instance.
   &MainWindow::buttonClick   // The address of the method.
);

By default, widgets are not visible. We need to set their visibility explicitly by calling show(). We do this for all widgets, including the m_box.

Next, we pack all elements in the box. During initialisation we already set the box to Gtk::ORIENTATION_VERTICAL, which means that widgets will be positioned one below the other, using all available horizontal space. pack_start adds a new widget to the box, Gtk::PACK_SHRINK specifies that the widget's vertical size should be as small as possible (Gtk::ORIENTATION_VERTICAL already specifies that all widgets in the box should have a width as big as possible, so the result will be widgets very wide and very flat.)

In contrast, Gtk::PACK_EXPAND_WIDGET specifies that the widget should take all available space. This makes the m_label2 to expand when the user expands the window (you can test this shortly).

The last step is to make the m_box itself visible.

As for the buttonClick method, we just display a message to the standard output.

Step 2 - A bit of refinement

This second step adds a couple of small improvements to the application. First one is to react to key events. As the application will be on the Raspberry Pi, and not always easily accessible via a mouse, it may be handy to have some keyboard shortcuts, in particular to toggle full size window or close application. The second improvement is to log events in the system log. Applications that run from command line can cout messages that are directly visible. However, if you launch your window application from a user desktop, you will not see the console, nor the result of cout.

Logging to syslog

As window application are usually launched from the desktop, the content produced via cout is not easily visible. Instead, you can submit syslog messages (see also a syslog example). It is quite easy to do, and works across all platforms.

In Linux (Ubuntu), you can see the system logs via the System Log application. In Mac OS X it is the Console application. In Windows you can use the Event viewer.

Reacting to keyboard events

Another way to react to events is to override a specific function in the Widget, that is called whenever that event happens. For example, to react to keyboard events in the MainWindow we have to override the on_key_press_event method. To override a function in C++:

  • The function has to be declared as virtual in one of the ancestors of our class.
  • We need to declare it again, in our class, with the exact same signature and accessibility.
  • We may add the override keyword on the function declaration, to acknowledge the fact that we're overriding an existing method from a parent class. The compiler shows an error when we use override on a method that doesn't exist or it is not virtual.

on_key_press_event is declared in class Gtk::Widget as follows:

/// This is a default handler for the signal signal_key_press_event().
virtual bool on_key_press_event(GdkEventKey* key_event);

In consequence, we have to re-declare it our class.

#ifndef MAIN_WINDOW_H
#define MAIN_WINDOW_H

#include <gtkmm.h>

#include "camera-drawing-area.hpp"

class MainWindow : public Gtk::Window {
public:
    MainWindow(int width, int height);
    virtual ~MainWindow() = default;

protected:
    bool on_key_press_event(GdkEventKey* event) override;

private:
    void buttonClick();
    bool probablyInFullScreen;
    Gtk::Button m_button;
    Gtk::Box m_box;
    Gtk::Label m_label1;
    CameraDrawingArea cameraDrawingArea;
};

#endif

The Widget where the event is originated is the first to receive it on its on_xx_xxx_event() event handler. If it returns true, the event is considered handled and nothing more is done about it. On the contrary, if the handler cannot do anything about the event, it returns false making the event to be offered to the same handler of its parent widget, and parent's parent, and so on (see about event propagation in the official documentation). The default implementation of handlers like on_key_press_event is to do nothing but returning false, so events are propagated unless otherwise specified.

[Ctrl] + [C] exits the application, [F] or [f] toggle the full screen mode, and [esc] turns the full mode off:

#include "main-window.hpp"
#include <syslog.h>

MainWindow::MainWindow(int witdh, int height)
    : probablyInFullScreen(false), m_button("Hello World"), m_box(Gtk::ORIENTATION_VERTICAL), m_label1("First Label")
{
    // Configure this window:
    this->set_default_size(witdh, height);

    // Connect the 'click' signal and make the button visible:
    m_button.signal_clicked().connect(
        sigc::mem_fun(*this, &MainWindow::buttonClick));
    m_button.show();

    // Make the first label visible:
    m_label1.show();

    // Make the second label visible:
    cameraDrawingArea.show();

    // Pack all elements in the box:
    m_box.pack_start(m_button, Gtk::PACK_SHRINK);
    m_box.pack_start(m_label1, Gtk::PACK_SHRINK);
    m_box.pack_start(cameraDrawingArea, Gtk::PACK_EXPAND_WIDGET);

    // Add the box in this window:
    add(m_box);

    // Make the box visible:
    m_box.show();

    // Activate Key-Press events
    add_events(Gdk::KEY_PRESS_MASK);
}

void MainWindow::buttonClick()
{
    syslog(LOG_NOTICE, "Hello world!");
}

bool MainWindow::on_key_press_event(GdkEventKey* event)
{
    switch (event->keyval) {
    // Ctrl + C: Ends the app:
    case GDK_KEY_C:
    case GDK_KEY_c:
        if ((event->state & GDK_CONTROL_MASK) == GDK_CONTROL_MASK) {
            get_application()->quit();
        }
        return true;

    // [F] toggles fullscreen mode:
    case GDK_KEY_F:
    case GDK_KEY_f:
        if (probablyInFullScreen) {
            unfullscreen();
            probablyInFullScreen = false;
        }
        else {
            fullscreen();
            probablyInFullScreen = true;
        }
        return true;

    // [esc] exits fullscreen mode:
    case GDK_KEY_Escape:
        unfullscreen();
        probablyInFullScreen = false;
        return true;
    }

    return false;
}

By default, the events are not captured. add_events(Gdk::KEY_PRESS_MASK) activates the capture of keyboard events.

The functions to toggle full screen are fullscreen() and unfullscreen(), both belong to the Gtk::Window class (here we're omitting the this pointer). The documentation warns that you shouldn’t assume the window is definitely full screen afterward, because other entities (e.g. the user or window manager) could toggle it on or off again, and not all window managers honor those requests. That's why we have added a probablyInFullScreen status, that conveys the possibility that window is not in the expected state.

To exit the application we are not directly calling exit(), instead we use get_application()->quit() to tell the Gtk execution context to call exit. It will, eventually, but first it is going to close all windows and open devices for us.

Step 3 - Display camera capture - Rough way

To capture images we're using OpenCV. If you just want to capture an image from the camera, you could use other libraries, but the final objective is to apply computer vision algorithms to the captured image. Hence, OpenCV is the best alternative.

Bundling Mac OS X application

For security reasons, Mac OS X forces applications to request user's permission before using the camera. The procedure for this is to include a Info.plist file with the necessary content:

  • Some indications about version number and author.
  • A key NSCameraUsageDescription with the description on why your application needs the camera. The message you write here is displayed to the user, who has to accept it. If he doesn't accept, the system terminates your application.

The procedure is to include a MacOSXBundleInfo.plist.in file, which is a template; some of the entries can be governed by properties in the CMakeLists.txt file, and others you can put yourself. The file is placed in src/res folder and has the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC 
    "-//Apple Computer//DTD PLIST 1.0//EN" 
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleDevelopmentRegion</key>
    <string>English</string>
    <key>CFBundleExecutable</key>
    <string>${MACOSX_BUNDLE_EXECUTABLE_NAME}</string>
    <key>CFBundleGetInfoString</key>
    <string>${MACOSX_BUNDLE_INFO_STRING}</string>
    <key>CFBundleIconFile</key>
    <string>${MACOSX_BUNDLE_ICON_FILE}</string>
    <key>CFBundleIdentifier</key>
    <string>${MACOSX_BUNDLE_GUI_IDENTIFIER}</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleLongVersionString</key>
    <string>${MACOSX_BUNDLE_LONG_VERSION_STRING}</string>
    <key>CFBundleName</key>
    <string>${MACOSX_BUNDLE_BUNDLE_NAME}</string>
    <key>CFBundlePackageType</key>
    <string>APPL</string>
    <key>CFBundleShortVersionString</key>
    <string>${MACOSX_BUNDLE_SHORT_VERSION_STRING}</string>
    <key>CFBundleSignature</key>
    <string>????</string>
    <key>CFBundleVersion</key>
    <string>${MACOSX_BUNDLE_BUNDLE_VERSION}</string>
    <key>CSResourcesFileMapped</key>
    <true/>
    <key>NSHumanReadableCopyright</key>
    <string>${MACOSX_BUNDLE_COPYRIGHT}</string>
    <key>NSCameraUsageDescription</key>
    <string>This app requires to access your camera to retrieve images and perform the demo</string>
</dict>
</plist>

These are some of the resources I used to found out how to do bundles in Mac OS X:

Linking to OpenCV

After installing OpenCV following the previous instructions, you can link your application to it in CMake.

# src/CMakeLists.txt
cmake_minimum_required(VERSION 3.3 FATAL_ERROR)

# Project name and current version
project(rascam VERSION 0.1 LANGUAGES CXX)

# Enable general warnings
# See http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")

# Use 2014 C++ standard.
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

# Must use GNUInstallDirs to install libraries into correct locations on all platforms:
include(GNUInstallDirs)

# Pkg_config is used to locate header and files for dependency libraries:
find_package(PkgConfig)

# Defines variables GTKMM_INCLUDE_DIRS, GTKMM_LIBRARY_DIRS and GTKMM_LIBRARIES.
pkg_check_modules(GTKMM gtkmm-3.0) 
link_directories( ${GTKMM_LIBRARY_DIRS} )
include_directories( ${GTKMM_INCLUDE_DIRS} )

# OpenCV can be linked in a more standard manner:
find_package( OpenCV REQUIRED )

# Compile files:
add_executable(rascapp
    cpp/main.cpp
    cpp/main-window.cpp
    cpp/camera-drawing-area.cpp
)

# Add folder with all headers:
target_include_directories(rascapp PRIVATE hpp)

# Link files:
target_link_libraries(rascapp
    ${GTKMM_LIBRARIES}
    ${OpenCV_LIBS}
)

# Apple requires a bundle to add a Info.plist file that contains the required
# permissions to access some restricted resources like the camera:
if (APPLE)
    set_target_properties(rascapp PROPERTIES
        MACOSX_BUNDLE TRUE
        MACOSX_FRAMEWORK_IDENTIFIER org.cmake.ExecutableTarget
        MACOSX_BUNDLE_INFO_PLIST ${CMAKE_CURRENT_SOURCE_DIR}/res/MacOSXBundleInfo.plist.in
    
        # This property is required:
        MACOSX_BUNDLE_GUI_IDENTIFIER "rascapp-${PROJECT_VERSION}"
    
        # Those properties are not required:
        MACOSX_BUNDLE_INFO_STRING "rascapp ${PROJECT_VERSION}, by jmgonet@agl-developpement.ch"
        MACOSX_BUNDLE_LONG_VERSION_STRING ${PROJECT_VERSION}
        MACOSX_BUNDLE_BUNDLE_NAME "rascapp"
        MACOSX_BUNDLE_SHORT_VERSION_STRING ${PROJECT_VERSION}
        MACOSX_BUNDLE_BUNDLE_VERSION ${PROJECT_VERSION}
    )
endif()

A graphic widget

Gtk::DrawingArea is a widget that holds a graphic area to display custom drawings or bitmaps. We define a CameraDrawingArea that extends this widget, and copies the image captured from the camera into the graphic area:

#ifndef CAMERA_DRAWING_AREA_H
#define CAMERA_DRAWING_AREA_H

#include <gtkmm.h>
#include <opencv2/highgui.hpp>

class CameraDrawingArea : public Gtk::DrawingArea {
public:
    CameraDrawingArea();
    virtual ~CameraDrawingArea();

protected:
    bool on_draw(const Cairo::RefPtr<Cairo::Context>& cr) override;
    void on_size_allocate(Gtk::Allocation& allocation) override;

    bool everyNowAndThen();

private:
    sigc::connection everyNowAndThenConnection;
    cv::VideoCapture videoCapture;
    cv::Mat webcam;
    cv::Mat output;
    int width, height;
};
#endif

We're going to extend this class and override two of its methods:

  • void on_size_allocate(Gtk::Allocation& allocation) - This method is called every time the size of the widget changes. This happens the very first time it is displayed, and every time some action of the user or the system makes the size change.
  • bool on_draw(const Cairo::RefPtr& cr) - This method is called every time the area, or part of the area, contained in the widget has to be redrawn. It receives a reference to a Cairo context. The method can use it to render any drawing or graphic. In our case we're going to use it to copy the image captured from the camera.

The cv::Mat is a class that contains a OpenCV image. We're using two of those: one contains the image captured from the camera, and the other contains the image resized to the Widget current size.

The width and height contain the current Widget size.

VideoCapture is a OpenCV access to the video camera.

everyNowAndThenConnection is a way to call everyNowAndThen() method at regular time intervals, similarly as we set the response to the button click, in previous steps.

This is the

#include "opencv2/core.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"

#include "camera-drawing-area.hpp"

CameraDrawingArea::CameraDrawingArea()
    : videoCapture(0)
{
    // Lets refresh drawing area very now and then.
    everyNowAndThenConnection = Glib::signal_timeout().connect(sigc::mem_fun(*this, &CameraDrawingArea::everyNowAndThen), 100);
}

CameraDrawingArea::~CameraDrawingArea()
{
    everyNowAndThenConnection.disconnect();
}

/**
 * Every now and then, we invalidate the whole Widget rectangle,
 * forcing a complete refresh.
 */
bool CameraDrawingArea::everyNowAndThen()
{
    auto win = get_window();
    if (win) {
        Gdk::Rectangle r(0, 0, width, height);
        win->invalidate_rect(r, false);
    }

    // Don't stop calling me:
    return true;
}

/**
 * Called every time the widget has its allocation changed.
 */
void CameraDrawingArea::on_size_allocate(Gtk::Allocation& allocation)
{
    // Call the parent to do whatever needs to be done:
    DrawingArea::on_size_allocate(allocation);

    // Remember the new allocated size for resizing operation:
    width = allocation.get_width();
    height = allocation.get_height();
}

/**
 * Called every time the widget needs to be redrawn.
 * This happens when the Widget got resized, or obscured by
 * another object, or every now and then.
 */
bool CameraDrawingArea::on_draw(const Cairo::RefPtr<Cairo::Context>& cr)
{

    // Prevent the drawing if size is 0:
    if (width == 0 || height == 0) {
        return true;
    }

    // Capture one image from camera:
    videoCapture.read(webcam);

    // Resize it to the allocated size of the Widget.
    resize(webcam, output, cv::Size(width, height), 0, 0, cv::INTER_LINEAR);

    // Initializes a pixbuf sharing the same data as the mat:
    Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(
        (guint8*)output.data,
        Gdk::COLORSPACE_RGB,
        false,
        8,
        output.cols,
        output.rows,
        (int)output.step);

    // Display
    Gdk::Cairo::set_source_pixbuf(cr, pixbuf);
    cr->paint();

    // Don't stop calling me.
    return true;
}

In the constructor we start by initialising videoConstructor to the default device. Then we set up a connection to call the everyNowAndThen method every 100ms (10 x per second). Later we call the connection to disconnect it during the destruction process.

The everyNowAndThen() method invalidates the whole area of the widget. This is an indirect way to provoke a call to on_draw.

on_size_allocate keeps track of the current size of the Widget. Just in case the parent class did implement something important in its own implementation, we call the parent method.

Everything we put in place before is so on_draw is called regularly, and we can use the reference to Cairo::Context to paste the image captured from the camera:

  • First we check that the current width and height are not zero. This may happen during startup phase. Calling resize with a 0 dimension would terminate instantly the application so we better avoid it.
  • videoCapture.read(webcam) copies the captured image into the provided cv::Mat, in this case webcam. If the provided material is not initialised, or it is not of the appropriate size, then the method will perform necessary tasks, and reserve needed memory.
  • resize copies the image from source (webcam) to destination (output), changing its size. If the destination is not initialised or not of the right size it performs necessary tasks and memory reservation. To keep the variable in scope, we also declared it as a class property: first call will initialise the variable, next calls can just reuse it.
  • Both webcam and output are class properties so they keep in scope as long as the class instance does. They will need to be initialised only during the first call to on_draw but will keep their values across the subsequent calls. This helps noticeably the performance.
  • create_from_data creates a new Gdk::Pixbuf object, which contains an image in a format compatible with Cairo. The long list of parameters are to be taken as a recipe to convert OpenCV's Material to Gtk's Pixbuf.
  • set_source_pixbuf sets the bitmap source to use for next call to paint
  • Finally, paint does the painting.

Placing the drawing area

The last step is to place the widget we just created in the main window. We need to declare it in the MainWindow header:

#ifndef MAIN_WINDOW_H
#define MAIN_WINDOW_H

#include <gtkmm.h>

#include "camera-drawing-area.hpp"

class MainWindow : public Gtk::Window {
public:
    MainWindow(int width, int height);
    virtual ~MainWindow() = default;

protected:
    bool on_key_press_event(GdkEventKey* event) override;

private:
    void buttonClick();
    bool probablyInFullScreen;
    Gtk::Button m_button;
    Gtk::Box m_box;
    Gtk::Label m_label1;
    CameraDrawingArea cameraDrawingArea;
};

#endif

And place it in the box, just the same as a label or a button:

#include <syslog.h>
#include <unistd.h>

#include "main-window.hpp"

MainWindow::MainWindow(int width, int height)
    : probablyInFullScreen(false), m_button("Hello World"), m_box(Gtk::ORIENTATION_VERTICAL), m_label1("First Label")
{
    // Configure this window:
    this->set_default_size(width, height);

    // Connect the 'click' signal and make the button visible:
    m_button.signal_clicked().connect(
        sigc::mem_fun(*this, &MainWindow::buttonClick));
    m_button.show();

    // Make the first label visible:
    m_label1.show();

    // Make the second label visible:
    cameraDrawingArea.show();

    // Pack all elements in the box:
    m_box.pack_start(m_button, Gtk::PACK_SHRINK);
    m_box.pack_start(m_label1, Gtk::PACK_SHRINK);
    m_box.pack_start(cameraDrawingArea, Gtk::PACK_EXPAND_WIDGET);

    // Add the box in this window:
    add(m_box);

    // Make the box visible:
    m_box.show();

    // Activate Key-Press events
    add_events(Gdk::KEY_PRESS_MASK);
}

void MainWindow::buttonClick()
{
    syslog(LOG_NOTICE, "User %d says 'Hello World'", getuid());
}

bool MainWindow::on_key_press_event(GdkEventKey* event)
{
    switch (event->keyval) {
    // Ctrl + C: Ends the app:
    case GDK_KEY_C:
    case GDK_KEY_c:
        if ((event->state & GDK_CONTROL_MASK) == GDK_CONTROL_MASK) {
            get_application()->quit();
        }
        return true;

    // [F] toggles fullscreen mode:
    case GDK_KEY_F:
    case GDK_KEY_f:
        if (probablyInFullScreen) {
            unfullscreen();
            probablyInFullScreen = false;
        }
        else {
            fullscreen();
            probablyInFullScreen = true;
        }
        return true;

    // [esc] exits fullscreen mode:
    case GDK_KEY_Escape:
        unfullscreen();
        probablyInFullScreen = false;
        return true;
    }

    return false;
}

Conclusion

This example is a working proof of concept on how write a cross-platform application based on two very popular libraries, OpenCV for processing computer vision and Gtk for user interface. However, there are a list of shortcomings that need to be addressed before you can use it as a base for the more complex application that you may have in mind:

  • The more visible one is that the image is deformed to match the size of the window. This can be easily solved using a more sophisticated algorithm to calculate a size that would fit in the window but keep the original aspect ratio.
  • A second one, very annoying if you plan a First Person View application is the perceptible lag between real life and the images stream. Lag seems variable depending on the camera and the light conditions. The reason is cameras having an image buffer; as we are retrieving one image every now and then, we're always consuming the oldest image in the buffer. To solve this we should let the capture process drive the window refresh, instead of using a timer.
  • Code has a general lack of decoupling that makes everything very dependent on everything else. This defect isn't very present in such a simple application, but it will show as soon as we try to solve the lag problem. Also, part of the code should be a pure OpenCV process that you could copy/paste from some other blog, without needing to adapt it to the CameraDrawingArea that is dependent on both OpenCV and Gtk libraries.
  • Finally, you should be able to unit test the OpenCV processing to increase the type-compile-debug cycle.

My next article covers those topics.

Clone this wiki locally