A Node-Based Image, Video and Data Processing Tool

• Overview
• Installation
  • Using the Installer
  • Building from Source
• Quick Start
  • User
  • Developer
• References

Graph-ICS is a tool for visualizing and creating image, video, and data streams. A stream always consists of an input and a series of filter nodes. Graph-ICS provides out-of-the-box usage of many filtering algorithms from the libraries ITK, OpenCV, and Qt. Up to 10 different streams can be processed and displayed simultaneously. A core concept of Graph-ICS is to give its users the freedom to easily extend the available nodes for their desired use-case.

• Go to the Releases section and download the installer for your platform.
• You can find an installer for Windows, MacOS, and Linux.
• Extract the content and run the executable.

The linux binary for Graph-ICS was built and tested on Ubuntu 18.04 LTS. According to linuxdeployqt the binary should also work on newer systems and different distributions.

I tried running the binary on EndeavourOS (Arch-based), but did not get it to work.

The distributed mac binary was built on macOSX High Sierra 10.13.6 and cannot run on older systems.

If the Graph-ICS binary doesn't run on your system you can always build from source!

If you encounter a problem anywhere in the process look up Troubleshooting! Maybe the problem has already been solved for you :)

Worth Mentioning:

The guide walks thorugh the steps on how to build Graph-ICS for the Debug configuration. You just need to replace Debug with Release in all following commands to produce the Release configuration.

• Install Git
• Install Build Tools
• Install CMake
• Install Qt Library and QtCreator
• Build OpenCV
• Build ITK
• Build Graph-ICS
• Develop via QtCreator
• Plugins, Examples and Documentation (optional)
• Troubleshooting

• Download and install Git from the official website or via your package manager.

Windows:

• Download and install Visual Studio Community 2019 from here.
• Install Windows 10 SDK for C++ using the Visual Studio Installer.
• Make sure the compiler tools are in your PATH.
• e.g. C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64

Linux:

• Install gcc, g++ and make packages.

e.g. on Ubuntu use:

$ sudo apt install gcc g++ make

MacOS:

• Download and install XCode using the AppStore.

• Go to the official CMake download website.
• Select the binary distributions of the latest relase for your platform.
• Make sure you have a CMake version greater than 3.14 installed.

Note: On Linux you can also install CMake via your package manager.

Make sure cmake runs from the commandline and check your version:

$ cmake --version

• Download the Qt Online Installer for open-source.
• We need Qt 5.15.2.
• Make sure to install the binaries for the compiler you use (e.g. on Windows using Visual Studio Community 2019: MSVC2019, on Linux/MacOS: Desktop gcc).
• Make sure to install all Qt Modules to have no dependency issues.

Windows:

• Additionally install Qt Creator CDB Debugger Support and Debugging Tools for Windows.

• Open a commandline window.
• cd to a location where you want to store the library (e.g. C:/Libraries/) and execute the following commands.
• Pay close attention to the commands and execute them in the specified order!

Do not choose a path wich is too long! CMake paths are restricted to 50 characters by default.

$ mkdir opencv && cd opencv
$ git clone https://github.com/opencv/opencv.git src && cd src
$ git checkout tags/4.5.5 -b 4.5.5 && cd ..
$ mkdir build-4.5.5 && cd build-4.5.5

In the following you will see the -j8 option. This is to speed up the build by a large factor. Replace 8 with the number of threads your CPU supports.

Inside the /opencv/build-4.5.5 directory execute:

Windows:

$ cmake ../src -DWITH_FFMPEG=ON
$ cmake --build . -j8 --config Debug

Linux:

Before configuring OpenCV you need to make sure to have all needed packages installed.

To install the dependencies on Ubuntu run:

$ sudo apt install pkg-config ffmpeg libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libavresample-dev mesa-common-dev libglu1-mesa-dev

$ mkdir Debug && cd Debug
$ cmake ../../src -DCMAKE_BUILD_TYPE=Debug -DWITH_FFMPEG=ON

Make sure that OpenCV finds FFMPEG and the output in the Video I/O section looks something like this:

Video I/O:
--     DC1394:                      NO
--     FFMPEG:                      YES
--       avcodec:                   YES (57.107.100)
--       avformat:                  YES (57.83.100)
--       avutil:                    YES (55.78.100)
--       swscale:                   YES (4.8.100)
--       avresample:                YES (3.7.0)

cmake --build . -j8

MacOS:

$ mkdir Debug && cd Debug
$ cmake ../../src -DCMAKE_BUILD_TYPE=Debug
$ cmake --build . -j8

• Only configure and start the build after OpenCV has finished.
• I know the build process takes long but you can grab a hot drink ☕ and enjoy some fresh air while waiting :)
• The next steps will seem familiar to you.
• Open a commandline window.
• cd to a location where you want to store the library and execute the following commands.
• Pay close attention to the commands and execute them in the specified order!

Do not choose a path wich is too long! CMake paths are restricted to 50 characters by default.

$ mkdir itk && cd itk
$ git clone https://github.com/InsightSoftwareConsortium/ITK.git src && cd src
$ git checkout tags/v5.2.1 -b v5.2.1 && cd ..
$ mkdir build-v5.2.1 && cd build-v5.2.1

Make sure to adjust the path for the OpenCV_DIR option depending on your OpenCV build directory.

ITK has a known issue in v5.2.1 regarding the VideoBridgeOpenCV module and the testing of it. ITK will only compile successfully without testing so we need to specify the -DBUILD_TESTING=OFF option to build without errors.

Inside the /itk/build-v5.2.1 directory execute:

Windows:

$ cmake ../src -DOpenCV_DIR=<your path to>/opencv/build-4.5.5 -DModule_ITKVideoBridgeOpenCV=ON -DBUILD_TESTING=OFF
$ cmake --build . -j8 --config Debug

Linux and MacOS:

$ mkdir Debug && cd Debug
$ cmake ../../src -DCMAKE_BUILD_TYPE=Debug -DOpenCV_DIR=<your path to>/opencv/build-4.5.5/Debug -DModule_ITKVideoBridgeOpenCV=ON -DBUILD_TESTING=OFF
$ cmake --build . -j8

• Open a commandline window and cd to the directory where you want to store Graph-ICS.

$ git clone https://github.com/Graph-ICS/Graph-ICS.git && cd Graph-ICS
$ mkdir build && cd build

Inside the /Graph-ICS/build directory execute:

$ mkdir Debug && cd Debug

Make sure to adjust the path for the Qt5_DIR and ITK_DIR options depending on your Qt and ITK-build directories.

Windows:

We need to specify the CMAKE_BUILD_TYPE to automatically place shared libraries into the directory with the Graph-ICS executable. We also keep Debug and Release build in seperate directories in contrast to building OpenCV and ITK (on Windows) to import the build easier into QtCreator.

$ cmake ../.. -DCMAKE_BUILD_TYPE=Debug -DQt5_DIR=<your path to>/Qt/5.15.2/<compiler>/lib/cmake/Qt5 -DITK_DIR=<your path to>/itk/build-v5.2.1
$ cmake --build . -j8 --config Debug

Linux and MacOS:

$ cmake ../.. -DCMAKE_BUILD_TYPE=Debug -DQt5_DIR=<your path to>/Qt/5.15.2/<compiler>/lib/cmake/Qt5 -DITK_DIR=<your path to>/itk/build-v5.2.1/Debug
$ cmake --build . -j8

• Start QtCreator and select Open Project.
• Find your Graph-ICS source code and select the CMakeLists.txt.
• Deselect all kits.
• Select Import Build From, enter the location of your Debug build and click Import.
• Click Configure Project.
• Now you should be able to develop Graph-ICS using the QtCreator.

• Plugins
• Examples
• Documentation

Many errors in itkOpenCV...BridgeTest.cxx files?

• I had this issue on Linux and MacOS.
• Building ITK without testing worked for me.
• Append -DBUILD_TESTING=OFF to the cmake configuration and rebuild.

CMake can't find Qt?

• Try deleting all Qt related CMake cache entries (or your whole build directory) and setting Qt5_DIR cache variable to: your path to/Qt/5.15.2/your compiler/lib/cmake/Qt5

QtCreator shows errors in C++ code after importing the build but compiles without a problem?

• Try disabling the QtCreator plugin ClangCodeModel (worked for me).
• I found the solution in this stackoverflow post.

• UI Tour
• Adding Nodes
• Node Structure
• Tasks and Views
  • Edit your Video
  • Save your Image or Video
• Save your Configurations
• Customize your Favoritesbar
• Node Overview

1: Searchpanel

• Here you can find all available nodes and query them using the searchbar at the top.
• Drag & drop, doubleclick or use the context menu to create a node (more in Adding Nodes).

2: Tasks

• This area is used to control your defined tasks.
• Read more about Tasks in Tasks and Views.

3: Messages

• Warnings and messages are shown in the messages-tab.

4: Canvas

• All created nodes will be placed inside the canvas.
• Here you can connect and arrange your nodes to your liking.
• You can use standard editing methods like copy, paste, remove using shortcuts or the menubar's edit menu

5: Favoritesbar

• All your favorite nodes in one place, for quick-access.

6: Viewarea

• The output of a stream will be shown in one of the views.
• Connect the output of a node with the view where you want to display your stream.
• You can add and remove views using the buttons above (+) and below (-).
• More about views in Tasks and Views.

There are several ways to create nodes in Graph-ICS. Nodes can be created in the same way using the searchpanel or the favoritesbar. You can create nodes via:

• Doubleclick
• Context menu
• Drag & drop

• Drag & drop a file

Input nodes (Image and Video) can also be created by dragging and dropping a file (e.g. jpg or mp4) from the file explorer (or desktop) into the canvas.

A node can be either an input or a filter. Available inputs are Image, Video or Camera. Image and Video represent a file on your system. Camera represents a camera device on your system.

A node can have any number of input and output ports (not practical, but possible). Ports can be connected with each other to represent a data flow from one node to the other. Ports always indicate the transfered data with an icon (image or data).

Nodes have attributes to change the parameters of for example a filtering algorithm. You can change the values and check the processed result right after.

Tasks and Views take a special role in the processing of streams. A stream can only be processed if the output is connected to a view. That is when tasks come into play. A task gets created as soon as the output of a node is connected with a view. The type of the task depends on the input node of the stream. For example a Video node as input will always lead to the creation of a video task. The same can be applied to Camera and Image nodes.

Each view-task relation can be identified by a color.

After defining your tasks you can control and start processing the stream. You can control your tasks using the global or task specific options. Additionally you can select a view and use the Run menu in the menubar or the context menu in the view to control your task.

Views can be added and removed using the buttons above (+) and below (-) the viewarea. The views to date are highly inflexible and impractical. This topic will be adressed in a future release!

Videos or camera recordings can be edited (cut, copy, paste) by clicking the edit button (pencil icon) at the right side of a video or camera task.

You can select a range to cut out using the range slider below and select a position to paste frames using the slider above the regular frame slider. Use the buttons at the top to execute or undo/redo your editing.

The context menu in a view can be used to save your image or video to a file. Also you will be asked to save your video if you recorded a camera stream or edited your video so you don't accidentally loose your progress.

A node setup with all attribute values and connections can be saved to a config file (.gconfig) to persist your configuration work. You can use standard keyboard shortcuts or the file menu in the menubar to interact with a configuration.

To customize your favoritesbar you can use the context menu in the searchpanel or a node. Also you can press and hold an item in the searchpanel and drag & drop into the favoritesbar. Use the context menu or press and hold an item in the favoritesbar to move it. Remove nodes from the favorites using the context menu.

Input:

Filter:

Every contribution is appreciated! Make sure to read the contributing guide first.

This guide assumes you already have built Graph-ICS from source.

• Adding a New Filter Node
  • Creating the Class Files using QtCreator
  • Implementing the Filter Class
  • Registering the Filter
  • Creating a Custom Front-End (optional)

Graph-ICS aims to keep an open and easy interface to add new filter nodes. The following guide shows the steps that need and also could be done to add a new filter node. The example shows the creation of the CvSobelOperator node with a custom front-end.

QtCreator quickly generates a nice starting point for filter development and saves you much time writing the class definition.

• Right-click on app/src/model/opencv and select Add New... .
• Select C++, C++ Class and click on choose.
• Enter a class name, here CvSobelOperator.
• Add Node as the custom base class and select Add Q_OBJECT.
• Click next, then finish.
• Right-click on the project root and select Run CMake (or from the menu Build/Run CMake)

Header File:

#ifndef CVSOBELOPERATOR_H
#define CVSOBELOPERATOR_H

#include "node.h"

// Add to the Graph-ICS Namespace
namespace G 
{
// Derive from Node
class CvSobelOperator : public Node
{
    // Important Q_OBJECT Macro
    Q_OBJECT

public:
    explicit CvSobelOperator();

    // Pure virtual function
    // Filter functionality needs to go in here
    bool retrieveResult() override;
    // Virtual function
    // Define special behavior if an attribute value changed
    void onAttributeValueChanged(Attribute* attribute) override;

private:
    // Declare your in- and out-ports
    // You can have as many as you want and need
    Port m_inPort;
    Port m_outPort;

    // Declare the attributes
    // CvSobelOperator has two attributes to specify the edge detection direction 
    Attribute* m_xDerivative;
    Attribute* m_yDerivative;
};
} // namespace G

#endif // CVSOBELOPERATOR_H

Source File:

#include "cvsobeloperator.h"

#include <cv.h>
#include <opencv2/imgproc/imgproc.hpp>

// Wrap the definition into the namspace to avoid writing G::Foo everywhere
namespace G
{
CvSobelOperator::CvSobelOperator()
    // !!!Pass the classname to the Node constructor!!!
    : Node("CvSobelOperator")
    // Initialize the ports with a supported datatype (GIMAGE or GDATA)
    , m_inPort(Port::TYPE::GIMAGE)
    , m_outPort(Port::TYPE::GIMAGE)
    // The Attribute class is really generiy and can be customized to your liking
    // attributeFactory is a prototype factory for predefined Attribute instances
    // For simplicity we just use the predefined IntTextField-Attribute
    , m_xDerivative(attributeFactory.makeIntTextField(1, 0, 2, 1, "x-Derivative"))
    , m_yDerivative(attributeFactory.makeIntTextField(0, 0, 2, 1, "y-Derivative"))
{
    // Register your in-and out-ports to the port system
    registerPorts({&m_inPort}, {&m_outPort});
    // Register your attributes to the attribute system
    registerAttribute("xDerivative", m_xDerivative);
    registerAttribute("yDerivative", m_yDerivative);
}

bool CvSobelOperator::retrieveResult()
{
    try
    {
        // We know that the in-port has an image when entering this function
        cv::Mat& cvImage = m_inPort.getGImage()->getCvMatImage();
        
        // We need to make sure the image has only one channel
        if (cvImage.channels() > 1)
        {
            cv::cvtColor(cvImage, cvImage, CV_BGR2GRAY);
        }

        // Retrieve the values from the attributes to use them in the filtering
        int xDerivative = m_xDerivative->getValue().toInt();
        int yDerivative = m_yDerivative->getValue().toInt();

        // Do the filtering
        cv::Sobel(cvImage, cvImage, CV_8U, xDerivative, yDerivative);
        
        // Forward the result to the out-port
        m_outPort.getGImage()->setImage(cvImage);
    }
    catch (int e)
    {
        qDebug() << "CvSobelOperator. Exception Nr. " << e << '\n';
        return false;
    }

    return true;
}

void CvSobelOperator::onAttributeValueChanged(Attribute* attribute)
{
    // Clear the cached data in the output stream
    // Because an attribute value changed and the result needs to be calculated again  
    clearStreamCache();
    // The specification of the cv::Sobel function does not allow both parameters to be 0
    // This is why this function needs to make sure that the values are never both 0.
    if (attribute->getValue().toInt() == 0)
    {
        Attribute* other;
        if (attribute == m_xDerivative)
        {
            other = m_yDerivative;
        }
        else
        {
            other = m_xDerivative;
        }
        if (other->getValue().toInt() == 0)
        {
            other->setValue(1);
        }
    }
}
} // namespace G
}

• Open Nodes.cmake located in Graph-ICS/app and add your newly created node to the OPENCV_NODES variable

set(OPENCV_NODES
    CvDilation
    CvErosion
    CvFourierTransformPSD
    CvHistogramCalculation
    CvHistogramEqualization
    CvMedian
    CvTransformation
    # Our newly added Filter
    CvSobelOperator)

Now you just need to run CMake and build Graph-ICS.

You also can customize the node front-end if you don't like the auto generated look or want additional features. As an example we add a button to the node to swap the attribute values.

• Add a new .qml file to app/qml/node/custom
• The file needs to have the exact same name as the filter class (here: CvSobelOperator).

import QtQuick 2.15
import QtQuick.Layouts 1.15

// Import our Theme
import Theme 1.0

import "../"
import "../attribute"
import "../../components"

// Import the c++ model
import Model.CvSobelOperator 1.0

// GNode provides a Component interface that can be used for customization
// GNode provides all functionality and default Components
GNode {
    id: sobel
    // Instantiate the c++ model and assign it to the model property
    model: CvSobelOperatorModel {}

    // Override the background
    // Do not specitfy size or anchors of the Rectangle, as it will fit to the content
    background: Rectangle {
        color: Theme.primary
        radius: 8
        border.width: 3
        border.color: {
            if (isSelected) {
                return Theme.secondary
            }
            if (isHovered) {
                return Theme.secondaryLight
            }
            return Theme.primaryDark
        }
    }

    // Override the content
    // Everything a sobel contains is inside of the content Component
    // e.g. Title and Attributes
    // Specify a size for this Component if no implicit size is available (e.g. Item)
    // You don't need to use Layouts (but they are convenient for sizing)
    content: ColumnLayout {
        spacing: Theme.smallSpaceing
        // Display the node name (id)
        Text {
            Layout.fillWidth: true
            text: id
            font: Theme.font.body2
            color: Theme.onprimary

            topPadding: Theme.largeSpaceing
            leftPadding: Theme.largeSpaceing
            rightPadding: Theme.largeSpaceing
            bottomPadding: Theme.smallSpaceing
            verticalAlignment: Text.AlignVCenter
            horizontalAlignment: Text.AlignHCenter
        }
        RowLayout {
            Layout.leftMargin: Theme.largeSpaceing
            Layout.rightMargin: Theme.smallSpaceing
            Layout.fillHeight: true
            Item {
                Layout.preferredHeight: xDerivative.height + yDerivative.height
                                        + Theme.smallSpaceing
                Layout.minimumWidth: Math.max(xDerivative.width,
                                              yDerivative.width)
                // Specify the Attributes
                // Custom Attributes that derive from GAttribute can be used too
                TextFieldAttribute {
                    id: xDerivative
                    // make sure to use the exact same name as in registerAttribute(...) (c++)
                    name: "xDerivative"
                    // get the attribute model from the node model by the name
                    node: sobel
                    model: node.model.getAttribute(name)
                    // Don't forget to add the Attributes to the attributes array
                    // Otherwise saving and loading the node will not work properly
                    // It is necessary to use the onCompleted function to push the
                    // attribute into the array, because content is a Component
                    // and the content object is only created during GNode's onCompleted function
                    Component.onCompleted: attributes.push(xDerivative)
                }
                TextFieldAttribute {
                    id: yDerivative
                    anchors.top: xDerivative.bottom
                    anchors.topMargin: Theme.smallSpaceing
                    name: "yDerivative"
                    node: sobel
                    model: node.model.getAttribute(name)
                    Component.onCompleted: attributes.push(yDerivative)
                }
            }
            // Simple demonstration
            // swap the attribute values
            GIconButton {
                Layout.preferredHeight: Theme.baseHeight
                Layout.preferredWidth: Theme.baseHeight

                // Make sure that the attribute values can not be altered if the node is locked
                // Otherwise unexpected behavior may occur if the values are changed
                // while a stream is processing
                enabled: !isLocked

                icon.source: Theme.icon.swapVert
                icon.color: {
                    if (hovered || pressed) {
                        return Theme.primaryDark
                    }
                    return Theme.onprimary
                }

                transparent: true

                onClicked: {
                    let value = xDerivative.value
                    xDerivative.setValue(yDerivative.value)
                    yDerivative.setValue(value)
                }
            }
        }
    }

    // If you just want to create your individual Attributes and don't want to override the content
    // you can just override the predefined attribute Components e.g.:
    // textFieldAttributeComponent: MyCustomTextFieldAttribute {}
}

Special thanks ❤️ to all open-source projects making life easier for developers all over the world and especially:

• linuxdeployqt
• macos-virtualbox
• ScreenToGif
• Greenshot
• Big Buck Bunny (Example Video)
• OpenCV
• ITK
• Qt