Copyright (c) 2022-2023 Antmicro
A minimalistic library for streaming and displaying frames from remote devices.
Work with image processing flows (e.g. from camera streams) usually involves analysing images from intermediate steps. However, when working with e.g. embedded or remote devices, where displaying live images on the monitor is not always possible, the entire image processing flow debugging process may become tedious.
farshow
provides:
farshow-conection
- a library for streaming frames via UDPfarshow
- a program for receiving and displaying such frames on PC, providing the user with a live preview from a remote device using OpenGL and ImGui
Project requirements:
Additional requirements for Python bindings:
To build the project, go to its root directory and execute:
cmake -S . -B ./build
cmake --build ./build
To install it with CMake, run:
cmake --install ./build
By default CMake installs files under /usr/local
directory.
The project consists of a streaming library and a receiving application. They communicate with each other using a UDP protocol.
It is a main application that takes frames from the remote device and displays them on screen. It accepts frames through named streams - each stream forms a separate window in the view.
To start it, just run:
farshow --ip <streamer-address> --port <streamer-port>
Where <streamer-address>
is the IP address of the device streaming frames, and <streamer-port>
is the port under which the UDP application is started.
After a successful run, a window with named streams should appear.
By default, the client will use port 1100, and receive messages from all available interfaces.
For more information on available options, check:
./farshow --help
After a successful build, in the build
directory there is a sample application called ./farshow-server-example
(implemented in src/app-example.cpp).
It implements a simple computer vision flow in which frames provided by either camera or video file (/dev/video0
is the default source) are passed through blurring and thresholding.
The application can be started using:
./farshow-server-example 127.0.0.1
This will start the demonstration server, streaming frames to the server running on 127.0.0.1
on the default port (1100
).
By default, the server sends frames in JPEG format, with a quality factor of 95.
To use e.g. png format with compression 4, add -e .png -q 4
to the runtime parameters.
Available options for the demo application can be found under:
./farshow-server-example --help
All below classes are available in the farshow
namespace.
The core of the library are the FrameSender
and FrameReceiver
classes.
They both derive from UdpInterface
.
To find farshow
in the CMake project, write:
find_package(farshow REQUIRED)
After this, add discovering of farshow
headers with:
include_directories(
${farshow_INCLUDE_DIRS}
)
or via include_target_directories
.
In the end, link the target using farshow
to farshow-connection
library, i.e.:
target_link_libraries(my-app PUBLIC
farshow-connection
)
farshow::FrameSender
class can be used to stream frames to the farshow
client.
It creates a UDP server for streaming frames.
First, create an instance of the farshow::FrameSender
class:
#include <farshow/framesender.hpp>
...
farshow::FrameSender streamer("196.168.1.15", 1111);
...
Where 196.168.1.15
is the client address and 1111
its IP port.
The constructor is also responsible for creating a socket.
To send a frame under the name my_stream
use:
streamer.sendFrame(frame, "my_stream");
Where frame
is a cv::Mat.
To match the client side, the frame should be sent as grayscale or BGR.
This will send the frame as a JPEG with quality 95. It is possible to send it in other formats, e.g. to send it as a png with compression 4, use:
streamer.sendFrame(frame, "my_stream", ".png", cv::IMWRITE_PNG_COMPRESSION=4);
Look for more information about supported formats in OpenCV image reading and writing documentation.
Sending consecutive frames to my_stream
stream will be visualized in farshow
client instance as an animation in a single window.
Creating other stream name, e.g. my_blur
will create a new window called my_blur
in farshow
instance and visualize it.
- To send a frame, it is encoded and checked if it fits the datagram.
- If it does not, it is split into parts.
- Each frame has an id and number of parts.
- Each part also has a separate id.
- The entire structure of the message is available in the
udpinterface.hpp
file asFrameMessage
. - Then the message is sent to the client, which we assigned when creating the instance of
FrameSender
.
The receiver is on the client side, in farshow
application.
It is responsible for collecting the frames from the sender, parsing them and forming a cv::Mat
object.
To receive the frame, create an instance of the FrameReceiver
:
#include "farshow/framereceiver.hpp"
...
farshow::FrameReceiver receiver();
...
Without arguments, it binds the socket to all available interfaces, with the default port 1100
.
It is of course possible to provide a different client IP address and port.
To obtain the frame from the sender, run:
farshow::Frame frame = receiver.receiveFrame();
This part is also included in the farshow-connection
library.
The function returns a Frame
structure with two fields: name
(the stream name as std::string
) and img
(cv::Mat
with the image).
It allows to use farshow-connection
with any frontend for visualization, e.g.:
#include <opencv2/highgui.hpp>
cv::namedWindow(frame.name, cv::WINDOW_AUTOSIZE); // Create a window
cv::imshow(frame.name, frame.img); // Show image in it
cv::waitKey(0); // Wait for a keypress before closing the window
receiveFrame
is a loop which receives parts of frames from various streams and joins them until any of the frames is complete (contains all parts).
To keep the frames in order, we've created a mapping from a stream name to a linked list of FrameContainer
s with all stream frames.
It's worth noting that the frames in the stream are mostly incomplete because when any of them is complete, we return it immediately.
Frames in the list are sorted by id.
When a new part of a frame appears, firstly we find the stream to which it belongs (by name). Then we look at the frame id and (like in insertion sort) look for a proper place for it. Then we copy the data from the frame part to the place where they should be in the actual frame. Since the ids can overflow, the algorithm assumes that when e.g. frame with id 0 comes after 4294967295, it should be placed at the end to ensure stream continuity.
When the frame is complete, we delete all incomplete frames before it (because we have a newer one), decode it and return its name and image (in a Frame
structure).
The farshow
program uses Dear ImGui to display frames.
The program has two threads.
One is responsible for receiving frames and the main one – for displaying them.
They communicate via a map with stream names and their most recent frames.
When the receiver thread receives a new frame, it puts it in the map, changing the most recent image.
When the main thread notices that the frame is changed, it reloads the texture assigned to the frame and the displayed image changes.
Look at Image Loading and Displaying examples with ImGui for more information on how Dear ImGui displays photos.
Both farshow::FrameSender
and farshow::FrameReceiver
C++ classes are available in the farshow
Python module.
Assuming that all afore-mentioned Python dependencies were available at build time, the cmake --install
will also install Python bindings for the library.
Otherwise, the Python bindings will be available after adding build
directory to PYTHONPATH
.
After installation (or from build
directory) it is possible to run a sample farshow
server in Python with:
python3 src/python-bindings/example.py -i 127.0.0.1
This will start the demonstration server, streaming to the server running on 127.0.0.1
on the default port (1100).
By default, /dev/video0
is taken as a stream source.
Find more information about available arguments in command-line help:
python3 example.py --help
Python streaming script is compatible with farshow
application and they can be used together.
FrameSender
, similarly as in C++ can be used to send frames to farshow
client application.
First, create an instance of the frame sender class:
import farshow
...
streamer = farshow.FrameSender("0.0.0.0", 1111)
Where 0.0.0.0
is the client address and 1111
is IP port.
The constructor will create a socket and structures for sending frames.
To send a frame under the name my_stream
, run:
streamer.sendFrame(frame, "my_stream")
Where the frame is a numpy.ndarray. The frame should be in BGR or grayscale format, because it will be passed to OpenCV functions.
This will send the frame as a JPEG with quality 95. As in C++, it is possible to configure streaming parameters:
streamer.sendFrame(frame, "my_stream", ".png", [cv2.IMWRITE_PNG_COMPRESSION, 4])
To receive the frame, create an instance of FrameReceiver
:
import farshow
...
receiver = farshow.FrameReceiver()
Without arguments, it binds the socket to all available interfaces, with the default port 1100.
Then, to fetch the frame, run:
frame = receiver.receiveFrame()
As in C++, it is possible to write custom way to display frames rather than using farshow
application.
import cv2
cv2.namedWindow(frame.name, cv2.WINDOW_AUTOSIZE) # Create a window
cv2.imshow(frame.name, frame.img) # Show image in it
cv2.waitKey(0) # Wait for a keypress before closing the window
The sources are published under the Apache 2.0 License, except for files located in the third-party/
directory. For those files, the license is either enclosed in the file header or a separate LICENSE file.
The README video samples were created using Elephant's Dream, released under Creative Commons Attribution license:
(c) copyright 2006, Blender Foundation / Netherlands Media Art Institute / www.elephantsdream.org