c-questdb-client

QuestDB - InfluxDB Line Protocol - Ingestion Client Library for C and C++

This library makes it easy to insert data into QuestDB.

This client library implements the InfluxDB Line Protocol (ILP) over TCP.

• Implementation is in C11, with no dependency on the C++ standard library for simpler inclusion into your projects.
• The C++ API is a header-only wrapper written in C++17.

Protocol

Inserting data into QuestDB can be done via one of three protocols.

Protocol	Record Insertion Reporting	Data Insertion Performance
ILP	Errors in logs; Disconnect on error	Best
CSV Upload via HTTP	Configurable	Very Good
PostgreSQL	Transaction-level	Good

This library mitigates the lack of confirmation and error reporting by validating data ahead of time, before any data is sent to the database instance.

For example, the client library will report that a supplied string isn't encoded in UTF-8. Some issues unfortunately can't be caught by the library and require some care and diligence to avoid data problems.

For an overview and code examples, see the ILP page of the developer docs.

To understand the protocol in more depth, consult the protocol reference docs.

Using this Library

Start with the build instructions, then read guide for including this library as a dependency to your project.

Once you've all set up, you can take a look at our examples:

From a C program

#include <questdb/ilp/line_sender.h>

...

line_sender_error* err = NULL;
line_sender* sender = line_sender_connect(
  "0.0.0.0",   // bind to all interfaces
  "127.0.0.1", // QuestDB hostname
  "9009",      // QuestDB port
  &err);

See a complete example in C.

From a C++ program

#include <questdb/ilp/line_sender.hpp>

...

// Automatically connects on object construction.
questdb::ilp::line_sender sender{
  "127.0.0.1",  // QuestDB hostname
  "9009"};      // QuestDB port

See a complete example in C++.

How to use the API

The API is sequentially coupled, meaning that methods need to be called in a specific order.

For each row you need to specify a table name and at least one symbol or column. Symbols must be specified before columns. Once you're done with a row you must add a timestamp calling at or at_now.

This ordering of operations is documented for both the C and C++ APIs below.

C function calling order

Note that this diagram excludes error handling paths: One can call line_sender_close(sender) after any operation.

The line_sender_close(sender) function will release memory and therefore must be called exactly once per created object.

In the C API, functions that can result in errors take a line_sender_error** parameter as last argument. When calling such functions you must check the return value for errors. Functions that return bool use false to indicate a failure.

You may then call line_sender_error_msg(err) and line_sender_error_get_code(err) to extract error details.

Once handled, the error object must be disposed by calling line_sender_error_free(err).

On error you must also call line_sender_close(sender).

Here's a complete example on how to handle an error without leaks:

line_sender* sender = ...;
line_sender_error* err = NULL;
if (!line_sender_flush(sender, &err))
{
  size_t msg_len = 0;
  const char* msg = line_sender_error_msg(err, &msg_len);
  fprintf(stderr, "Could not set table name: %.*s", (int)msg_len, msg);

  // Clean-up
  line_sender_error_free(err);
  line_sender_close(sender);
  return;
}

This type of error handling can get error-prone and verbose, so you may want to use a goto to simplify handling (see example).

C++ method calling order

Note how if you're using C++, .close() can be called multiple times and will also be called automatically on object destruction.

For simplicity the the diagram above does not show that the .close() method and the ~line_sender destructor at any time.

Note that most methods in C++ may throw questdb::ilp::line_sender_error exceptions. The C++ line_sender_error type inherits from std::runtime_error and you can obtain an error message description by calling .what() and an error code calling .code().

Data types

The ILP protocol has its own set of data types which is smaller that the set supported by QuestDB. We map these types into QuestDB types and perform conversions as necessary wherever possible.

Strings may be recorded as either the STRING type or the SYMBOL type.

SYMBOLs are strings with which are automatically interned by the database on a per-column basis. You should use this type if you expect the string to be re-used over and over. This is common for identifiers, etc.

For one-off strings use STRING columns which aren't interned.

For more details see our datatypes page.

Data quality considerations

When inserting data through the API, you must follow a set of considerations.

Library-validated considerations

• Strings and symbols must be passed in as valid UTF-8 which need not be nul-terminated.
• Table names, symbol and column names can't contain the characters ?, ., ,, ', ", \, /, :, (, ), +, -, *, %, ~, ' ' (space), \0 (nul terminator), ZERO WIDTH NO-BREAK SPACE.
• Each row should contain, in order:
  • table name
  • at least one of:
    • symbols, zero or more
    • columns, zero or more
  • timestamp, optionally

Breaking these rules above will result in an error in the client library.

Additional considerations

Additionally you should also ensure that:

• For a given row, a column name should not be repeated. If it's repeated, only the first value will be kept. This also applies to symbols.
• Values for a given column should always have the same type. If changing types the whole row will be dropped (unless we can cast).
• If supplying timestamps these need to be at least equal to previous ones in the same table, unless using the out of order feature. Out of order rows would be dropped.
• The timestamp column should be written out through the provided line_sender_at function (in C) or or .at() method in (C++). It is also possible to write out additional timestamps values as columns.

The client library will not check any of these types of data issues.

To debug these issues you may consult the QuestDB instance logs.

Resuming after an error

If you intend to retry, you must create a new sender object: The same sender object can't be reused.

If you don't see any data

You may be experiencing one of these issues:

QuestDB configuration

If you can't initially see your data through a select query straight away, this is normal: by default the database will only commit data it receives though the line protocol periodically to maximize throughput.

For dev/testing you may want to tune the following database configuration parameters as so:

# server.conf
cairo.max.uncommitted.rows=1
line.tcp.maintenance.job.interval=100

The defaults are more applicable for a production environment.

For these and more configuration parameters refer to database configuration documentation.

API usage

The API doesn't send any data over the network until the line_sender_flush function (if using the C API) or .flush() method (if using the C++ API API) is called.

Closing the connection will not auto-flush.

Community

If you need help, have additional questions or want to provide feedback, you may find us on Slack.

You can also sign up to our mailing list to get notified of new releases.

License

The code is released under the Apache License.