BitStream

An extensible C++ library for serializing and quantizing types into tightly packed bitstreams.
Based on Glenn Fiedler's articles about packet serialization.

Table of Content

Compatibility
Installation
Usage
Documentation
Serialization Examples
Serializables - serialize_traits
Extensibility
Building and running tests
3rd party
License

Compatibility

This library was made with C++17 in mind and is not compatible with earlier versions. Many of the features use if constexpr, which is only available from 17 and up. If you really want it to work with earlier versions, you should just be able to replace the newer features with C++1x counterparts.

Installation

As this is a header-only library, you can simply copy the header files directly into your project and include them where relevant. The header files can either be downloaded from the releases page or from the include/ directory on the master branch.

The source and header files inside the src/ directory are only tests and should not be included into your project, unless you wish to test the library as part of your pipeline.

Usage

The library has a main header file (bitstream/bitstream.h) which includes every other header file in the library.

If you only need certain features you can instead opt to just include the files you need. The files are stored in categories:

quantization/ - Files relating to quantizing floats and quaternions into fewer bits
stream/ - Files relating to streams that read and write bits
traits/ - Files relating to various serialization traits, like serializble strings, integrals etc.

Unlike most serilization libraries the default type traits are setup to use in and out parameters and thus share the same interface. This greatly simplifies user-defined serialization logic, as you can now share the same template function for both reading and writing.

// Some user-defined type that isn't inherently serializable
struct custom_type
{
    bool enabled = true;
    int count = 42;
};

// Writing and reading share the same interface, so we can template it
template<typename Stream>
bool serialize_custom_type(Stream& stream, custom_type& value)
{
    if (!stream.serialize<bool>(value.enabled))
        return false;
    
    return stream.serialize<int>(value.count);
}

byte_buffer<32> buffer;
fixed_bit_writer writer(buffer);

custom_type in_value;
serialize_custom_type(writer, in_value); // Serialize the value

uint32_t num_bits = writer.flush();
fixed_bit_reader reader(buffer, num_bits);

custom_type out_value;
serialize_custom_type(reader, out_value); // Deserialize the value

An important aspect of the serialiaztion is performance, since the library is meant to be used in a tight loop, like with networking. This is why the default operations don't use exceptions, but instead return true on success and false on failure. It's important to check these return values after every operation, especially when reading from an unknown source. You can check it manually or use the BS_ASSERT(x) macro for this, if you want your function to return false on failure.

It is also possible to dynamically put a break point or trap when a bitstream would have otherwise returned false. This can be great for debugging custom serialization code, but should generally be left out of production code. Simply #define BS_DEBUG_BREAK before including any of the library header files if you want to break when an operation fails.

For more concrete examples of usage, see the Serialization Examples below. If you need to add your own serializable types you should also look at the Extensibility section. You can also look at the unit tests to get a better idea about what you can expect from the library.

Documentation

Refer to the documentation for more information about what different classes provide.

Serialization Examples

The examples below follow the same structure: First writing to a buffer and then reading from it. Each example is littered with comments about the procedure, as well as what outcome is expected.

Writing the first 5 bits of an int to the buffer, then reading it back:

// Create a writer, referencing the buffer and its size
alignas(uint32_t) uint8_t buffer[4]; // Buffer must be a multiple of 4 bytes / 32 bits and 4-byte-aligned
fixed_bit_writer writer(buffer, 4);

// Write the value
uint32_t value = 27; // We can choose any value below 2^5. Otherwise we need more than 5 bits
writer.serialize_bits(value, 5);

// Flush the writer's remaining state into the buffer
uint32_t num_bits = writer.flush();

// Create a reader, referencing the buffer and bits written
fixed_bit_reader reader(buffer, num_bits);

// Read the value back
uint32_t out_value; // We don't have to initialize it yet
reader.serialize_bits(out_value, 5); // out_value should now have a value of 27

Writing a signed int to the buffer, within a range:

// Create a writer, referencing the buffer and its size
byte_buffer<4> buffer; // byte_bufer is just a wrapper for a 4-byte aligned buffer
fixed_bit_writer writer(buffer);

// Write the value
int32_t value = -45; // We can choose any value within the range below
writer.serialize<int32_t>(value, -90, 40); // A lower and upper bound which the value will be quantized between

// Flush the writer's remaining state into the buffer
uint32_t num_bits = writer.flush();

// Create a reader, referencing the buffer and bits written
fixed_bit_reader reader(buffer, num_bits);

// Read the value back
int32_t out_value; // We don't have to initialize it yet
reader.serialize<int32_t>(out_value, -90, 40); // out_value should now have a value of -45

Writing a c-style string into the buffer:

// Create a writer, referencing the buffer and its size
byte_buffer<32> buffer;
fixed_bit_writer writer(buffer);

// Write the value
const char* value = "Hello world!";
writer.serialize<const char*>(value, 32U); // The second argument is the maximum size we expect the string to be

// Flush the writer's remaining state into the buffer
uint32_t num_bits = writer.flush();

// Create a reader, referencing the buffer and bits written
fixed_bit_reader reader(buffer, num_bits);

// Read the value back
char out_value[32]; // Set the size to the max size
reader.serialize<const char*>(out_value, 32U); // out_value should now contain "Hello world!\0"

Writing a std::string into the buffer:

// Create a writer, referencing the buffer and its size
byte_buffer<32> buffer;
fixed_bit_writer writer(buffer);

// Write the value
std::string value = "Hello world!";
writer.serialize<std::string>(value, 32U); // The second argument is the maximum size we expect the string to be

// Flush the writer's remaining state into the buffer
uint32_t num_bits = writer.flush();

// Create a reader, referencing the buffer and bits written
fixed_bit_reader reader(buffer, num_bits);

// Read the value back
std::string out_value; // The string will be resized if the output doesn't fit
reader.serialize<std::string>(out_value, 32U); // out_value should now contain "Hello world!"

Writing a float into the buffer with a bounded range and precision:

// Create a writer, referencing the buffer and its size
byte_buffer<4> buffer;
fixed_bit_writer writer(buffer);

// Write the value
bounded_range range(1.0f, 4.0f, 1.0f / 128.0f); // Min, Max, Precision
float value = 1.2345678f;
writer.serialize<bounded_range>(range, value);

// Flush the writer's remaining state into the buffer
uint32_t num_bits = writer.flush();

// Create a reader, referencing the buffer and bits written
fixed_bit_reader reader(buffer, num_bits);

// Read the value back
float out_value;
reader.serialize<bounded_range>(range, out_value); // out_value should now be a value close to 1.2345678f

These examples can also be seen in src/test/examples_test.cpp.

Serializables - serialize_traits

Below is a noncomprehensive list of serializable traits. A big feature of the library is extensibility, which is why you can add your own types as you please, or choose not to include specific types if you don't need them.