Skogul v0.5.3

Release date: 2020-03-03

Fixes a bug with -version which wouldn't work for prebuilt binaries/packages.

⚠️: This release has a wrong version number for the RPM. Please avoid using this, and use v0.5.4 instead.

Skogul v0.5.2

Release date: 2020-03-02

Fixes a bug with the install RPMs which would error during install
and uninstall. The RPMs are now built on a system running yum, which
will make sure build dependencies are correctly satisfied. Also reworks the
build system using Make, and introduces a -version flag.

Refactors the internal Handler API to expose Parse, Transform and Send
(also TransformAndSend which transforms a container and then sends it,
and Handle which parses, transforms and sends a container). This should
make the API easier to understand and makes sure relevant procedures (
like Validate()) are run before e.g. Send()-ing things off.

Specific changes:

• Fixes a bug with the RPM building which would make it error
  during install and uninstall (reinstall). The package gets
  installed correctly, but not uninstalled. This bug has to be
  manually worked a round if any previous skogul RPM is
  installed, by manually uninstalling the version using rpm.
  (rpm -e --noscripts skogul*)
• Inverts the influxdb-sender field validation to white-list instead of
  black-list data types, which should fix #119.

Skogul v0.5.1

v0.5.1

Release date: 2020-01-30

Add custom flattening separator, and a magic 'drop' separator to
remove any intermediate keys. Also adds some benchmarks for custom
parsers as well as cleaning up some package names for test files.

Specific changes:

• Support for custom flattening separator as well as dropping intermediate keys.

Skogul v0.5.0

Release date: 2020-01-22

Skogul 0.5.0 is a small release, but should make life much easier for
InfluxDB-users.

Specific changes:

• InfluxDB sender now detects invalid metrics instead of sending "garbage"
  to InfluxDB that fails - it will send what it can.
• InfluxDB sender has much better logging
• Split transformer now correctly transforms remainder of metrics in a
  container after encountering one it can't handle.
• RPM packages and systemd unit files now agree upon default config file
  path.

Skogul v0.4.1

Release date: 2020-01-06

Skogul 0.4.1 brings support for exposing MQTT topics in the metadata. Yup, that's it.

Skogul v0.4.0

Release date: 2019-12-04

Skogul 0.4.0 brings MQTT configuration up to speed with other receivers and senders.
A new transformer to set the timestamp is introduced, as well as a custom JSON parser
for use where the incoming JSON format does not conform to the one of Skogul.
Furthermore, new versions now ship with prebuilt RPMs as well as prebuilt archives.
We also run automated tests on all example configuration in ./docs/examples!

Specific changes:

• JSON syntax errors offsets correctly with tabs
• Support building RPMs and build RPMs for new releases
• Bring MQTT configuration up to speed (configuration updates required!)
• MQTT wildcard support (+ and #)
• A custom JSON parser, for use when the JSON is not already on Skogul format
• Timestamp transformer, used to set the timestamp of a metric from data

On towards glory

Skogul 0.3.0 contains several changes aimed at project management, some to improve core functionality, and some module improvements. It's not a huge change since 0.2.0, and that's how we like it. As per usual, we aim for high quality releases.

Specific changes:

• Use of CI and automated building of release tar-balls
• Cleanup and refresh of documentation
• There's now a single unified way of handling modules that dramatically reduces code duplication in core.
• Modules can now provide documentation for nested data structures, which is correctly exposed in manual pages.
• A conditional transformer (#71) was added that provides a mechanism to only execute a transformer if certain conditions are present.
• The "extract" feature of the metadata transformer no longer creates empty metadata keys, and will remove the original to avoid duplicating information.
• The logrus receiver replaces the log-receiver, allowing better control of log data. More work in this area is to be expected.
• The HTTP receiver can now apply different authorization rules for different paths.

The latter constitutes a bit of a change in configuration.

Ready, set, GO!

v0.2.0 consists of a number of minor polish-fixes oriented around logging, aimed at improving quality of life of a production installation of Skogul. Additionally, the "replace" transformer has been included, which allows modifications of metadata using regular expression substitution.

While v0.1.0 was able to receive and parse handle Junos Telemetry Data, the addition of the "replace" transformer marks this as the first release where we don't have any known missing features. Prior to this, system names would be stored as "foobar-re0:ip.address.of.host", which meant duplicates when the router switched between active routing engines.

We're not marking this as a "pre-release". It's still a v0.x-release, which indicates that you should anticipate significant changes in the future, but we believe the code quality is suited for production use given the features provided. As we will be using this in production, future releases will always aim to maintain high quality.

Future focus will be richer authentication to support enterprise setups where there are multiple data sources operating in different security domains. This will most likely mean changes to the HTTP receiver at the very least.

A bump in the road to glory

Preparing for v0.2.0, this release mainly bundles new features in the transformer-space. It is provided to simplify testing.

New features:

• Telemetry for Junos 19-ish (versus 17), with some minor deviations due to buggy proto-files
• Add regex/replacement transformer for modify metadata values
• Improve logging
• Fix syntax hilighting bug #54
• Other minor bug fixes

Hello world

This is the first alpha-release of Skogul and is suitable for testing by external users, and should satisfy many use-cases. It offers:

• Simple JSON-based configuration, including underlining of syntax errors and several examples
• Extensive documentation for every configurable aspect of Skogul
• Module-based senders, receivers and parsers
• Support for writing to PostgreSQL, MySQL (or MariaDB), InfluxDB, MQTT or other Skogul instances
• Support for parsing Skogul JSON and Juniper Streaming Telemetry (which is google protobuf)
• Support for receiving data over HTTP, HTTPS, UDP messages, files, FIFO, and more
• Extensive tests: unit tests, unit benchmarks and tools for doing full end-to-end performance tests

Future work will focus on filling in the gaps of transformers, streamlining logging and generally adding more bits and pieces to ensure a more streamlined end-user and module-developer experience.

No large changes to the core architecture is planned as of this moment.