v0.23.2

Changes

• Encode AuthenticationContext token
• Change baseplate tween position in pyramid integration

v0.23.1

Changes

• EdgeRequestContext.create now supports loid_id=None.

v0.23.0

New Features

Edge Request Context Propagation

The EdgeRequestContext provides an interface into both authentication and context information about the original request from a user. For edge services, it provides helpers to create the initial object and serialize the context information into the appropriate headers. Once this object is created and attached to the context, Baseplate will automatically forward the headers to downstream services so they can access the authentication and context data as well.

The first step to this getting wide use will be to get the services closest to clients wired up to get authentication tokens from the authentication service and to start parsing the LoID and Session Tracker values from cookies/headers, then internal services will automatically get the request context brought along to them.

Changes

• The Experiments client works with the new Edge Request Context objects. You can pass a User object (context.request_context.user) directly to the call to Experiments.variant and the client will extract the standard user inputs from that object and pass those to the call to experiment.variant as well as include any User event fields in the bucketing event. The Experiments client will also pull event fields out of the Edge Request Context that is attached to the current context.
• The experiment config format has been updated. The "expires" parameter has been deprecated and several new fields have been added. Configs that still use the "expires" field will continue to work as before with a warning that the field is deprecated.

Bug Fixes

• The formatting string for the expiration date of an experiment was incorrect.

Upgrading

Authentication Context

Direct use of the Authentication Context object is now deprecated. You should use the Edge Request Context and it's wrappers for the Authentication Context rather than using the Authentication Context directly. Also, baseplate will only forward the authentication token if it is included in the Edge Request Context, it will not forward the token from an Authentication Context object that is attached directly to the context.

v0.22.0

New Features

Authentication Context Propagation

Authentication tokens provided by the authentication service can now be automatically propagated between services when making Thrift calls. This allows internal services to securely and accurately understand on whose behalf a given request is being made so they can decide if the requester is authorized for a particular action. The context is passed implicitly, in request headers, so no extra parameters need be added to service IDLs. Baseplate provides APIs for validating and accessing the tokens from within request context and will automatically pass upstream credentials to downstream services without extra work.

The first step to this getting wide use will be to get the services closest to clients wired up to get authentication tokens, and then internal services will automatically get the authentication context brought along to them.

v2 Events Support

Baseplate's event system now supports the Thrift-based schemaful v2 events system. Event publishers can be configured to send the new V2 style event payloads. Please see below for instructions on upgrading.

Changes

• Many bug fixes, compatibility improvements, and performance boosts to the experiments framework based on testing in r2. However, it's still an experimental experiment framework so please hold off on using it in your services yet.
• For same-machine MessageQueue use cases that need more buffering room, a MessageQueue implementation backed by Redis has been added.

Bug Fixes

• Spans created as children of local spans are now properly wired up to the instrumentation systems. Metrics, tracing, etc did not work before in that context but will now.

Upgrading

Dependencies

Baseplate now has a new hard dependency on PyJWT. This should be automatically installed in new development environments going forward, but for existing ones you'll want to install the package inside your development VM:

$ sudo apt install python-jwt python3-jwt 

V2 Events

In your application's entry point, add a V2 event queue:

@@ -2,7 +2,7 @@
-from baseplate.events import Event, EventQueue
+from baseplate.events import Event, EventQueue, serialize_v2_event

@@ -97,7 +97,7 @@ def make_wsgi_app(app_config):
     baseplate.add_to_context("events_production", EventQueue("production"))
     baseplate.add_to_context("events_test", EventQueue("test"))
+    baseplate.add_to_context("events_v2", EventQueue("v2", event_serializer=serialize_v2_event))

Install the event schemas IDL files in your application and then you can then instantiate Thrift-based events and put them into the queue like normal.

from event_schemas.event.ttypes import Event

def some_handler(request):
    event = Event(
        source="baseplate",
        action="test",
        noun="baseplate",
        client_timestamp=time.time() * 1000,
        uuid=str(uuid.uuid4()),
    )
    request.events_v2.put(ev2)

There will not be any v2 test publishers on production machines going forward. Instead, the v2 queue will publish to test or production depending on if the application is running in staging or production. You can remove the v1 test queue if you like.

Authentication Context

Until edge services are wired up to the authentication service, there won't be any authentication contexts flowing around, but this is what integration looks like when it's time.

For a service deep inside the call graph, all you need to do is check the authentication.

def some_handler(request):
    if not request.authentication.valid:
        raise NotAuthenticatedError
        
    if request.authentication.account_id in allowed_ids:
        ...

Services at the edge need to do some extra work to get a token from the authentication service and then add it to the context. For this and other deeper use cases, check out the docs.

v0.21.0

New Features

Experimental Experiments System

Baseplate now has an A/B testing experiment system. This is currently very experimental, but will be progressing in the next few weeks to something production ready. The goal is to allow rapid bucketing of users into experiment variants even deep within the call graph of services. This initial version is focused on compatibility with r2's experiment logic but with a cleaner API.

Live Data

This is a new low-level feature that will power other features down the line (like the experiments system). It consists of a sidecar daemon that watches ZooKeeper and updates files on disk, and associated tools for interacting with ZooKeeper from CI etc. This is intended to bring live config type powers to Baseplate services. FileWatcher instances can be used in applications to watch the data the sidecar daemon copies locally.

Changes

• The FileWatcher will now continue to use its in-memory data if the underlying file fails to parse.

Upgrading

No changes should be needed when upgrading from v0.20.

v0.20.0

This release has a few small new features but is focused mainly
on some behind-the-scenes cleanups and improvements in preparation
for bigger additions coming soon.

New Features

HVAC client integration for advanced Vault usage

Baseplate v0.19 added support for securely fetching secrets from Vault via a
sidecar daemon. For more advanced usages, like allowing Vault to manage
encryption of data for you without you ever knowing the key, you need to talk
directly to Vault in-request. This integration brings Baseplate's
instrumentation and metrics to the HVAC client library for Vault.

Counters for success/error rate on spans

Baseplate now sends additional information to Graphite about the success and
failure of operations in your application. This can be helpful to understand
the volume of requests flowing in and out, and what percentage of them failed.

FileWatcher

Both the SecretsStore and ServiceInventory work by watching files on local
disk for changes and updating the application's in-memory state on change. This
functionality has been factored out into the FileWatcher which
allows you to apply this pattern directly in your application for
configuration, local data models, etc.

DictOf

This is a new option type for the configuration parser which allows you to
ingest groups of keys that you don't know the full names of at coding time. This is rather abstract, so check out some examples:

[app:main]
population.cn = 1383890000
population.in = 1317610000
population.us = 325165000
population.id = 263447000
population.br = 207645000

>>> cfg = config.parse_config(raw_config, {
...     "population": config.DictOf(config.Integer),
... })

>>> len(cfg.population)
5

>>> cfg.population["br"]
207645000

See the docs for more examples and details.

Changes

• Greatly improve test coverage, particularly of instrumentation integrations.
  Many of the bug fixes listed below came from this work.
• Clean up formatting and do a general readability pass on the documentation.
• Atomically swap out contents of secrets file. This removes a potential race
  condition where the secret fetcher daemon is caught in the middle of writing
  out updated secrets.
• Add Vault URL to secrets file. This allows the HVAC integration to get the
  full information on how to connect to and authenticate with Vault from one
  place.
• Overhaul development environment puppet manifests for more comprehensive
  setup.
• BREAKING Remove gauge increment/decrement support. We don't want to
  encourage use of this feature as it's unreliable when the statsd service is
  restarted since the "current" value is only stored in memory.

Bug fixes

• Raise an informative error instead of crashing when fetching secrets and the
  secrets file is not available.
• Fix SQLAlchemy instrumentation's reporting of errors. Previously, failed
  queries did not have their associated spans properly finished and so were not
  instrumented.
• Fix Cassandra instrumentation's reporting of errors. Previously, failed queries
  would have an error tag set but would not be finished in an error state, so
  observers got an incomplete view of the error information.
• Set error tags on spans consistently in Zipkin reporting. This ensures that all
  failed spans get an error=true tag set in Zipkin.
• Fix a crash detecting the hostname on systems that do not have a valid address-
  to-host mapping configured.
• Catch, log, and discard exceptions when posting trace data in Zipkin integration.

Upgrading

Secrets file format changes

The secrets fetcher daemon writes out a new vault field to the secrets file
containing both the original token and new url fields. A vault_token
field is also written for compatibility with applications running on Baseplate
v0.19. The secrets store code expects this new vault field to exist. To upgrade
safely:

1. Update Baseplate to v0.20+
2. Restart the secret fetcher daemon and verify the secrets file is updated.
3. Restart all application processes.

Gauge increment/decrement usage

This feature is completely removed, so you'll need to delete any references to
it in your code. This is quite unlikely to affect any Reddit services right now
as our previous statsd implementation, Tallier, didn't support gauges at all.

v0.19.0

New Features

Secure secret storage in Vault

Baseplate now includes support for fetching secrets in a secure, auditable, manner from Hashicorp Vault. A sidecar daemon manages the infrastructure-level authentication with Vault and fetches secrets to a file on disk. Helpers in Baseplate then allow your application to fetch these secrets efficiently from the sidecar daemon with some helpful conventions for versioning/key rotation. This is now the right way to get secret tokens into your application going forward. See the docs for more info.

Error reporting via Sentry

A new span observer integrates Baseplate applications with Raven, the Sentry client. This makes it easy to automatically report exceptions in the application to Sentry. Extra context like the trace ID, git revision of the running application, and context from the incoming request are automatically included in the exception event sent to Sentry. The raven client is also attached to the context object as context.sentry for sending custom events as desired.

Changes

• Rework message signer API to take advantage of versioned secrets from Vault. The old class-based API is still supported but is now marked deprecated.
• Rename make_metrics_client to metrics_client_from_config and make_tracing_client to tracing_client_from_config to be more consistent with other _from_config functions used throughout Baseplate. The old names are still supported but are now marked as deprecated.

Bug fixes

• Fix exception capture in server spans for the Thrift integration.
• Suppress noisy urllib3 logs from zipkin integration.
• Fix context property attachment in local spans, this caused observers like the metrics observer to be called multiple times on the same server span when local spans were used.
• Fix Python 3 import incompatibilities in baseplate-healthcheck3.

Upgrading

Adding Vault integration

In your application's entry point, instantiate a SecretsStore with secrets_store_from_config and attach it to the context object with add_to_context.

 def make_processor(app_config):
+    secrets = secrets_store_from_config(app_config)
+    baseplate.add_to_context("secrets", secrets)

Make sure that any time you use a secret you fetch it from the secret store rather than storing fetched secrets in the application. This ensures that as secrets expire and rotate your application stays up to date.

secret = context.secrets.get_versioned("secret/my-service/signing-key")
return make_signature(secret, "This is a signed message!", max_age)

Adding Sentry Integration

Sentry integration is just a matter of configuring and registering the new observer at application startup. For example (from the activity service):

@@ -9,6 +9,7 @@
 from baseplate import (
     Baseplate,
     config,
+    error_reporter_from_config,
     metrics_client_from_config,
     tracing_client_from_config,
 )
@@ -122,6 +123,7 @@ def make_processor(app_config):  # pragma: nocover
 
     metrics_client = metrics_client_from_config(app_config)
     tracing_client = tracing_client_from_config(app_config)
+    error_reporter = error_reporter_from_config(app_config, __name__)
     redis_pool = redis.BlockingConnectionPool.from_url(
         cfg.redis.url,
         max_connections=cfg.redis.max_connections,
@@ -132,6 +134,7 @@ def make_processor(app_config):  # pragma: nocover
     baseplate.configure_logging()
     baseplate.configure_metrics(metrics_client)
     baseplate.configure_tracing(tracing_client)
+    baseplate.configure_error_reporting(error_reporter)
     baseplate.add_to_context("redis", RedisContextFactory(redis_pool))
 
     counter = ActivityCounter(cfg.activity.window.total_seconds())

To configure the Sentry client in your application's config file, get a DSN from Sentry (for Reddit crew, contact IO) then see the docs for error_reporter_from_config for all the available options.

Make sure to add python-raven or python3-raven to your development environment.

Updating renamed functions

Update the import and call for make_metrics_client and make_tracing_client in your application's entry point with the new names metrics_client_from_config and tracing_client_from_config.

Updating usage of the MessageSigner

The previous MessageSigner API was a class that took a static secret and then provided signature generation and validation methods. The new API is bare functions that take the secret as a parameter. This allows you to refetch the secret from the SecretsStore immediately before use.

Before:

signer = MessageSigner(b"hunter2")
signature = signer.make_signature("message", max_age)
signer.validate_signature("message", signature)

After:

secret = context.secrets.get_versioned("secret/my-service/my-secret")
signature = signer.make_signature("message", max_age)

...

secret = context.secrets.get_versioned("secret/my-service/my-secret")
validate_signature(secret, "message", signature)

If you have not set up Vault yet, you can also use VersionedSecret.from_simple_secret to make a fake versioned secret from a static value for use with the new API.

v0.18.0

Changes

New Features

Memcached serialization helpers

The pymemcached integration now comes with helpers for serializing and deserializing python objects for storage in memcached. For green-field applications, use the non-pickle variants. Applications interacting with data from r2's caches can use the pickle variants.

CQLMapper integration

Baseplate now has instrumented clients for use with the Cassandra object mapper CQLMapper which is a stripped down fork of cqlengine better suited for use in environments like Baseplate that avoid application-global context.

API Modifications

• The Baseplate.configure_tracing convenience method has been reworked so that Baseplate can standardize configuration parsing for your application.
• BREAKING: The baseplate commands which parse INI files no longer allow interpolation of variables in the config file syntax.

Upgrading

Updating tracing setup

Replace manual configuration parsing with a call to Baseplate.make_tracing_client. Here's a theoretical example of upgrading the activity service:

@@ -10,6 +10,7 @@ from baseplate import (
     Baseplate,
     config,
     make_metrics_client,
+    make_tracing_client,
 )
 from baseplate.context.redis import RedisContextFactory
 from baseplate.integration.thrift import BaseplateProcessorEventHandler
@@ -113,10 +114,6 @@ def make_processor(app_config):  # pragma: nocover
             "window": config.Timespan,
             "fuzz_threshold": config.Integer,
         },
-        "tracing": {
-            "endpoint": config.Optional(config.Endpoint),
-            "service_name": config.String,
-        },
         "redis": {
             "url": config.String,
             "max_connections": config.Optional(config.Integer, default=100),
@@ -124,6 +121,7 @@ def make_processor(app_config):  # pragma: nocover
     })
 
     metrics_client = make_metrics_client(app_config)
+    tracing_client = make_tracing_client(app_config)
     redis_pool = redis.BlockingConnectionPool.from_url(
         cfg.redis.url,
         max_connections=cfg.redis.max_connections,
@@ -133,10 +131,7 @@ def make_processor(app_config):  # pragma: nocover
     baseplate = Baseplate()
     baseplate.configure_logging()
     baseplate.configure_metrics(metrics_client)
-    baseplate.configure_tracing(
-        cfg.tracing.service_name,
-        cfg.tracing.endpoint,
-    )
+    baseplate.configure_tracing(tracing_client)
     baseplate.add_to_context("redis", RedisContextFactory(redis_pool))
 
     counter = ActivityCounter(cfg.activity.window.total_seconds())

v0.17.1

• Add configuration for trace debug logging
• Move child span event listening to base SpanObserver

v0.17.0

• Fix string coercion for binary annotations in tracing
• Add Percent helper in config module
• Add local span support for tracing