Skip to content

Administration

Jump to bottom
Ryan Smith edited this page Sep 10, 2013 · 38 revisions

This article contains information on how to setup and run l2met.

Security

Data comes in the receiver over HTTP. Data goes out to Librato over HTTPs.

The l2met receiver accepts HTTP requests. Log delivery to the receivers should be encrypted using HTTPs. This means that you will need to handle SSL termination outside of l2met. If you are deploying to Heroku, you should use the SSL Endpoint add-on. If you are running on AWS, run l2met behind an ELB and handle SSL termination in the ELB.

The l2met outlet makes authenticated requests to Librato's HTTPs API. L2met is multi-tennant in the sense that it handles data for many Librato user accounts. Librato authenticates its API with HTTP Basic Authentication using an email as a username and an expireable token as the password. The Librato user gives l2met these credentials in the authentication section of l2met drain url. The Librato credentials in the drain url must be signed and encrypted by the l2met server. L2met uses a secret to sign and encrypt the Librato credentials. There is an HTTP endpoint in the l2met server that will sign credentials. The API requires the secret to authenticate the request. E.g.

$ export SECRETS=$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | openssl base64)
$ ./l2met -port=8080 &
$ curl "http://localhost:8080/sign" --data "$libratouser:$libratopass" -u "$SECRETS:"
gAAAAABR-UAfqpquaZLK77ROMwL3dHXtuHC8jzTVRc6LLsPIPn9Vbe5yVqZ2aRjfyt82KH7mn9iEkah8tO4udeyDH8bguepTebR_tOqnO0bwQuCyIjG-hABav8NmxbrKPGsq5FvdL3ZjdyHr2ewjEWQRJIEKs8vqNlUrBRQKSLqxbtBHiVmWwmM=

The body the the /sign response is the signed & encrypted librato credentials. The l2met receiver will verify and decrypt this data when logs are delivered to l2met. This means that l2met does not have to permanently store user credentials.

Decrypting the User's Librato Credentials

When the l2met receiver receives log data, it performs a verification and attempts to decrypt the Librato credentials, if the verification and decryptions was successful, the decrypted data is discarded and the receivers proceeds to process the data. The encrypted credentials are passed through the entire data flow until the moment before submitting a request to Librato on behalf of the user. It is at this time that l2met will decrypt the credentials and attach them to the header of the Librato HTTP request. This means that credentials passed through Redis are encrypted.

Rolling Credentials

It is good practice to roll secrets. L2met makes this less of a pain by allowing multiple values in the $SECRETS environment variable. You store a : separated list of secrets in the $SECRETS variable and l2met will verify & decrypt incoming data with the first successful secret in the list and eventually failing the authentication if none of the secrets in the list will make verification.

Responsible Disclosure

If you have found a security concern. You can send a private email to the maintainers of l2met. l2met-security@googlegroups.com

Process Formation

L2met can run in a single UNIX process for small to medium size installations. For larger installations, or for installations that require redundancy, you can run multiple instances of l2met which coordinate their data via Redis.

Single Process

This setup will use an in-memory store to coordinate data between the receiver and the outlet.

$ ./l2met -receiver -outlet

Multi Process

This setup must use Redis to coordinate data between the receiver and the outlet.

$ /usr/local/bin/redis-server &
$ export REDIS_URL=redis://localhost:6379
$ ./l2met -receiver &
$ ./l2met -outlet &

Dependencies

When the $REDIS_URL is set, l2met will use Redis as the store. L2met is developed on Redis 2.6.14.

Monitoring

The first thing to monitor is the health endpoint:

GET /health

The result of this endpoint will let your know if your l2met HTTP server is online and if you are using redis, it will issue a PING command to the redis server. If this does not return a 200, you are in trouble.

In addition to the health endpoint on the l2met HTTP server, there are a few components to keep an eye on:

  • Receiver (HTTP)
  • Redis
  • Outlet (HTTP delivery to Librato)

Monitoring the Receiver

The Receiver is responsible for serving HTTP requests and storing data in the Store. It is important to monitor the latency of the HTTP server and the length of the buffers that move the data to the store. Metchan produces the following metrics:

  • http.handle - The time it takes to serve the incoming HTTP request.
  • receiver.accept - The time it takes to parse the HTTP body and convert into buckets.
  • receiver.outlet - The time it takes to deliver a single bucket to the store. (e.g. Redis)

Logplex QoS

Heroku's logplex is the primary sender of logs to l2met. Logplex has a connection and service timeout of 5s. Requests are delivered at most 2 times before they are dropped. When they are dropped, an L10 error message is sent to l2met containing the number of dropped messages. The L10s "roll over" in the case that the L10 message also times out. L2met tracks L10s in 2 ways. The first way is by printing a log message when an L10 is received. The log message will include the message received from logplex along with the username of the decrypted librato credentials for the account. The second form of visibility to this problem is an internal metric. The metric is delivered by metchan and you can find it in your Librato metrics listings under the name logplex.l10.

Monitoring Redis

  • All keys should have an expiration set
  • L2met uses connection pooling with client connections. Keep an eye on the connection created per second using your Redis stats. If there are a lot (e.g. 10 per second) then there is a problem.

Monitoring the Outlet

The outlet will read data from the store, compute statistics, and send the data to the Librato API. There are buffers in-between the store and the outlet that are instrumented with Metchan.

  • reader.scan - The reader pulls down a list of bucket ids and passes them along to be fetched from the store.
  • reader.get - Using the ids from scan, the reader will fetch the bucket from the store and send it to the outlet.
  • outlet.post - The outlet makes an HTTP request to the Librato API.

KPIs

  • reader.scan - The latency on the scan operation should be <= your outlet interval. The default is 1s.
  • outlet.drops - The outlet makes 3 attempts to resubmit data to Librato before dropping. There should be 0 drops.
  • receiver.drops - The receiver drops data which is N time units past the time of the log. N is configurable by the recv-deadline flag.

L2met dashboards

There is a metrics manifest in the repo. Use librato-dash(1) to create metrics, instruments and a dashboard for the l2met metrics.

Scaling

There are a couple of dimensions by which you can scale l2met.

  1. Increase log receivers behind HTTP load balancer.
  2. Increase outlets by partitioning bucket storage.

Scaling Receivers

If the latency in the receiver becomes greater than desirable (e.g. 500ms), the receiver processes can be scaled out horizontally behind a load balancer such as HAProxy of Amazon's ELB.

Scaling Outlets

The receiver will organize buckets into queues where the number of queues is equal to the number of partitions. By default, l2met will be started with 1 partition. You can add more partitions by specifying the command line flag:

$ ./l2met -partitions=2 -outlet &
$ ./l2met -partitions=2 -outlet &

There is no support for onlinle re-partitioning. You must execute a FLUSHALL prior to changing your config.

Operational Playbooks

  • The Receiver is receiving logs but there is not data in Librato
  • The Receiver is unable to receive logs

Maintaining a Private Fork

There are situations in which you would want to maintain a private fork of l2met. For example, if you are deploying l2met to your production environment, and you use a build service like TravisCI to build l2met, you must keep private files in your repository. If your l2met installation can be described as such, the following workflow may be relevant:

$ git clone git://github.com/ryandotsmith/l2met.git $GOPATH/src/github.com/ryandotsmith/l2met
$ cp -r $GOPATH/src/github.com/ryandotsmith/l2met $GOPATH/src/github.com/your-account/l2met
$ cd $GOPATH/src/github.com/your-account/l2met
$ git remote add origin https://github.com/your-account/l2met.git
$ git remote add ryandotsmith https://github.com/ryandotsmith/l2met.git

Your private l2met is now located in $GOPATH/src/github.com/your-account/l2met. You should keep master in coordination with ryandotsmith/l2met:master and add any private files to a separate branch. For example, create a branch named integration which contains files specific to your build.

$ cd $GOPATH/src/github.com/your-account/l2met
$ git checkout master
$ git checkout -b integration
$ git branch
* integration
  master
$ cat 'secrets' >> my-secrets

When there are changes you want to fetch upstream, simply rebase against ryandotsmith/l2met.

$ git checkout master
$ git fetch ryandotsmith
$ git rebase ryandotsmith/master
$ git rebase origin/master
$ git push origin master

$ git checkout integration
$ git rebase master
$ git rebase origin/integration
$ git push origin integration

If there are changes that you would like to send upstream, you must first gain collaborator access to ryandotsmith/l2met. Send an email to the mailing list requesting access. Then, you can create a feature branch off of your master, then push the feature branch to ryandotsmith/l2met and open a pull request using the normal flow.

Clone this wiki locally