flow-scanner is a standalone service that can monitor the Flow blockchain for one or more Cadence event types and broadcast those events to one or more consumers. You can run it as a standalone service using the CLI, a Docker container, or use the underlying flow-scanner-lib to integrate it into your own application.

---

Quickstart

You can install the preconfigured standalone CLI version of flow-scanner from NPM:

npm install -g @rayvin-flow/flow-scanner

Check out the Environment Variables documentation below for configuration settings. Once your environment has been configured, you can start the scanner:

flow-scanner

Alternatively, a Docker image is also provided:

The Docker Hub repository can be found at rayvinflow/flow-scanner. The current stable version uses the 1.x tag.

Launch docker image with some default configuration values (check the Environment Variables section for more info):

docker run -e FLOW_ACCESS_NODE=https://access-mainnet-beta.onflow.org -e CADENCE_EVENT_TYPES=A.c1e4f4f4c4257510.TopShotMarketV3.MomentListed -e APP_LOG_LEVEL=debug -e APP_LOG_TYPE=pretty rayvinflow/flow-scanner:1.x

You can also use an .env file (or any other method of setting environment variables for the docker container):

docker run --env-file=.env rayvinflow/flow-scanner:1.x

If you want to build your own docker image, you can follow the instructions below:

• Clone this repository
• Copy the .env.example file to .env and modify the appropriate entries (additional info on this below)
• Run npm install to install all dependencies
• Build the docker image using docker-compose build
• Launch the Docker container using docker-compose up (There is an equivalent command in the Makefile make up, or you can use make up-d to start and detach from the docker container)
• By default, the docker container is watching for changes to the code and will relaunch if any files are modified (using nodemon). If you do not want this, you can launch the container using docker-compose -f docker-compose.yml -f docker-compose.prod.yml up (or make up-prod) to launch in production mode

flow-scanner is easily customized, so check out the Development section below for more information.

---

Environment Variables

All configuration values are read from the environment (using dotenv). This will look for a .env file in the root of the project, and if you are deploying to AWS (or another cloud provider) then you can set environment variables however is applicable. All available environment variables are dsecribed below, but here is a quick example to configure Top Shot marketplace listings to be broadcast to an HTTP endpoint:

FLOW_ACCESS_NODE="https://access-mainnet-beta.onflow.org"
CADENCE_EVENT_TYPES="A.c1e4f4f4c4257510.TopShotMarketV3.MomentListed"
HTTP_EVENT_BROADCASTER_ENDPOINT="http://localhost/your-http-endpoint"

Required

Name	Type	Value
FLOW_ACCESS_NODE	string	The flow access node to use for API requests (ie: https://access-mainnet-beta.onflow.org for mainnet)
CADENCE_EVENT_TYPES	string, comma-separated	List of the Cadence event types that should be monitored

Optional

Name	Type	Value
MAX_FLOW_REQUESTS_PER_SECOND	integer	Maximum number of API requests per second that will be made to the Flow API
DEFAULT_START_BLOCK_HEIGHT	integer	The block height to start scanning for events. Please note that the scanner currently only supports reading from the current spork, so you will not be able to go any further back than that. Make sure to set this to a reasonable starting number. The settings provider (if one was configured) will be used to store the processed block height and resume from there after the first run.
APP_LOG_TYPE	pretty, json, hidden	The format that will be used for console logging. This reference implementation uses ts-log for logging
APP_LOG_LEVEL	debug, info, warn, error	Minimum level of messages that will be logged

Aws Configuration

If you will be using AWS services (SNS, SQS, CloudWatch), then you will need to provide access credentials.

Name	Type	Value
AWS_ACCESS_KEY_ID	string	AWS credentials if using AWS services
AWS_SECRET_ACCESS_KE	string	AWS credentials if using AWS services
AWS_REGION	string	Region of AWS resources if using AWS services
AWS_USE_IAM	true false	USE IAM role rather than credentials to connect to AWS resources

Database Connection

If you are going to be using a database connection (for the settings provider or unique checker), you must define the connection configuration. Knex is used for connecting to the database, and the mysql2 driver is the default. You can use any database that Knex supports, but you will need to modify the application in order to add the necessary driver and update the configuration.

Name	Type	Value
DB_HOST	string	Hostname for the database connection
DB_PORT	integer	Database port
DB_USER	string	Database user
DB_PASSWORD	string	Database password
DB_DATABASE	string	Name of the database
DB_USE_SSL	boolean	Use SSL for the database connection

Settings Provider

The settings provider will be used to persist the block height that has been processed by the scanner. If you restart the scanner, it will resume at the last processed block height. You can also use the memory settings service, which will store the currently processed block height in-memory and it will not be persisted across application executions.

Settings Provider (sqlite)

Name	Type	Value
SETTINGS_PROVIDER	sqlite	Type of settings provider to use
SQLITE_SETTINGS_FILE	string	Path to the SQLite file to use if using the SQLite settings provider

Settings Provider (db)

If you are using the db settings provider, you need to specify the table name to store settings data. The table must already exist, and have the following schema (this example uses a table name of "settings"):

CREATE TABLE `settings` (
  `key` VARCHAR(32) NOT NULL,
  `value` VARCHAR(255) NOT NULL,
  PRIMARY KEY (`key`)
);

Name	Type	Value
SETTINGS_PROVIDER	db	Type of settings provider to use
DB_SETTINGS_TABLE_NAME	string

Event Broadcaster

An event broadcaster is used to send the events that are scanned from the blockchain to a consumer. You can configure one or more of the event broadcasters described below. If you do not configure an event broadcaster, then all events are logged to the console.

Event Broadcaster (sqs)

If you want to use the AWS SQS event broadcaster, make sure you have configured the AWS credentials described above.

Name	Type	Value
SQS_EVENT_BROADCASTER_QUEUE_URL	string	Required. Queue URL for the SQS queue to broadcast events to
SQS_EVENT_BROADCASTER_MESSAGE_GROUP_ID	string	Optional. Message group ID used for message deduplication. By default, this is "flow-scanner-events"

The SQS message will be JSON string in the following format:

type Message = {
  chunkIndex: number // if there are more than 256 events in a transaction, it will be broken into chunks and sent as multiple messages
  chunkCount: number // number of chunks in this transaction
  blockHeight: number // block height of the transaction
  transactionId: string // id of the transaction
  events: [
    blockId: string // id of the block for this event
    blockHeight: number // block height for this event
    blockTimestamp: number // UNIX timestamp for the event
    type: string // Cadence event type
    transactionId: string // id of the transaction
    transactionIndex: number // index of the transaction
    eventIndex: number // index of the event in the transaction
    data: any // decoded payload for the event
  ]
}

Event Broadcaster (sns)

If you want to use the AWS SNS event broadcaster, make sure you have configured the AWS credentials described above.

Name	Type	Value
SNS_EVENT_BROADCASTER_TOPIC_ARN	string	Topic ARN for the AWS SNS topic to broadcast events to
SNS_EVENT_BROADCASTER_MESSAGE_GROUP_ID	string	Optional. Message group ID used for message deduplication. By default, this is "flow-scanner-events"

The SNS message will be JSON string in the following format:

type Message = {
  chunkIndex: number // if there are more than 256 events in a transaction, it will be broken into chunks and sent as multiple messages
  chunkCount: number // number of chunks in this transaction
  blockHeight: number // block height of the transaction
  transactionId: string // id of the transaction
  events: {
    blockId: string // id of the block for this event
    blockHeight: number // block height for this event
    blockTimestamp: number // UNIX timestamp for the event
    type: string // Cadence event type
    transactionId: string // id of the transaction
    transactionIndex: number // index of the transaction
    eventIndex: number // index of the event in the transaction
    data: any // decoded payload for the event
  }[]
}

Event Broadcaster (http)

If you want to broadcast events to an HTTP/s endpoint, define the following environment variables.

Name	Type	Value
HTTP_EVENT_BROADCASTER_ENDPOINT	string	Required. Endpoint to broadcast events to
HTTP_EVENT_BROADCASTER_SHARED_SECRET	string	Optional. If this is defined, it will be used to generate an HMAC-SHA512 hash to include with the payload for message verification. Details are included below.

The HTTP event broadcaster will send a POST request to the configured endpoint. It will continually attempt delivery until a 200 response is received. The body of the request will be following JSON string:

type Payload = {
  // this is the payload containing the events
  payload: {
    timestamp: number // UNIX timestamp that this payload was sent
    data: {
      transactionId: string // id of the Flow transaction
      blockHeight: number // Flow block height
      events: {
        blockId: string // id of the block for this event
        blockHeight: number // block height for this event
        blockTimestamp: number // UNIX timestamp for the event
        type: string // Cadence event type
        transactionId: string // id of the transaction
        transactionIndex: number // index of the transaction
        eventIndex: number // index of the event in the transaction
        data: any // decoded payload for the event
      }[]
    }
  }
  // this is only included if you have enabled message signing (details below)
  hmac?: {
    nonce: string // nonce to use when verifying the signature
    hash: string // HMAC-SHA512 hash (see below for verification instructions)
  }
}

If your HTTP endpoint is publicly accessible, you may want to enable message verification to make sure you are only processing data that you trust. If you have enabled messaging signing using the HTTP_EVENT_BROADCASTER_SHARED_SECRET environment variable, you can verify the payload using the crypto-js library (or any equivalent library).

import sha256 from 'crypto-js/sha256'
import Base64 from 'crypto-js/enc-base64'
import hmacSHA512 from 'crypto-js/hmac-sha512'

// this should be the body of the request sent to your endpoint
const payload = ...
// this should match the shared secret you defined in the HTTP_EVENT_BROADCASTER_SHARED_SECRET environment variable
const sharedSecret = ...

const message = JSON.stringify(payload.payload)
const hashDigest = sha256(payload.hmac.nonce + message)
const hmacDigest = Base64.stringify(hmacSHA512(message + hashDigest, sharedSecret))

if (payload.hmac.hash === hmacDigest) {
  // the message has been verified
}

HTTP Event Broadcaster Message Deduplication

The SNS and SQS event broadcasters use message deduplication to prevent the delivery of the same events more than once. If you are using HTTP, you can use one of the following providers to perform message deduplication before messages are sent.

Message Deduplication (sqlite)

You can use a SQLite database to eliminate the need for an external database server. The required database/tables will be created automatically at runtime.

Name	Type	Value
SQLITE_UNIQUE_CHECKER_FILE	string	Path to SQLite file to use for deduplication data
UNIQUE_CHECKER_GROUP_ID	string	If multiple unique checkers are using the same sqlite file, then uniqueness will be limited to entries with the same group ID. This can be a string up to 16 characters in length

Message Deduplication (db)

If you are using the db message deduplication, you need to specify the table name to store deduplication data. The table must already exist, and have the following schema (this example uses a table name of "deduplication"):

CREATE TABLE `deduplication` (
  `key` VARCHAR(128) NOT NULL,
  `lock_id` VARCHAR(64) NULL,
  `lock_timestamp` INT UNSIGNED NULL,
  `consumed` VARCHAR(255) NOT NULL,
  PRIMARY KEY (`key`)
);

Name	Type	Value
DB_UNIQUE_CHECKER_TABLE_NAME	string	Name of the table to use to store deduplication data.
UNIQUE_CHECKER_GROUP_ID	string	If multiple unique checkers are using the same table, then uniqueness will be limited to entries with the same group ID. This can be a string up to 16 characters in length

CloudWatch Metrics

The scanner can optionally submit metrics for collection and reporting. Integration with AWS CloudWatch is included out-of-the-box.

Name	Type	Value
METRICS_PROVIDER	cloudwatch	Only cloudwatch is implemented for now, but this can be extended
CLOUDWATCH_METRICS_NAMESPACE	string	Namespace to use for CloudWatch metrics
CLOUDWATCH_METRICS_ENV	string	Environment to use for CloudWatch metrics

The following metrics are reported:

Name	Description
FlowApiRequests	How many Flow API requests were made
FlowBlockHeightRequests	How many requests were made for the current block height from the Flow API
FlowBlockHeightRequestDuration	How long the request for the block height took (in ms)
FlowBlockHeightRequestErrors	How many errors occurred while requesting the current block height from Flow
FlowEventRequests	How many requests were made for events from the Flow API
FlowEventRequestDuration	How long the request for events took (in ms)
FlowEventRequestErrors	How many errors occurred while requesting events
FlowEventRequestFailures	How many failures occurred while requesting events (all retries exceeded)
ProcessBlockDuration	How long it took to process a block once all events were received (in ms)
BlocksProcessed	How many blocks have been processed (all events received and events were broadcast)
EventsBroadcasted	How many events have been broadcast
ProcessedBlockHeight	The current block height that has been processed

---

Development

flow-scanner is designed to be easily customized to fit into any solution. It is a wrapper around flow-scanner-lib, so check out the docs in that repository for detailed info about the implementation and how to customize the features. A high-level overview of the flow-scanner repository is included below.

Directory structure

./config/env-config.ts This file is responsible for reading all environment variables and putting them into a structured format

./helpers - Helper utility functions

./providers - Providers for the configuration and logging features

./main.ts - The main application code (explained in more detail below)

./server.ts - The entry point for the application

./bin/cli.js - Entry point for the CLI (when installed through npm install)

Customization

The default implementation of flow-scanner is mostly just building a configuration based on the environment variables and passing them to the FlowScanner from flow-scanner-lib. You can review main.ts to check out that implementation. Review the docs for flow-scanner-lib to read more about extending and customizing the scanner.