Bifrost

CPU Build	GPU Build	Coverage

A stream processing framework for high-throughput applications.

Bifrost Documentation

Bifrost Roadmap

A Simple Pipeline

Here's a snippet that reads Sigproc filterbank files, applies a Fast Dispersion Measure Transform (FDMT) on the GPU, and writes the results to a set of dedispersed time series files:

import bifrost as bf
import sys

filenames = sys.argv[1:]

print "Building pipeline"
data = bf.blocks.read_sigproc(filenames, gulp_nframe=128)
data = bf.blocks.copy(data, 'cuda')
data = bf.blocks.transpose(data, ['pol', 'freq', 'time'])
data = bf.blocks.fdmt(data, max_dm=100.)
data = bf.blocks.copy(data, 'cuda_host')
bf.blocks.write_sigproc(data)

print "Running pipeline"
bf.get_default_pipeline().run()
print "All done"

A More Complex Pipeline

Below is a longer snippet that demonstrates some additional features of Bifrost pipelines, including the BlockChainer tool, block scopes, CPU and GPU binding, data views, and dot graph output. This example generates high-resolution spectra from Guppi Raw data:

import bifrost as bf
import sys

filenames = sys.argv[1:]
f_avg = 4
n_int = 8

print "Building pipeline"
bc = bf.BlockChainer()
bc.blocks.read_guppi_raw(filenames, core=0)
bc.blocks.copy(space='cuda', core=1)
with bf.block_scope(fuse=True, gpu=0):
    bc.blocks.transpose(['time', 'pol', 'freq', 'fine_time'])
    bc.blocks.fft(axes='fine_time', axis_labels='fine_freq', apply_fftshift=True)
    bc.blocks.detect('stokes')
    bc.views.merge_axes('freq', 'fine_freq')
    bc.blocks.reduce('freq', f_avg)
    bc.blocks.accumulate(n_int)
bc.blocks.copy(space='cuda_host', core=2)
bc.blocks.write_sigproc(core=3)

pipeline = bf.get_default_pipeline()
print pipeline.dot_graph()
print "Running pipeline"
pipeline.shutdown_on_signals()
pipeline.run()
print "All done"

Feature Overview

• Designed for sustained high-throughput stream processing
• Python API wraps fast C++/CUDA backend
• Fast and flexible ring buffer specifically designed for processing continuous data streams
• Native support for both system (CPU) and CUDA (GPU) memory spaces and computation
• Fast kernels for transposition, dedispersion, correlation, beamforming and more
• bfMap: JIT-compiled ND array transformations
• Fast UDP data capture
• A growing library of ready-to-use pipeline 'blocks'
• Rich metadata enables seamless interoperability between blocks

Installation

C Dependencies

$ sudo apt-get install exuberant-ctags

Python Dependencies

• numpy
• contextlib2
• pint
• ctypesgen

$ sudo pip install numpy contextlib2 pint git+https://github.com/olsonse/ctypesgen.git@9bd2d249aa4011c6383a10890ec6f203d7b7990f

Bifrost Installation

Edit user.mk to suit your system, then run:

$ make -j
$ sudo make install

which will install the library and headers into /usr/local/lib and /usr/local/include respectively.

You can call the following for a local Python installation:

$ sudo make install PYINSTALLFLAGS="--prefix=$HOME/usr/local"

Docker Container

Install dependencies:

• Docker Engine
• NVIDIA Docker

Build Docker image:

$ make docker

Launch container:

$ nvidia-docker run --rm -it ledatelescope/bifrost

For CPU-only builds:

$ make docker-cpu
$ docker run --rm -it ledatelescope/bifrost

Running Tests

To run all CPU and GPU tests:

$ make test

Documentation

Online Bifrost Documentation

Building the Docs with Docker

To quickly build the docs using Docker, ensure that you have built a Bifrost container as ledatelescope/bifrost. Then, inside the docs folder, execute ./docker_build_docs.sh, which will create a container called bifrost_docs, then run it, and have it complete the docs-building process for you, outputting the entire html documentation inside docs/html on your machine.

Building the Docs from Scratch

Install sphinx and breathe using pip, and also install Doxygen.

Doxygen documentation can be generated by running:

$ make doc

This documentation can then be used in a Sphinx build by running

$ make html

inside the /docs directory.

Telemetry

By default Bifrost installs with basic Python telemetry enabled in order to help inform how the software is used and to help inform future development. The data collected as part of this consist seven things:

• a timestamp for when the report is generated,
• a unique installation identifier,
• the Bifrost version being used,
• the execution time of the Python process that imports Bifrost,
• which Bifrost modules are imported,
• which Bifrost functions are used and their average execution times, and
• which Bifrost scripts are used. These data are sent to the Bifrost developers using a HTTP POST request where they are aggregated.

Users can opt out of telemetry collection using:

from bifrost import telemetry
telemetry.disable()

or by using the included bifrost_telemetry.py script:

python bifrost_telemetry.py --disable

This command will set a disk-based flag that disables the reporting process.

Acknowledgement

If you make use of Bifrost as part of your data collection or analysis please include the following acknowledgement in your publications:

This research has made use of Bifrost (Cranmer et al. 2017). Continued development of Bifrost is supported by NSF award OAC/2103707.

and cite:

\bibitem[Cranmer et al.(2017)]{2017JAI.....650007C} Cranmer, M.~D., Barsdell, B.~R., Price, D.~C., et al.\ 2017, Journal of Astronomical Instrumentation, 6, 1750007. doi:10.1142/S2251171717500076

Contributors

• Ben Barsdell
• Daniel Price
• Miles Cranmer
• Hugh Garsden
• Jayce Dowell