Skip to content

Latest commit

 

History

History
293 lines (224 loc) · 20.2 KB

README.md

File metadata and controls

293 lines (224 loc) · 20.2 KB
Raw

FlowGrid

FlowGrid is an immediate-mode interface for Faust (functional audio language) programs. The full project state is backed by a persistent (as in persistent data structures) store supporting constant-time navigation to any point in project history.

Includes:

  • A from-scratch syntax aware embedded text editor with a language-complete Faust tree-sitter grammar for editing LLVM JIT-compiled Faust
  • A complete implementation of the Faust DSP UI spec, including layout and metadata
  • A highly configurable/monitorable audio graph editor and matrix mixer
  • Automatic, configurable resampling at any audio node
  • Any number of (separately running/controllable) Faust DSP nodes
  • Any number of audio I/O nodes wrapping any native audio I/O format
  • Much more!

Still in its early stages, with no supported release yet - expect things to be broken!

Note: I'm still interested and committed to this project, but work is temporarily paused to focus on other things.

Project goals

FlowGrid aims to be a fast, effective, and fun tool for creative real-time audiovisual generation and manipulation. My goal with FlowGrid is to create a framework for patching together artful interactive audiovisual programs (modular, programmable, multimodal jam boxes) by freely connecting media/data streams together. Long-term, I aim for FlowGrid to be expressive and general enough to be considered a full-fledged audiovisual programming language for creatively generating, manipulating, and combining streams across many domains (media/data/network/...). This is a long-term passion project that started in 2022 (with a previous approach starting in 2019) that I hope to poke at for many years :) Tim Exile's Flow Machine is the project's spiritual North Star 🌟

The "Flow" half of FlowGrid reflects the user experience goal to fascilitate a state of joyful and explorative flow. The "Grid" half refers to an ambition to use, as the primary UX paradigm, an "infinitely" nested grid of subgrids, with each cell being a module accepting connections on all sides. This idea stemmed from considering how to maximize the expressive power of a single Push 2 controller (or any grid-like controller, such as many from Akai, equipped with an LED and/or paired with a screen). I haven't gotten to this aspect yet here, but I explored it in a previous approach using JUCE before starting fresh with this project with a new stack and development goals.

Architecture goals

Early on in the development process, I am focusing on building a solid application architecutre to make feature development fast and fun.

Any robust application needs to get a lot right apart from just its features - project state data structure, low-latency state updates from actions & view updates from state, sane event system, serialization, undo/redo history, load/save, backup, debuggability, error management. More than anything, it's important to find ways to make it as easy as possible to add new feasures, ideally without needing to think about any of the above at all.

Doing it the other way around - starting with features and then eating the broccolli and trying to tie together all existing features into a legit application with the above meta-features - is very difficult and slow in my experience. A big part of this project is about exploring patterns for getting this right. I'd say overall, I've happy-ish with the direction here so far, but there is a lot I'd like to improve and explore.

Towards this end, here are some (in-progress) development goals/thoughts:

  • Determinism: The state should, as much as possible, fully specify the current application instance at any point in time. Closing and opening a project should continue all streams where they left off, including the DrawData stream ImGui uses to render its viewport(s).

  • Low latency: Minimize the duration between the time the application receives an input signal, and the corresponding output device respond time (e.g., pixels on screen, audio output, file I/O). Note that low latency applies to all types of input, which currently include:

    • Audio signals from any audio input device on your machine, at any natively-supported sample rate.
    • Mouse/keyboard input.
    • TODOs: MIDI, most likely using libremidi as the backend, targeting Push 2 first (see Old-FlowGrid implementation). USB (including writing to LED displays - see Old-FlowGrid implementation but will rewrite from scratch since the API has likely changed and it wasn't rock-solid anyway). OSC (Open Sound Control). WebSockets.

  • Fast random access to application state history: FlowGrid uses persistent data structures to store its state. After each action, FlowGrid creates a snapshot of the application store and adds it to the history (which will eventually be a full navigation tree), allowing for constant-time navigation to any point in the history. In most applications, if a user e.g. just performed their 10th action and wants to go back to where they were after their first action, they would either manually undo 9 times, or if a random access interface is provided, the application would do this under the hood (in linear time, like rewinding a tape). FlowGrid, on the other hand, provides navigating to any point in the application history (almost always*) at frame rate or faster. This opens up many potential creative applications that are not possible with other applications, like, say, muting the audio output device, and then issuing [undo, redo] actions at audio rate, for a makeshift square wave generator!

    * Some kinds of state changes have higher effect latency, like changing an audio IO device.

  • Fast rebuild: Keeping build times low is crucial, as full rebuilds are frequent. Minimizing the duration between the edit time of a valid FlowGrid source-code file and the application start time after recompilation is crucial. Making compilation snappy isn't just about saving the extra seconds waiting. It's about minimizing the feedback loop between ideation and execution, and making the process of building more engaging. FlowGrid is currently not great in this department and I want to make recompile times faster.

Application State Architecture

FlowGrid uses a unidirectional data-flow architecture, similar to Redux. The architecture draws heavy inspiration from Lager, albeit with fewer features and dependencies, with immer being the sole exception.

User actions are encapsulated in plain structs, containing all necessary information to update the project state. Actions are organized into std::variant types, arranged in a nested domain hierarchy. At the root of this structure is a variant type named Action::Any. All actions are enqueued into a single concurrent queue, with each action applied subsequently overwriting the project store. Actions are grouped by their relative queue time and type, merging into "gestures" at the time of commitment. Each gesture represents a coherent, undoable group of actions. Every committed gesture is recorded in a "history record," which includes the gesture itself and a logical snapshot of the entire project state resulting from its application. If you're unfamiliar with persistent data structures, this is much less memory intensive than it sounds! Each snapshot only needs to track a relatively small amount of data representing its changes to the underlying store, a concept referred to as "structural sharing".

Application docs

Project files

FlowGrid supports two project formats. When saving a project, you can select any of these formats using the filter dropdown in the lower-right of the file dialog. Each type of FlowGrid project file is saved as plain JSON.

  • .fgs: FlowGridState
    • The full project state. An .fgs file contains a JSON blob with all the information needed to get back to the saved project state. Loading a .fgs project file will completely replace the project state with its own.
    • As a special case, the project file ./flowgrid/empty.fgs (relative to the project build folder) is used internally to load projects. This empty.fgs file is used internally to implement the open_empty_project action, which can be triggered via the File->New project menu item, or with Cmd+n. FlowGrid (over-)writes this file every launch, after initializing to empty-project values (and, currently, rendering two frames to let ImGui fully establish its context). This approach provides a pretty strong guarantee that loading a new project will always produce the same, valid empty-project state.
  • .fga: FlowGridActions
    • FlowGrid can also save and load projects as a list of action gestures. This format stores an ordered record of every action that affected the project state up to the time it was saved. More accurately, an .fga file is a list of lists of (action, timestamp) pairs. Each top-level list represents a logical gesture, composed of a list of actions, along with the absolute time they occurred. Each action item contains all the information needed to carry out its effect on the project state. In other words, each list of actions in an .fga file tells you, in application-domain semantics, what happened.
    • Gesture compression: Actions within each gesture are compressed down to a potentially smaller set of actions. This compression is done in a way that retains the same project state effects, while also keeping the same application-domain semantics.

Features

  • A fast from-scratch embedded text editor with the following features:
    • tree-sitter syntax parsing and highlighting, with a full Faust tree-sitter grammar built by me with complete language support
    • Hover to show tree-sitter syntax node and ancestors, integrated into application info pane
    • Persistent buffer data structure, based on ewig, with variant-based actions fully integrated into application state & history
    • Multiple cursors with character/word/line/page-based movement, selection, and deletion
    • Indentation and commenting shortcuts
    • Debug view showing editor state
    • Find/select next occurrence
    • Usual insert/delete/copy/paste/load/save capabilities
    • Edit-recompile integration with Faust features (graph, audio, DSP parameter UI)
  • Extensive audio device configuration, supporting selection of any input device and output device, with separate control over input/output device native configuration, with automatic format/sample-rate conversion when necessary.
    • This is a rare feature in DAWs, which usually provide a single application-level sample-rate conversion, and then automatically select native sample rates that match. This exemplifies the design philosophy of FlowGrid, to enable full control when possible, with solid defaults.
    • In fact, each node in the audio graph (see below) can have its own sample rate, with automatic conversion into and out of the node!
    • Also, you can add and connect multiple audio input/output devices, including duplicates with different configurations!
  • Comprehensive, fully-undoable style editing of layout and style
    • ImGui state management is fully reimplemented to integrate with FlowGrid's persistent data store, so that e.g. undocking a window, changing the docking layout, changing style configuration, or anything affecting what you see in the UI is fully undoable!
  • Complete Faust graph visualization, navigation, and SVG export with comprehensive style editing
    • FlowGrid includes a complete reimplementation of Faust's SVG graph visualization to render in both SVG and ImGui. It is deeply configurable, including adjustable fold complexity to control the number of boxes within a graph before folding into a sub-graph. It has a persistent/undoable navigation history and style editor, with a style preset designed to mimic the original Faust style exactly, with navigable SVG export.
  • Faust DSP ImGui parameter UI implementation
    • Using ImGui's table layout, supports all Faust param UI groupings and widgets, including hover tooltip metadata and display style metadata (e.g. knob vs. slider or horizontal vs. vertical), with arbitrary group nesting.
  • Audio graph
    • Matrix mixer for arbitrary audio graph node connections
    • Optional level/pan and waveform/spectrum monitoring for each node.
    • Add multiple device input/output nodes, multiple Faust nodes (each running in a separate Faust process and managed by its own set of DSP text editor/graph/params UI components)
    • Waveform node with configurable waveform type, frequency and amplitude
      • This is a custom DSP nodes separate from Faust, and is a proof-of-concept for integrating Faust with other custom C++ DSP, in addition to audio I/O.
  • Integrated hierarchical hover info panel
  • Extensive debugging capabilities and insight into application state
    • Live histogram showing the update frequency of each state path, with gesture grouping.
    • Detailed tree view of:
      • All project gestures/actions
      • Full project state (both the complete hierarchical state representation and action state)
      • Live auto-expand/navigate/flash-highlight to changed state tree node
  • Much more!

Clean/Build/Run

This project uses LLVM IR to JIT-compile Faust code. To simplify, make things more predictable, and reduce bloat, we use the LLVM ecosystem as much as possible - clang++/clang to compile, and LLVM's lld for linking. Even if it's not strictly required, I generally aim to use the latest LLVM release available on HomeBrew. If the project does not build correctly for you, please make sure your clang, lld, and clang-config point to the newest available point-release of LLVM. If that doesn't work, try the latest release in the previous LLVM major version.

Mac

  • Install system requirements:

    $ git clone --recursive git@github.com:khiner/flowgrid.git
$ brew install cmake pkgconfig llvm freetype fftw
$ brew link llvm --force

  • Download and install the latest SDK from https://vulkan.lunarg.com/sdk/home

  • Set the VULKAN_SDK environment variable. For example, add the following to your .zshrc file:

    export VULKAN_SDK="$HOME/VulkanSDK/{version}/macOS"

All scripts can be run from anywhere, but to the root repo directory (clean/build).

  • Clean:
    • Clean up everything: ./script/Clean
    • Clean debug build only: ./script/Clean -d [--debug]
    • Clean release build only: ./script/Clean -r [--release]
  • Build:
    • Debug build (default): ./script/Build
    • Release build: ./script/Build -r [--release]
    • Tracy build: ./script/Build -t [--trace]

Debug build is generated in the ./build directory relative to project (repo) root. Release build is generated in ./build-release. Tracy build generated in ./build-tracing

To run the freshly built application:

# The application assumes it's being run from the build directory when locating its resource files (e.g. font files).
$ cd build # or build-release
$ ./FlowGrid # Must be run from a directory above root. todo run from anywhere

If the build/run doesn't work for you, please file an issue, providing your environment and any other relevant details, and I will try and repro/fix!

Stack

Audio

  • Faust for DSP
  • miniaudio for the audio backend
  • fftw for computing spectrograms (visualized with ImPlot)

UI

Backend

  • immer: persistent data structures for the main project state store
    • Used to quickly create, store, and restore persistent state snapshot (used for undo/redo, and for debugging/inspection/monitoring)
  • json: state serialization
  • tree-sitter: Language parsing for syntax highlighting in text buffers
  • ConcurrentQueue: the main action queue
    • Actions are processed synchronously on the UI thread, but any thread can submit actions to the queue.

Debugging

  • Tracy for real-time profiling

Development

I try and keep all dependencies up to date. LLVM 19+ is required to build.

Formatting

FlowGrid uses clang-format for code formatting. ./script/Format formats every cxx file in src.

Tracing

Use ./script/Build -t [--trace] to create a traced build.

To build and run the Tracy profiler, run:

$ brew install gtk+3 glfw capstone freetype
$ cd lib/tracy/profiler/build/unix
$ make release
$ ./Tracy-release

Updating submodules

All submodules are in the lib directory.

Non-forked submodules

Most submodules are not forked. Here is my process for updating to the tip of all the submodule branches:

$ git submodule update --remote lib/{submodule}
$ git add .
$ git cm -m "Update libs"

Forked submodules

The following modules are forked by me, along with the upstream branch the fork is based on:

I keep my changes rebased on top of the original repo branches. Here's my process:

$ cd lib/{library}
$ git pull --rebase upstream {branch} # `upstream` points to the original repo. See list above for the tracked branch
$ ... # Resolve any conflicts & test
$ git push --force

License

This software is distributed under the GPL v3 License.

GPL v3 is a strong copyleft license, which basically means any copy or modification of the code in this repo (excluding any libraries in the lib directory with different licenses) must also be released under the GPL v3 license.

Why copyleft?

The audio world has plenty of open-source resources, but proprietary intellectual property dominates the commercial audio software industry.

A permissive license allowing closed-source commercial usage may help more end users (musicians, artists, creators) in the short term, but it doesn't help developers. As a music producer, finding excellent software or hardware is relatively easy. As a developer, however, I've had a much harder time finding resources, tools, and strategies for effective audio software development.

Although this project is first and foremost a creative tool, the intention and spirit is much more about hacking, learning, educating and researching than it is about producing end media products. For these purposes, keeping the information open is more important than making the functionality freely and widely available.