Ena

An imageboard archiver written in Rust

Low resource and high performance archiver to save posts, images and all relevant data from an imageboard into a local database and local image store. Currently only supports 4chan.

Features

• Memory efficient
  Using less than 5mb for a single board and less than 30mb for all 76 boards^*

• Bandwidth efficient
  Minimal API requests by using threads.json and archive.json instead of continuously polling every thread for updates

• Accurate
  Threads are diffed, patched, and merged by the database so posts are always correct and up-to-date

• Collision resistant
  Media files and thumbnails are hashed with SHA256 and deduped by using it as the filename

• Preserved comments
  Untouched in their original HTML format

• Asagi compatible
  Capable as a drop-in replacement

^{* Without full media & thumbnails}

Runtime dependencies

• PostgreSQL >= 11.0

Getting Started

Download the latest release or build from source.

Set your ENA_LOG environment variable to ena=info for console output.

Edit the ena_config.json file and put in your database connection details, media directory, what boards you want to archive, and tweak any of the other settings that look interesting. Don't go below 0.12s or so for the ratelimit or you'll get banned. Follow the API rules and keep it at or above 1 unless you really need to (If you're not sure whether you need to, you probably don't need to). For more information, see the configuration page.

Make sure your PostgreSQL server is running.

You should now be able to run ena and have it start archiving, and report status to the standard output, showing requests as they happen, as well as a display of current queued tasks.
Ctrl-C will stop Ena. To leave Ena running long term, you can use screen (or byobu or any such tool).

Building

Make sure Rust is installed. Then run:

cargo install --git https://github.com/shiimizu/ena

Or the traditional way:

git clone https://github.com/shiimizu/ena.git
cd ena
cargo build --release
# Grab the binary from: target/release/

You can also append -jN to the cargo commands to speed up builds
(where N is the number of CPU cores you have).

For more information, see the building guide.

Usage

ena
An ultra-low resource imageboard archiver

USAGE:
    ena
    ena [OPTIONS] [-c CONFIGFILE]
    ena [OPTIONS]
    command | ena -c -

OPTIONS:
    -c, --config            Specify a config file or pass '-' for stdin
    -h, --help              Prints help information
    -V, --version           Prints version information

Why do this?

Much of my personal time and research went into 4chan for educational purposes and self development. I value the things I've learned there and have a plethora of threads saved. Although there are available archivers, I wanted a better, more efficient, and robust way to do this.

Why Rust?

I wanted something fast with a focus on durability, stability, reliabiility, and longevity. Rust has memory safety guarantee, no GC, speed on par with C, and an ecosystem like Python's. Paying upfront in development time in return for less debugging and runtime errors is a small price to pay. If it compiles, it just werks. For more reasons, see the Why Rust? page.

What's with the name?

Asagi is the eldest of the Ayase sisters. Fuuka is the middle sister. The Ayase family lives next door to Yotsuba. Get it?
– ekopsl

Ena is the next in line and youngest of the Ayase sisters.