hoarder: init at 0.21.0

[three]

fixes #359226

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

This PR packages the Hoarder bookmarking app (homepage: https://hoarder.app/) and resolves #359226.

Project Homepage: https://hoarder.app/

Project v0.21.0 Release: https://github.com/hoarder-app/hoarder/releases/tag/v0.21.0

This package is the sum of multiple related components:

• The app web server
• The workers process, meant to be run in the background and support the web server
• Migration script, necessary to setup the db and start the web server
• A cli to interact with hoarder via its API

I am not affiliated with the upstream project.

Basic server test script

Script to test basic server functionality after running nix build .#hoarder. This isn't as full-featured as the NixOS configuration below.

#!/usr/bin/env bash
set -x

export DATA_DIR=/tmp/test-data-dir
mkdir -p $DATA_DIR
export NEXTAUTH_SECRET=deadbeefdeadbeefdeadbeef

rm -rf $DATA_DIR/*

./result/lib/hoarder/migrate

./result/lib/hoarder/start-workers & sleep 1
./result/lib/hoarder/start-web &

# Function to clean up background processes
cleanup() {
  echo "Cleaning up..."
  # Query all child PIDs of this script and kill them
  CHILD_PIDS=$(pgrep -P $$)
  if [[ -n "$CHILD_PIDS" ]]; then
    kill $CHILD_PIDS
  fi
}

# Trap Ctrl-C (SIGINT) and call cleanup
trap cleanup SIGINT

wait

Example NixOS configuration and video demonstrating basic functionality

This example config demonstrates how this package might be used,

This configuration supports the following features,

• Sign in and sign up functionality
• Adding new articles
• The Cached Content, Screenshot (fullscreen), Archive and Video scrapers
• AI functionality via ollama running on the same server
• Search via meilisearch

{ config, lib, pkgs, ... }:

let
  hoarderEnv = [
    "DATA_DIR=/var/lib/hoarder"
    "NEXTAUTH_URL=https://localhost"
    "NEXTAUTH_SECRET=deadbeefdeadbeefdeadbeef"
    "DISABLE_NEW_RELEASE_CHECK=true"
    "OLLAMA_BASE_URL=http://127.0.0.1:11434"
    "INFERENCE_TEXT_MODEL=llama3.2:3b"         # Must be downloaded manually after ollama enabled
    "BROWSER_WEB_URL=http://127.0.0.1:9222"
    "NEXT_CACHE_DIR=/var/tmp/hoarder"
    "MEILI_ADDR=http://127.0.0.1:7700"
    "MEILI_MASTER_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="

    "CRAWLER_FULL_PAGE_SCREENSHOT=true"
    "CRAWLER_FULL_PAGE_ARCHIVE=true"
    "CRAWLER_VIDEO_DOWNLOAD=true"
  ];
in
  {
    nixpkgs.overlays = [
      (self: super: {
        hoarder = pkgs.callPackage "${
          pkgs.fetchgit {
            url = "https://github.com/three/nixpkgs.git";
            rev = "add-hoarder-app-pr";
            sha256 = "";
          }
        }/pkgs/by-name/ho/hoarder/package.nix" {};
      })
    ];

    services.meilisearch.enable = true;
    services.ollama.enable = true;

    users.groups.hoarder = {};
    users.users.hoarder = {
      description = "hoarder user";
      isSystemUser = true;
      group = "hoarder";
    };

   environment.systemPackages = [ pkgs.hoarder ];

   systemd.services.hoarder-init = {
     serviceConfig = {
       User = "hoarder";
       Group = "hoarder";
       ExecStart = "${pkgs.hoarder}/lib/hoarder/migrate";
       Type = "oneshot";
       RemainAfterExit = true;
       StateDirectory = "hoarder";
       Environment = hoarderEnv;
     };
     wantedBy = [ "multi-user.target" ];
   };
   systemd.services.hoarder-workers = {
     path = [ pkgs.monlith pkgs.yt-dlp ];
     serviceConfig = {
       User = "hoarder";
       Group = "hoarder";
       Restart="on-failure";
       ExecStart = "${pkgs.hoarder}/lib/hoarder/start-workers";
       Type = "simple";
       StateDirectory = "hoarder";
       Environment = hoarderEnv;

       After = [ "network.target" "hoarder-init.service" ];
     };
     wantedBy = [ "multi-user.target" ];
   };
   systemd.services.hoarder-web = {
     serviceConfig = {
       User = "hoarder";
       Group = "hoarder";
       Restart = "on-failure";
       ExecStart = "${pkgs.hoarder}/lib/hoarder/start-web";
       Type = "simple";
       StateDirectory = "hoarder";
       Environment = hoarderEnv;
       After = [ "network.target" "hoarder-init.service" "hoarder-workers.service" ];
     };
     wantedBy = [ "multi-user.target" ];
   };
   systemd.services.hoarder-browser = {
     serviceConfig = {
       User = "nobody";
       Group = "nobody";
       Restart="on-failure";
       TimeoutStopSec=5;
       ExecStart = "${pkgs.chromium}/bin/chromium --headless --no-sandbox --disable-gpu --disable-dev-shm-usage --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222 --hide-scrollbars";
       Type = "simple";
       Environment = hoarderEnv;
       After = [ "network.target" ];
     };
     wantedBy = [ "multi-user.target" ];
   };
  }

Here is a video demonstrating this functionality,

Screen.Recording.2025-01-16.at.8.25.38.AM.mov

Known Cache Issue

Caching does not seem to work properly. The following types of errors get regularly logged during normal usage,

Jan 16 13:58:52 server start-web[783]:  ⨯ Failed to write image to cache M0Ukvr+LGSQPpdca8soE5S76+B3PA0Qd14LKSWOWkss= Error: ENOENT: no such file or directory, mkdir '/nix/store/jcnpr8qz40fikn747gz21fv0nv1a7i7q-hoarder-0.21.0/lib/hoarder/apps/web/.next/standalone/apps/web/.next/cache'
Jan 16 13:58:52 server start-web[783]:     at async Object.mkdir (node:internal/fs/promises:858:10)
Jan 16 13:58:52 server start-web[783]:     at async writeToCacheDir (/nix/store/jcnpr8qz40fikn747gz21fv0nv1a7i7q-hoarder-0.21.0/lib/hoarder/apps/web/.next/standalone/node_modules/next/dist/server/image-optimizer.js:178:5)
Jan 16 13:58:52 server start-web[783]:     at async ImageOptimizerCache.set (/nix/store/jcnpr8qz40fikn747gz21fv0nv1a7i7q-hoarder-0.21.0/lib/hoarder/apps/web/.next/standalone/node_modules/next/dist/server/image-optimizer.js:440:13)
Jan 16 13:58:52 server start-web[783]:     at async /nix/store/jcnpr8qz40fikn747gz21fv0nv1a7i7q-hoarder-0.21.0/lib/hoarder/apps/web/.next/standalone/node_modules/next/dist/server/response-cache/index.js:121:25
Jan 16 13:58:52 server start-web[783]:     at async /nix/store/jcnpr8qz40fikn747gz21fv0nv1a7i7q-hoarder-0.21.0/lib/hoarder/apps/web/.next/standalone/node_modules/next/dist/lib/batcher.js:45:32 {
Jan 16 13:58:52 server start-web[783]:   errno: -2,
Jan 16 13:58:52 server start-web[783]:   code: 'ENOENT',
Jan 16 13:58:52 server start-web[783]:   syscall: 'mkdir',
Jan 16 13:58:52 server start-web[783]:   path: '/nix/store/jcnpr8qz40fikn747gz21fv0nv1a7i7q-hoarder-0.21.0/lib/hoarder/apps/web/.next/standalone/apps/web/.next/cache'
Jan 16 13:58:52 server start-web[783]: }

I don't believe this is possible to fix without changing the upstream project. This does not seem to effect functionality.

---

Add a 👍 reaction to pull requests you find important.

[bachp]

Please provide comments why these patches are needed and where they are comming from.

This makes it much easier for the long term maintenance.