Silicon Notes: A somewhat lightweight, low-friction personal knowledge base.
Features:
- Plaintext editing in Markdown, rendering in HTML
- Language syntax highlighting in rendered HTML
- Bi-directional page relationships
- Powerful full-text and page title search
- Page history
- A table of contents in the left sidebar, where it belongs
- A quite-usable mobile layout
- Built-in documentation
- No-frills UI
- No big frameworks, just a few smallish dependencies
For the rationale on why this was created and paper-thin justifications on certain design decisions, see this blog article.
Projects we rely on and appreciate!
- Python, of course.
- Poetry for project management.
- Flask, the micro-framework.
- Mistune to render Markdown into HTML.
- Pygments for syntax highlighting of code blocks.
- python-slugify creates URL-friendly "slugs" from strings.
- python-dotenv for configuration management.
- Gunicorn for deployment.
- Pytest and Beautiful Soup for functional testing.
- CodeMirror (optional) for editor syntax highlighting
This project is configured and deployed much like any other Flask project. For details, see the Flask configuration handling Docs.
If you want to take this for a quick spin, the following commands should get
you going. You obviously need to have Docker
installed. (If you have Podman, simply substitute docker
for podman.)
docker run \
-ti \
--rm \
-p 127.0.0.1:5000:5000 \
-v silicon_instance:/home/silicon/instance \
docker.io/bityard/siliconAnd then open http://localhost:5000/ with your local web browser.
You could also start it with docker-compose (or docker compose):
cd deploy/docker-local
docker-compose upIf you want to build the image, a Dockerfile and a docker-compose.yaml file
are provided, so you can build the container with:
docker build -t docker.io/bityard/silicon .Or with buildah:
buildah build --format docker -t docker.io/bityard/silicon .Silicon will listen on port 5000 (plaintext HTTP) and stores all application
data in /home/silicon/instance.
This project requires Python 3.9 or greater. On a Debian/Ubuntu system, that means the following packages:
python3python3-pip(unless installed via other means)python3-devpython3-venvnpm(ordocker) (optional to enable CodeMirror editor)
Install Poetry if necessary. Everyone has their own way of setting up their Python tooling but I'm a fan of pipx:
pip3 install --user pipx
pipx install poetryUse poetry to create a virtual environment and install the dependencies:
poetry installSome settings can either be set as environment variables or written to a
file named .env in the project root. For development, this will suffice:
WERKZEUG_DEBUG_PIN=offYou can set any environment variables mentioned in the Flask or Werkzeug docs, but these are some you might care to know about:
FLASK_RUN_HOST: defaults to127.0.0.1FLASK_RUN_PORT: defaults to5000INSTANCE_PATH: where the silicon data (in particular the database) is storedWERKZEUG_DEBUG_PIN: the PIN to enable the Werkzeug debug console. Set to "off" to disable it if you are sure the app is only listening on localhost.SECRET_KEY: A string used in session cookies. For development purposes, this can be anything, but for production it should be a 16-byte (or larger) string of random characters. Setting this is optional as the app will create one (and write it to a file inINSTANCE_PATH) if one doesn't exist.SILICON_EDITOR: When set totextarea, this disables the CodeMirror text editor when editing pages and uses a standard textarea element instead.
To initialize the database after the configuration settings have been set,
run the following command. It will create an instance directory in the root
of the project and initialize the SQLite database from schema.sql.
poetry run flask --app silicon init-dbRun the project via the flask development server:
poetry run flask --app silicon run --debugUnless you changed the defaults, you should be able to access the UI on http://localhost:5000/
To run the tests, install the dev dependencies and run pytest:
poetry install --with dev
poetry run pytestIf you have a tmpfs filesystem, you can set the TMP environment variable to
have test databases created there (which is faster and results in less
wear-and-tear on your disk):
TMP=/dev/shm poetry run pytestTo make sure all code is formatted to flake8 standards, run flake8:
poetry run flake8Silicon Notes is a fairly simple web application which contains no built-in authentication or authorization mechanisms whatsoever. If deploying the application on its own, you should only deploy this to a trusted private network such as a local LAN segregated from the public Internet by a firewall or VPN. If deploying on a public server, you are responsible for ensuring all access to it is secure.
The deploy direcctory contains various sample deployments that may be helpful
as starting points for a production deployment.
Normally, it is easiest to host applications like this on their own domain or subdomain, such as https://silicon.example.com/. If you would rather host it under a prefix instead (as in https://example.com/silicon), see this issue for hints on how to do that.
Support for CodeMirror as a text editor is included by default. It does add a lot of "heft" to the UI, mostly around having to make a separate network request for each language and addon specified. To use it, you also have to install third-party Javascript/CSS static packages by running ONE of the following commands:
# If you have `npm` installed locally
(cd silicon/static && npm ci)Or:
# if you have `docker` installed
docker run -ti --rm -v $PWD/silicon/static:/app -w /app node:alpine npm ciCurrently only a handful of languages are enabled for syntax highlighting, if
you want to edit the list to suit your needs, you can edit
silicon/static/js/edit.js. You can find a list of supported lanauges
here.
To disable CodeMirror and use a regular textarea instead, add the following to
your .env file or environment:
SILICON_EDITOR=textareaIn the event that a database migration is needed, follow these steps:
- Stop the Silicon instance.
- Pull down the latest version of this repository.
- Run
scripts/dump.sh > silicon_data.sql.
To import the data:
- Move or rename the old
instance/silicon.sqlite, if it exists. - Run
poetry run flask --app silicon init-db. - Run
sqlite3 instance/silicon.sqlite < silicon_data.sql. - Start the Silicon instance.
Once you are satisfied that there are no issues, you can archive (or delete)
the old silicon.sqlite file and silicon_data.sql.
If you want to dump your data as JSON (perhaps to import into another system or hack on with your own tools), these scripts are not well tested but might do the job.
Export:
#!/usr/bin/env sh
DB=instance/silicon.sqlite
for table in pages relationships; do
sqlite3 $DB -cmd '.mode json' "select * from $table;" > $table.json
doneImport:
#!/usr/bin/env sh
sqlite3 instance/silicon.sqlite << EOF
INSERT INTO pages (revision, title, body)
SELECT
json_extract(value, '$.revision'),
json_extract(value, '$.title'),
json_extract(value, '$.body')
FROM json_each(readfile('pages.json'));
INSERT INTO relationships
SELECT
json_extract(value, '$.title_a'),
json_extract(value, '$.title_a')
FROM json_each(readfile('relationships.json'));
EOFThe current style sheets were more or less arrived at by trial and error. Any help in organizing the rules in a more coherent yet extensible way would be much appreciated.
It would be nice if the CSS adjusted itself to a dark theme based on the preference set by the user in the browser. This should be pretty easy since almost all of the colors are in one file.
To put it mildly, my JS skills are not the best. I would very much appreciate any suggestions on improvements to the small amount of code there is, or whether there is a better way to organize it. I won't bring in any kind of Javascript "build" tool as a project dependency, though.
This is pretty standard on wiki-like apps, but it's not a critical feature for me so I haven't yet mustered up the fortitude to implement it. (It would also likely involve adding a diff library as a dependency.)
To prevent the loss of unsaved changes while editing, we use the browser's "are you sure?" nag if there has been a change to the editing area since the page was loaded. However, there are still (at least) two opportunies to lose work:
- The browser crashes.
- Two simulatenous edits of a page in separate tabs or windows.
The first is rare and the second is not as serious since both revisions are saved. But it is currently up to the user to recognize what happened and remedy the situation by hand.
These are technically three separate features but I believe they would be quite closely coupled if implemented together.
The tests directory contains functional tests that were deemed the most
important. But they could be better organized and optimized/flexible. Code
coverage is not likely very high. Some tests are lacking or missing because I
was not able to work out the right way to test certain things.
Anchors on headers are a very common feature on various CMSes. They let you link directly to headings, e.g.:
http://example.com/view/page_title#some-section
Mistune (the Markdown->HTML renderer) ships with a task_lists plugin. It is
functional, but it renders task list items inside an ordinary <ul> as just
another kind of list item. This means the task list items get prefixed with
both a bullet point and a checkbox. IMO, this is fairly ugly and the "right"
way to display a task list item is the have the checkbox replace the bullet
point.
To do this right, I think task lists should be their own separate kind of list rather than just another item type in regular lists. It should be possible to accomplish this with a Mistune plugin.
