Self-hosted instance randomly stopped storing events

[HalfVoxel]

My self-hosted instance has been doing just fine on version 2.0 for at least half a year. However, once November started, it registered only a single pageview, and after that it's not been logging a single event.

The docker compose logs contained some errors relating to updating tzdata, unfortunately I didn't save the error message (but I think it contained "GenServer :tzdata_release_updater terminating").

In any case, I upgraded to the latest (2.1.4) to try to resolve the issue. However, I still get the same result:

• The dashboard works just fine
• I can view old events
• I can send events, and the backend responds with 202 "ok".
• Using docker compose exec -it plausible_events_db clickhouse-client and then checking the events_v2 table, I see no new events being stored.
• If I change the docker container to use a fresh volume for the clickhouse container, events, events_v2 and sessions_v2 are all empty, even after submitting many pageview events.
• Events do not show up even after multiple days.

Dashboard:

docker compose logs:

Considered causes:

• /api/<domain>/status returns "waiting" even though there are page views #4143, this seems similar, but even in that case, events should still be stored in the database.

The docker compose config is the standard, except that all services have been added to a named network (for nginx-proxy purposes).

compose.yml

services:
  plausible_db:
    image: postgres:16-alpine
    restart: always
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=postgres
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      start_period: 1m
    networks:
      - plausible
plausible_events_db:

image: clickhouse/clickhouse-server:24.3.3.102-alpine

restart: always

volumes:

- event-data:/var/lib/clickhouse

- event-logs:/var/log/clickhouse-server

- ./clickhouse/logs.xml:/etc/clickhouse-server/config.d/logs.xml:ro

# This makes ClickHouse bind to IPv4 only, since Docker doesn't enable IPv6 in bridge networks by default.

# Fixes "Listen [::]:9000 failed: Address family for hostname not supported" warnings.

- ./clickhouse/ipv4-only.xml:/etc/clickhouse-server/config.d/ipv4-only.xml:ro

ulimits:

nofile:

soft: 262144

hard: 262144

healthcheck:

test: ["CMD-SHELL", "wget --no-verbose --tries=1 -O - http://127.0.0.1:8123/ping || exit 1"]

start_period: 1m

networks:

- plausible
plausible:

image: ghcr.io/plausible/community-edition:v2.1.4

restart: always

command: sh -c "/entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"

depends_on:

plausible_db:

condition: service_healthy

plausible_events_db:

condition: service_healthy

volumes:

- plausible-data:/var/lib/plausible

ulimits:

nofile:

soft: 65535

hard: 65535

networks:

- plausible

environment:

- TMPDIR=/var/lib/plausible/tmp

# required: https://github.com/plausible/community-edition/wiki/configuration#required

- BASE_URL=${BASE_URL}

- SECRET_KEY_BASE=${SECRET_KEY_BASE}

# optional: https://github.com/plausible/community-edition/wiki/configuration#optional

# registration: https://github.com/plausible/community-edition/wiki/configuration#registration

- TOTP_VAULT_KEY

- DISABLE_REGISTRATION

- ENABLE_EMAIL_VERIFICATION

# web: https://github.com/plausible/community-edition/wiki/configuration#web

- HTTP_PORT

- HTTPS_PORT

# databases: https://github.com/plausible/community-edition/wiki/configuration#database

- DATABASE_URL

- CLICKHOUSE_DATABASE_URL

# Google: https://github.com/plausible/community-edition/wiki/configuration#google

- GOOGLE_CLIENT_ID

- GOOGLE_CLIENT_SECRET

# geolocation: https://github.com/plausible/community-edition/wiki/configuration#ip-geolocation

- IP_GEOLOCATION_DB

- GEONAMES_SOURCE_FILE

- MAXMIND_LICENSE_KEY

- MAXMIND_EDITION

# email: https://github.com/plausible/community-edition/wiki/configuration#email

- MAILER_ADAPTER

- MAILER_EMAIL

- MAILER_NAME

- SMTP_HOST_ADDR

- SMTP_HOST_PORT

- SMTP_USER_NAME

- SMTP_USER_PWD

- SMTP_HOST_SSL_ENABLED

- POSTMARK_API_KEY

- MAILGUN_API_KEY

- MAILGUN_DOMAIN

- MAILGUN_BASE_URI

- MANDRILL_API_KEY

- SENDGRID_API_KEY

- VIRTUAL_HOST

- LETSENCRYPT_HOST

- LETSENCRYPT_EMAIL
volumes:

db-data:

event-data:

event-logs:

plausible-data:

networks:

plausible:

Questions

Where do I even start looking for answers?
I've tried to find various logs, but none have lead to any insights. Is there anywhere I could look?

[ruslandoga]

👋 @HalfVoxel

I don't think anyone has reported any similar errors, so it's something new.

Where do I even start looking for answers?

First, you can lower the log level in Plausible:

$ cd plausible-ce
$ docker compose exec plausible bin/plausible remote

iex> Logger.configure(level: :debug)
:ok

# use ctrl+c ctrl+c to exit IEx

$ docker compose logs plausible -f

Note that the event buffer is flushed every five seconds. So if there is an error somewhere in the connection to ClickHouse it would show up within 5 seconds after the first /api/events request.

If these debug logs don't show anything interesting, you can also try tracing the functions on the ingestion path.
It probably makes sense to start at the end of the pipeline, at the point where we submit an event to the buffer https://github.com/plausible/analytics/blob/master/lib/plausible/event/write_buffer.ex. If the no event reaches this point during tracing, it would mean they are dropped somewhere in the ingestion pipeline.

iex> mod_fun_args = {Plausible.Event.WriteBuffer, :insert, 1}
iex> how_many_times = 5
iex> :recon_trace.calls(mod_fun_args, how_many_times)

[ruslandoga]

Off-topic: I see some "new" env vars in your compose.yml

- VIRTUAL_HOST
- LETSENCRYPT_HOST
- LETSENCRYPT_EMAIL

I doubt they can cause issues but I don't think they are used in Plausible.

[HalfVoxel]

These are just for the nginx proxy. They automatically expose the service to the web with a certificate. Shouldn't cause any issues.

[ruslandoga]

Oh interesting. How does it work? Aren't these env vars internal to the Plausible container?

[HalfVoxel]

It uses an api equivalent to docker inspect to check which docker containers are started on the machine, and automatically exposes them and gets certificates if they have the appropriate env variables. https://github.com/nginx-proxy/nginx-proxy

I checked the logs as you mentioned, and I think the proxy might be the culprit here actually.
I am getting a dropped_not_found error in the log, which lead me to this issue: #3758 which looks similar. I'm going to investigate a bit more.

[ruslandoga]

dropped_not_found is probably related to the domain (for which the event is posted) not being present in the database, i.e. there is no "site" on Plausible for this domain.

Relevant modules would be:

• https://github.com/plausible/analytics/blob/master/lib/plausible/site/gate_keeper.ex -- this is the check that probably drops the event

• analytics/lib/plausible/ingestion/event.ex

  Line 60 in 9d06d45

  case GateKeeper.check(domain) do

  -- this is where GateKeeper.check/1 is called

• analytics/lib/plausible/ingestion/request.ex

  Lines 154 to 179 in 9d06d45

  	defp map_domains(changeset, %{} = request_body) do
  	raw = request_body["d"] || request_body["domain"]
  	raw = if is_binary(raw), do: String.trim(raw)
  	case raw do
  	"" ->
  	Changeset.add_error(changeset, :domain, "can't be blank")
  	raw when is_binary(raw) ->
  	domains =
  	raw
  	|> String.split(",")
  	|> Enum.map(&sanitize_hostname/1)
  	Changeset.put_change(changeset, :domains, domains)
  	nil ->
  	from_uri = sanitize_hostname(Changeset.get_field(changeset, :uri))
  	if from_uri do
  	Changeset.put_change(changeset, :domains, [from_uri])
  	else
  	Changeset.add_error(changeset, :domain, "can't be blank")
  	end
  	end
  	end

  -- this is how the domains are extracted from the incoming request

[HalfVoxel]

Aha!
I figured out that due to a config change, we were accidentally sending events to plausible with a data-domain that included the "https://" part. I even looked at the request multiple times, but didn't notice anything that was off XD.

Now everything is working properly!

Might be a good thing to check in the client javascript and log a warning? Just to avoid other users making a similar mistake.