Citus Data Persistance issue

[abhi7788]

Hi Team,

I am facing a challenge when running the citus on kubernetes. I am using citus docker image along with citus memebership manager provided in your repor.After the first deployment on K8. my database runs fine and i am able to query all the tables. However If i scale it down and rescale it back it the database start throwing following error :

[Code: , SQL State: 42P01] ERROR: relation "tbl_102040" does not exist
Where: while executing command on citus-workers-0.citus-workers:5432

Looking at the details in the log it seems like the postgressql.conf has some issue on the worker node while getting started. I can see following entries in log:

2020-04-16 12:27:28.728 UTC [46] LOG: configuration file "/var/lib/postgresql/data/pgdata/postgresql.conf" contains errors; unaffected changes were applied
CREATE EXTENSION
2020-04-16 12:27:28.870 UTC [65] LOG: starting maintenance daemon on database 13408 user 10
2020-04-16 12:27:28.870 UTC [65] CONTEXT: Citus maintenance daemon for database 13408 user 10
UPDATE 1
COMMIT
2020-04-16 12:27:28.879 UTC [46] LOG: received fast shutdown request
waiting for server to shut down....2020-04-16 12:27:28.880 UTC [46] LOG: aborting any active transactions
2020-04-16 12:27:28.880 UTC [65] FATAL: terminating connection due to administrator command
2020-04-16 12:27:28.880 UTC [65] CONTEXT: Citus maintenance daemon for database 13408 user 10
2020-04-16 12:27:28.882 UTC [46] LOG: background worker "logical replication launcher" (PID 54) exited with exit code 1
2020-04-16 12:27:28.882 UTC [46] LOG: background worker "Citus Maintenance Daemon: 13408/10" (PID 65) exited with exit code 1
2020-04-16 12:27:28.882 UTC [48] LOG: shutting down
2020-04-16 12:27:28.903 UTC [46] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
2020-04-16 12:27:29.013 UTC [1] LOG: number of prepared transactions has not been configured, overriding
2020-04-16 12:27:29.013 UTC [1] DETAIL: max_prepared_transactions is now set to 200
2020-04-16 12:27:29.013 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2020-04-16 12:27:29.014 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-04-16 12:27:29.027 UTC [67] LOG: database system was shut down at 2020-04-16 12:27:28 UTC
2020-04-16 12:27:29.033 UTC [1] LOG: database system is ready to accept connections