'Password Expired' Error

[dexma-dev]

Description of the issue

Helm deployment is not working properly. I followed all steps described on the website.
The 'scheduler' deployment and all 'workers' fail to start.

Steps to reproduce the issue

1. install chart via helm
2. wait for a couple of seconds

Expected result

pods should start up

Stacktrace / full error message if available

error message in scheduler and worker pods.

You are required to change your password immediately (password expired)
Changing password for frappe.
Current password: su: Authentication token manipulation error

[revant]

I'm not able to reproduce this issue.

Server datetime is setup correctly?

Click on details section for screen capture gif.

Install prerequisites

install erpnext helm chart

check logs of running containers

[revant]

Re open if needed

[mind1949]

when i deploy frappe , i encountered the same problem @revant

[revant]

I just deployed on aws eks and gke, I encountered no such error there.

[ajs11174]

@revant Just ran into this same issue. I have an erpnext system that I turn on once every few weeks, do some work and then shut it down. When I went to turn it on today, I ran into this issue.

[dexma-dev]

@revant i am still having this issue. I think you should reopen this issue and we should find the cause of it.
It is most likely not connected to wrong server datetime, I checked that.

[ajs11174]

@revant Can you reopen this? I just deleted the helm chart, deleted the erpnext namespace, recreated a blank erpnext namespace, and did a fresh helm install. The same message is appearing regarding the password change.

[revant]

which kubernetes environment are you using?

I can confirm it is working on scaleway, gke and eks on my end.

[ajs11174]

I have a cluster which I prop up with kubeadm. I'm using kubernetes version 1.18 and frappe v12.10 for the python image (frappe/erpnext-worker).

This is what my cluster looks like after running helm delete to remove erpnext:

Basically this is the bare minimum to run erpnext: MariaDB, Shared Filesystem and a few extras: Ingress, Cert Manager & external DNS.

Here is values.yaml file for the helm install:

# Default values for erpnext.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

nginxImage:
  repository: frappe/erpnext-nginx
  tag: v12.10.0
  pullPolicy: IfNotPresent

pythonImage:
  repository: frappe/erpnext-worker
  tag: v12.10.0
  pullPolicy: IfNotPresent

socketIOImage:
  repository: frappe/frappe-socketio
  tag: v12.7.3
  pullPolicy: IfNotPresent

# Asset Image Env Variables
frappePyPort: "8000"
socketIOPort: "9000"

# Python Image Env Variables
mariadbHost: "mariadb.mariadb.svc.cluster.local"

redisQueueHost: ""
redisCacheHost: ""
redisSocketIOHost: ""

migrateJob:
  # Set this to true to run migrate as part of helm install/upgrade
  enable: false
  backup: true

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true

podSecurityContext:
  supplementalGroups: [1000]

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 80

# Additional Services
socketIOService:
  type: ClusterIP
  port: 9000

redisSocketIOService:
  type: ClusterIP
  port: 11000

redisQueueService:
  type: ClusterIP
  port: 12000

redisCacheService:
  type: ClusterIP
  port: 13000

persistence:
  enabled: true
#   existingClaim: "erpnext-pvc"
#   size: 8Gi
  storageClass: "aws-efs"

resources: {}
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

After running helm install I get the errors listed above.

---

At this point I decided to upgrade Kubernetes to version 1.19.3

---

Then I upgraded to version 12.12.1 of erpnext with the following config:

# Default values for erpnext.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

nginxImage:
  repository: frappe/erpnext-nginx
  tag: v12.12.1
  pullPolicy: IfNotPresent

pythonImage:
  repository: frappe/erpnext-worker
  tag: v12.12.1
  pullPolicy: IfNotPresent

socketIOImage:
  repository: frappe/frappe-socketio
  tag: v12.10.2
  pullPolicy: IfNotPresent

# Asset Image Env Variables
frappePyPort: "8000"
socketIOPort: "9000"

# Python Image Env Variables
mariadbHost: "mariadb.mariadb.svc.cluster.local"

redisQueueHost: ""
redisCacheHost: ""
redisSocketIOHost: ""

migrateJob:
  # Set this to true to run migrate as part of helm install/upgrade
  enable: false
  backup: true

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  # Specifies whether a service account should be created
  create: true

podSecurityContext:
  supplementalGroups: [1000]

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

service:
  type: ClusterIP
  port: 80

# Additional Services
socketIOService:
  type: ClusterIP
  port: 9000

redisSocketIOService:
  type: ClusterIP
  port: 11000

redisQueueService:
  type: ClusterIP
  port: 12000

redisCacheService:
  type: ClusterIP
  port: 13000

persistence:
  enabled: true
  # existingClaim: ""
  size: 8Gi
  storageClass: "aws-efs"

resources: {}
  # If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

Not sure what the exact issue was but I was able to get the system back up and running after upgrading to the latest versions of kubernetes and erpnext.

[revant]

If you have setup nodes/master(s)/VMs manually ensure the timezone, ntp settings are appropriate.

I was able to setup successfully on minikube. I tried it locally.

[mtogiz]

I also got this problem when i deploy on my local (tested on single-bench and multi-bench). I'm using v13-beta, it still work fine but today when i open docker and this issue happened 🤷‍♂️

If you have setup nodes/master(s)/VMs manually ensure the timezone, ntp settings are appropriate.

I was able to setup successfully on minikube. I tried it locally.

[Thunderbottom]

Seems like the issue is caused because of PASS_MAX_DAYS being set within the python-3.7 docker image:

https://github.com/bitnami/bitnami-docker-python/blob/9507e4f3d02b48c22be87a45697f60aebcb0dce2/3.7-prod/debian-10/Dockerfile#L17-L19

[revant]

If that's the case can anyone check setting PASS_MAX_DAYS=99999 during run time?

[mtogiz]

Seems like the issue is caused because of PASS_MAX_DAYS being set within the python-3.7 docker image:

https://github.com/bitnami/bitnami-docker-python/blob/9507e4f3d02b48c22be87a45697f60aebcb0dce2/3.7-prod/debian-10/Dockerfile#L17-L19

Is there anyway to set PASS_MAX_DAYS in docker compose file, bro?

[Thunderbottom]

You can mount a file to /etc/login.defs without the values. I suppose if no value is specified (or -1), it disables the restriction. From the man page for useradd:

PASS_MAX_DAYS (number)
   The maximum number of days a password may be used. If the password
   is older than this, a password change will be forced. If not
   specified, -1 will be assumed (which disables the restriction).

PASS_MIN_DAYS (number)
   The minimum number of days allowed between password changes. Any
   password changes attempted sooner than this will be rejected. If
   not specified, -1 will be assumed (which disables the restriction).

PASS_WARN_AGE (number)
   The number of days warning given before a password expires. A zero
   means warning is given only upon the day of expiration, a negative
   value means no warning is given. If not specified, no warning will
   be provided.

If this works then we can add it to the Dockerfile.

[revant]

Is there anyway to set PASS_MAX_DAYS in docker compose file, bro?

compose file? pass it as environment variables for services using frappe/erpnext-worker image.

for Kubernetes deployments do

kubectl edit deployments -n <erpnext-namespace> <deployment-name-using-python-image>

[mtogiz]

I tested on my local host and see no /etc/login.defs when mapping /etc to host folder. I deleted all image and download newest v13-beta and now it worked for me. Thank you all for your support

[revant]

The change is not yet in any image.

May be just the creation of new pods solved it for a while.

[chungyan5]

i updated /etc/login.defs by docker volume mounting, it seems not work.

I tried a short team solution as:

• stop all containers
• remove frappe/erpnext-worker:v12 image
• re-build again
  it works.

i login into containers:

• /etc/login.defs as

PASS_MAX_DAYS   99999
PASS_MIN_DAYS   0
PASS_WARN_AGE   0

• i tried below:

root@370ea3bc7dbc:/home/frappe/frappe-bench/sites# chage -l frappe
Last password change                                    : Nov 06, 2020
Password expires                                        : Feb 04, 2021
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 90
Number of days of warning before password expires       : 7

It seems no extended the expiring date.

[revant]

Seems like the issue is caused because of PASS_MAX_DAYS being set within the python-3.7 docker image:

https://github.com/bitnami/bitnami-docker-python/blob/9507e4f3d02b48c22be87a45697f60aebcb0dce2/3.7-prod/debian-10/Dockerfile#L17-L19

If it is set in base image should we really set it again in our image?

I feel just recreate the containers, that will reset the limits.

[revant]

If anyone wants to remove the security feature from base image please send PR.

I personally don't wish to remove the security feature. For people who need to use containers like VMs, re create the containers when this happens and it'll solve the problem.

[jmfarina]

Hello, sorry to resurrect this, but I seem to be experiencing this same problem and I can't find anything else or figure it out.

I'm running ERPNext v12 using the method recommended in the docs, that is, with Docker.
It had been working fine for a while, but all of a sudden (I think after a reboot) it stopped working and all I find in the logs is:

You are required to change your password immediately (password expired)
Changing password for frappe.
Current password: su: Authentication token manipulation error

I had already tried stopping and removing the containers, but the error persists. Maybe I'm doing something wrong? could you provide details as to how to do it properly, if that's the solution?
Also, I was wondering what exactly is the nature of the error: is it that some user's password expired? why would that prevent the app from starting?

Thanks!

[revant]

The container starts as root user,

docker-entrypoint.sh switches the user to frappe to run any frappe process.

can you set PASS_MAX_DAYS=99999 in docker-compose for all containers using worker image and try to re-create containers?

[jmfarina]

Thanks for your reply, I understand now.
I couldn't get it to work, though. I'm not sure if I'm doing something wrong here... I've stopped and rebuilt all containers, previously having changed the docker-compose.yml to include the specified parameter for all worker images, as follows:

version: "3"

services:
  traefik:
    image: "traefik:v2.2"
    restart: unless-stopped
    command:
      - "--log.level=DEBUG"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--providers.docker=true"
      - "--providers.file.directory=/etc/traefik/dynamic_conf"
    labels:
      # enable traefik
      - "traefik.enable=true"
      # global redirect to https for production only
      - "${HTTPS_REDIRECT_RULE_LABEL}"
      - "${HTTPS_REDIRECT_ENTRYPOINT_LABEL}"
      - "${HTTPS_REDIRECT_MIDDLEWARE_LABEL}"
      # middleware redirect for production only
      - "${HTTPS_USE_REDIRECT_MIDDLEWARE_LABEL}"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./certs/:/certs/:ro
      - ./traefik.yml:/etc/traefik/dynamic_conf/conf.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

  erpnext-nginx:
    image: frappe/erpnext-nginx:${ERPNEXT_VERSION}
    restart: unless-stopped
    environment:
      - FRAPPE_PY=erpnext-python
      - FRAPPE_PY_PORT=8000
      - FRAPPE_SOCKETIO=frappe-socketio
      - SOCKETIO_PORT=9000
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.erpnext-nginx.rule=Host(${SITES})"
      - "${ENTRYPOINT_LABEL}"
      - "${CERT_RESOLVER_LABEL}"
      - "traefik.http.services.erpnext-nginx.loadbalancer.server.port=80"
    volumes:
      - sites-vol:/var/www/html/sites:rw
      - assets-vol:/assets:rw

  erpnext-python:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: unless-stopped
    environment:
      - MARIADB_HOST=${MARIADB_HOST}
      - REDIS_CACHE=redis-cache:6379
      - REDIS_QUEUE=redis-queue:6379
      - REDIS_SOCKETIO=redis-socketio:6379
      - SOCKETIO_PORT=9000
      - AUTO_MIGRATE=1
      - PASS_MAX_DAYS=99999
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - assets-vol:/home/frappe/frappe-bench/sites/assets:rw

  frappe-socketio:
    image: frappe/frappe-socketio:${FRAPPE_VERSION}
    restart: unless-stopped
    depends_on:
      - redis-socketio
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

  erpnext-worker-default:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: unless-stopped
    command: worker
    environment:
      - PASS_MAX_DAYS=99999
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

  erpnext-worker-short:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: unless-stopped
    command: worker
    environment:
      - WORKER_TYPE=short
      - PASS_MAX_DAYS=99999
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

  erpnext-worker-long:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: unless-stopped
    command: worker
    environment:
      - WORKER_TYPE=long
      - PASS_MAX_DAYS=99999
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

  erpnext-schedule:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: unless-stopped
    command: schedule
    environment:
      - PASS_MAX_DAYS=99999
    depends_on:
      - redis-queue
      - redis-cache
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

  redis-cache:
    image: redis:latest
    restart: unless-stopped
    volumes:
      - redis-cache-vol:/data

  redis-queue:
    image: redis:latest
    restart: unless-stopped
    volumes:
      - redis-queue-vol:/data

  redis-socketio:
    image: redis:latest
    restart: unless-stopped
    volumes:
      - redis-socketio-vol:/data

  mariadb:
    image: mariadb:10.3
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    volumes:
      - ./installation/frappe-mariadb.cnf:/etc/mysql/conf.d/frappe.cnf
      - mariadb-vol:/var/lib/mysql

  site-creator:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: unless-stopped
    command: new
    environment:
      - PASS_MAX_DAYS=99999
    depends_on:
      - erpnext-python
    environment:
      - SITE_NAME=${SITE_NAME}
      - DB_ROOT_USER=${DB_ROOT_USER}
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - INSTALL_APPS=${INSTALL_APPS}
      - PASS_MAX_DAYS=99999
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw

volumes:
  mariadb-vol:
  redis-cache-vol:
  redis-queue-vol:
  redis-socketio-vol:
  assets-vol:
  sites-vol:
  cert-vol:

Log files still report the same password error.
Any ideas of what could be wrong, or what other things I could check?