2535 - Crash Loop Back offs Error (#2577)

Note: The branch name is created without a # in the front, to test my
build and deployments from my local.

- [x] Investigate the issue and apply the fix on all our custom
Dockerfile (Api, web, workers, queue-consumers, load-test-gateway &
db-migration)
- [x] Most updated image in node has to be deployed


What does EACCES error means in node -
https://betterstack.com/community/guides/scaling-nodejs/nodejs-errors/#13-eacces

On Analysing the issue, i was able to understand that this issue happens
in certain version of node & npm libraries, some related bugs reported
in the years are given below for reference.
- npm/cli#5114
- https://stackoverflow.com/questions/71450392/npm-cache-issue
-
https://stackoverflow.com/questions/67163194/jenkins-build-issue-npm-err-your-cache-folder-contains-root-owned-files

The possible solutions given by the people are alwauys these 2
- give sudo permission and access for the user 1001 for the /.npm
directory/ - which is not always right to have a sudo persmission for a
user in dockerfile
- upgrade the node version as it states to a newer version that this
issue is not happening - going with this route as we have a newer
version of redhat that contains the new version from the one deplyoed
already with some security issues.


https://catalog.redhat.com/software/containers/ubi8/nodejs-18/6278e5c078709f5277f26998?architecture=amd64&image=65302e01ec5935b621691d22&container-tabs=packages

![image](https://github.com/bcgov/SIMS/assets/62901416/1f7136b4-3daf-43d3-9af3-6232865be1e3)


https://catalog.redhat.com/software/containers/ubi8/nodejs-18/6278e5c078709f5277f26998?architecture=amd64&image=6543c3d67371c4bd3014291a&container-tabs=packages
<img width="1290" alt="image"
src="https://github.com/bcgov/SIMS/assets/62901416/a492658d-bd7f-4a92-9513-2b26038a475c">

Analyzing the changelogs of npm js, there has been bugs related to cache
that has been fixed.
https://docs.npmjs.com/cli/v9/using-npm/changelog#981-2023-07-18
https://docs.npmjs.com/cli/v9/using-npm/changelog#967-2023-05-17


![image](https://github.com/bcgov/SIMS/assets/62901416/e35782a3-28f3-4c5c-982b-58b06ac6404b)
npm/cli#6464
The bug reported may not state the same issue we are facing but its
related to the cache error that is happening in the version we were
using.

So updating the redhat image to the latest version is what was taken as
an action to solve this issue.

Note:
https://app.zenhub.com/workspaces/student-information-management-system-5fce9df5aa1b45000e937014/issues/gh/bcgov/sims/2453
is also done as part of this PR.
- [x] Remove variables from Makefile
- [x] Remove references from docker-build.yml


As suggested by @andrewsignori-aot
#2577 (comment) changed
the permission of users in the group 0 for the folder ./.npm to have
write access

Container before assigning the write permission
<img width="889" alt="image"
src="https://github.com/bcgov/SIMS/assets/62901416/a8d95861-d538-4661-9e6c-429b30f2f6e3">

Container after assigning the write permission
<img width="740" alt="image"
src="https://github.com/bcgov/SIMS/assets/62901416/2343b44b-2b4b-4849-8f55-084beea91d0b">


Addded only the permissions for the ./.npm folder as in the past month
the failed logs show this error happening only in the ./.npm folder.

https://kibana-openshift-logging.apps.silver.devops.gov.bc.ca/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(message),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'8841c680-a15b-11eb-a4dc-e5bf19f04239',key:kubernetes.namespace_name,negate:!f,params:(query:'0c27fb-test',type:phrase),type:phrase,value:'0c27fb-test'),query:(match:(kubernetes.namespace_name:(query:'0c27fb-test',type:phrase))))),index:'8841c680-a15b-11eb-a4dc-e5bf19f04239',interval:auto,query:(language:lucene,query:'%22sudo%20chown%20-R%20%22'),sort:!('@timestamp',desc))

<img width="923" alt="image"
src="https://github.com/bcgov/SIMS/assets/62901416/ba796f55-5447-4c4a-9fef-aba1ab129d5c">

devops/Makefile

@@ -243,47 +243,47 @@ build-db-migrations:
 	test -n "$(BUILD_REF)"
 	test -n "$(DB_MIGRATIONS_BUILD_REF)"
 	@echo "+\n++ BUILDING DB migrations with tag: $(BUILD_REF)\n+"
-	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p BASE_IMAGE_NAME="nodejs-18" -p BASE_IMAGE_TAG="1-71.1697652955" -p BASE_IMAGE_REPO="artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/" -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/db-migrations/Dockerfile -p NAME=$(DB_MIGRATIONS_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
+	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/db-migrations/Dockerfile -p NAME=$(DB_MIGRATIONS_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
 	@oc -n $(BUILD_NAMESPACE) start-build bc/$(DB_MIGRATIONS_BUILD_REF) --wait
 
 build-api:
 	test -n "$(BUILD_NAMESPACE)"
 	test -n "$(BUILD_REF)"
 	test -n "$(API_BUILD_REF)"
 	@echo "+\n++ BUILDING API with tag: $(BUILD_REF)\n+"
-	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p BASE_IMAGE_NAME="nodejs-18" -p BASE_IMAGE_TAG="1-71.1697652955" -p BASE_IMAGE_REPO="artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/" -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/api/Dockerfile -p NAME=$(API_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
+	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/api/Dockerfile -p NAME=$(API_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
 	@oc -n $(BUILD_NAMESPACE) start-build bc/$(API_BUILD_REF) --wait
 
 build-workers:
 	test -n "$(BUILD_NAMESPACE)"
 	test -n "$(BUILD_REF)"
 	test -n "$(WORKERS_BUILD_REF)"
 	@echo "+\n++ BUILDING WORKERS with tag: $(BUILD_REF)\n+"
-	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p BASE_IMAGE_NAME="nodejs-18" -p BASE_IMAGE_TAG="1-71.1697652955" -p BASE_IMAGE_REPO="artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/" -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/workers/Dockerfile -p NAME=$(WORKERS_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
+	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/workers/Dockerfile -p NAME=$(WORKERS_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
 	@oc -n $(BUILD_NAMESPACE) start-build bc/$(WORKERS_BUILD_REF) --wait
 
 build-queue-consumers:
 	test -n "$(BUILD_NAMESPACE)"
 	test -n "$(BUILD_REF)"
 	test -n "$(QUEUE_CONSUMERS_BUILD_REF)"
 	@echo "+\n++ BUILDING QUEUE_CONSUMERS with tag: $(BUILD_REF)\n+"
-	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p BASE_IMAGE_NAME="nodejs-18" -p BASE_IMAGE_TAG="1-71.1697652955" -p BASE_IMAGE_REPO="artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/" -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/queue-consumers/Dockerfile -p NAME=$(QUEUE_CONSUMERS_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
+	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/queue-consumers/Dockerfile -p NAME=$(QUEUE_CONSUMERS_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
 	@oc -n $(BUILD_NAMESPACE) start-build bc/$(QUEUE_CONSUMERS_BUILD_REF) --wait
 
 build-load-test-gateway:
 	test -n "$(BUILD_NAMESPACE)"
 	test -n "$(BUILD_REF)"
 	test -n "$(LOAD_TEST_GATEWAY_BUILD_REF)"
 	@echo "+\n++ BUILDING LOAD_TEST_GATEWAY with tag: $(BUILD_REF)\n+"
-	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p BASE_IMAGE_NAME="nodejs-18" -p BASE_IMAGE_TAG="1-71.1697652955" -p BASE_IMAGE_REPO="artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/" -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/load-test-gateway/Dockerfile -p NAME=$(LOAD_TEST_GATEWAY_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
+	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)backend -p DOCKER_FILE_PATH=apps/load-test-gateway/Dockerfile -p NAME=$(LOAD_TEST_GATEWAY_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
 	@oc -n $(BUILD_NAMESPACE) start-build bc/$(LOAD_TEST_GATEWAY_BUILD_REF) --wait
 
 build-web:
 	test -n "$(BUILD_NAMESPACE)"
 	test -n "$(BUILD_REF)"
 	test -n "$(WEB_BUILD_REF)"
 	@echo "+\n++ BUILDING WEB with tag: $(BUILD_REF)\n+"
-	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p BASE_IMAGE_NAME="nginx-122" -p BASE_IMAGE_TAG="1" -p BASE_IMAGE_REPO="artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/" -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)web -p NAME=$(WEB_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
+	@oc -n $(BUILD_NAMESPACE) process -f $(BUILD_TEMPLATE_PATH) -p TAG=$(BUILD_REF) -p SOURCE_REPOSITORY_REF=$(BUILD_REF) -p SOURCE_CONTEXT_DIR=$(SOURCE_CONTEXT_DIR)web -p NAME=$(WEB_BUILD_REF) | oc -n $(BUILD_NAMESPACE) apply -f -
 	@oc -n $(BUILD_NAMESPACE) start-build bc/$(WEB_BUILD_REF) --wait
 
 init-patroni:

devops/openshift/docker-build.yml

@@ -5,28 +5,6 @@ kind: Template
 metadata:
   name: ${NAME}
 objects:
-  - apiVersion: image.openshift.io/v1
-    kind: ImageStream
-    metadata:
-      name: ${BASE_IMAGE_NAME}
-    spec:
-      lookupPolicy:
-        local: false
-  - apiVersion: v1
-    kind: ImageStreamTag
-    lookupPolicy:
-      local: false
-    metadata:
-      name: ${BASE_IMAGE_NAME}:${BASE_IMAGE_TAG}
-    tag:
-      annotations: null
-      from:
-        kind: DockerImage
-        name: ${BASE_IMAGE_REPO}${BASE_IMAGE_NAME}:${BASE_IMAGE_TAG}
-      importPolicy:
-        scheduled: true
-      referencePolicy:
-        type: Source
   - kind: ImageStream
     apiVersion: v1
     metadata:
@@ -88,11 +66,6 @@ parameters:
     value: Dockerfile
   - name: TAG
     value: "latest"
-  - name: BASE_IMAGE_REPO
-  - name: BASE_IMAGE_NAME
-    required: true
-  - name: BASE_IMAGE_TAG
-    required: true
   - name: CPU_LIMIT
     value: "2"
   - name: MEMORY_LIMIT

sources/packages/backend/apps/api/Dockerfile

@@ -1,5 +1,5 @@
 # Base Image
-FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-71.1697652955
+FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-81
 
 LABEL maintainer="BC GOV"
 
@@ -23,5 +23,10 @@ RUN npm run build api
 # Exposing application port
 EXPOSE ${PORT}
 
+# Grant access to group 0 to allow npm v9 to work
+# See: https://docs.openshift.com/container-platform/4.11/openshift_images/create-images.html#use-uid_create-images
+RUN chgrp -R 0 ./.npm && \
+    chmod -R g=u ./.npm
+
 # Entry point
 CMD [ "npm", "run", "start:prod:api" ]

sources/packages/backend/apps/db-migrations/Dockerfile

@@ -1,5 +1,5 @@
 # Base Image
-FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-71.1697652955
+FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-81
 
 LABEL maintainer="BC GOV"
 
@@ -15,5 +15,10 @@ COPY tsconfig.json ./
 COPY tsconfig.build.json ./
 COPY env-setup.js ./
 
+# Grant access to group 0 to allow npm v9 to work
+# See: https://docs.openshift.com/container-platform/4.11/openshift_images/create-images.html#use-uid_create-images
+RUN chgrp -R 0 ./.npm && \
+    chmod -R g=u ./.npm
+
 # Entry point
 CMD [ "npm", "run", "start:prod:db-migrations" ]

sources/packages/backend/apps/load-test-gateway/Dockerfile

@@ -1,5 +1,5 @@
 # Base Image
-FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-71.1697652955
+FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-81
 
 LABEL maintainer="BC GOV"
 
@@ -16,5 +16,10 @@ COPY ./libs ./libs
 # Building app
 RUN npm run build load-test-gateway
 
+# Grant access to group 0 to allow npm v9 to work
+# See: https://docs.openshift.com/container-platform/4.11/openshift_images/create-images.html#use-uid_create-images
+RUN chgrp -R 0 ./.npm && \
+    chmod -R g=u ./.npm
+
 # Entry point
 CMD [ "npm", "run", "start:prod:load-test-gateway" ]

sources/packages/backend/apps/queue-consumers/Dockerfile

@@ -1,5 +1,5 @@
 # Base Image
-FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-71.1697652955
+FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-81
 
 LABEL maintainer="BC GOV"
 
@@ -16,5 +16,10 @@ COPY ./libs ./libs
 # Building app
 RUN npm run build queue-consumers
 
+# Grant access to group 0 to allow npm v9 to work
+# See: https://docs.openshift.com/container-platform/4.11/openshift_images/create-images.html#use-uid_create-images
+RUN chgrp -R 0 ./.npm && \
+    chmod -R g=u ./.npm
+
 # Entry point
 CMD [ "npm", "run", "start:prod:queue-consumers" ]

sources/packages/backend/apps/workers/Dockerfile

@@ -1,5 +1,5 @@
 # Base Image
-FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-71.1697652955
+FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-81
 
 LABEL maintainer="BC GOV"
 
@@ -16,5 +16,10 @@ COPY ./libs ./libs
 # Building app
 RUN npm run build workers
 
+# Grant access to group 0 to allow npm v9 to work
+# See: https://docs.openshift.com/container-platform/4.11/openshift_images/create-images.html#use-uid_create-images
+RUN chgrp -R 0 ./.npm && \
+    chmod -R g=u ./.npm
+
 # Entry point
 CMD [ "npm", "run", "start:prod:workers" ]

sources/packages/web/Dockerfile

@@ -1,4 +1,4 @@
-FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-71.1697652955 AS builder
+FROM artifacts.developer.gov.bc.ca/redhat-access-docker-remote/ubi8/nodejs-18:1-81 AS builder
 
 # Application Port.
 ENV PORT 3030