Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Staging deployment failing #402

Closed
niraymak opened this issue Mar 16, 2021 · 3 comments · Fixed by #403, #404 or #518
Closed

Staging deployment failing #402

niraymak opened this issue Mar 16, 2021 · 3 comments · Fixed by #403, #404 or #518
Assignees
Labels
bug Something isn't working infrastructure priority Only assign this label if it's asked to assign this label

Comments

@niraymak
Copy link
Member

niraymak commented Mar 16, 2021

Describe the bug
The Github action which is responsible for the deploying the develop branch to staging environment is failing regularly lately.

https://github.com/DigitalExcellence/dex-backend/actions/workflows/staging-deployment.yml
https://github.com/DigitalExcellence/dex-frontend/actions/workflows/staging-deployment.yml

Ruben and I think this is happening because of the sleep time being too short in the GitHub action (see file below). We think the database is not yet down when the docker-compose up command is starting. If this will solve the problem, the action in production should also be extended.
https://github.com/DigitalExcellence/dex-backend/blob/develop/.github/workflows/staging-deployment.yml#L29

EDIT: EXTENDING TIME TO 25 SECONDS DID NOT HELP.
https://github.com/DigitalExcellence/dex-backend/actions/runs/657962692

Fixed it I think :)
I will test more but I think I found the problem, the compose file has a static IP listed in the database section, no idea what it is for but loads of people seem to advise not to use a static ip since docker can resolve ip's via the service discovery itself. I commented the ip out and the application runs fine without the ip. After some testing the problem seems to be gone :). Will test more during comming PR's and pipeline triggers before closing. - Mees

Expected behavior
GitHub action should succeed.

Screenshots
image

Production URL

Additional context
The frontend also has a GitHub action for staging deployment.

We fixed this on the server by running

docker-compose down
docker-compose up -d
@niraymak niraymak added the bug Something isn't working label Mar 16, 2021
@niraymak niraymak self-assigned this Mar 16, 2021
@niraymak niraymak mentioned this issue Mar 16, 2021
13 tasks
@niraymak niraymak reopened this Mar 16, 2021
@niraymak niraymak mentioned this issue Mar 16, 2021
13 tasks
@niraymak
Copy link
Member Author

Seems like 25s is not enough aswell, will try 90seconds now and test again.

@niraymak
Copy link
Member Author

Still an issue after setting the time to 90s.

https://github.com/DigitalExcellence/dex-backend/actions/runs/664727811

@niraymak niraymak removed their assignment Mar 23, 2021
@Brend-Smits
Copy link
Member

Brend-Smits commented Mar 24, 2021

The sleep was more of a temporary workaround. Consider setting up a proper solution like; https://docs.docker.com/compose/startup-order/ or via healthchecks(preferred I think): https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck in combination with depends_on

@MeesvanStraten MeesvanStraten added infrastructure priority Only assign this label if it's asked to assign this label labels Sep 16, 2021
@MeesvanStraten MeesvanStraten self-assigned this Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working infrastructure priority Only assign this label if it's asked to assign this label
Projects
None yet
4 participants