Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace ol-www1 with docker-based ol-www0 #4252

Closed
28 tasks done
cdrini opened this issue Dec 12, 2020 · 13 comments · Fixed by #4725
Closed
28 tasks done

Replace ol-www1 with docker-based ol-www0 #4252

cdrini opened this issue Dec 12, 2020 · 13 comments · Fixed by #4725
Assignees
@cclauss @cdrini
Labels
Affects: Configuration Related to the configuration of the dev/staging/prod environments, CI, docker, etc. [managed] Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] Module: Docker Issues related to the configuration or use of Docker. [managed] Priority: 2 Important, as time permits. [managed] Theme: Provisioning Type: Epic A feature or refactor that is big enough to require subissues. [managed] Type: Subtask of Epic A subtask that is part of the work breakdown of an epic issue (see comments). [managed]
Milestone
2023

Comments

@cdrini
Copy link
Collaborator

cdrini commented Dec 12, 2020 •
edited by cclauss
Loading

ol-www1 hosts static files as well as the nginx that serves the main site.

Related:

To Investigate:

  • ol-www1 crons: Sitemaps, ipstats
  • ol-www1 has some relation to sitemaps; make sure they're still generated. Might need to copy them over.
  • ol-www1's nginx logs are read to determine some of the openlibrary.org/stats . Confirm they all still work.
  • ipstats crons jobs
  • Log strategy: log rotation, where to store; need them to live a long time.

Considerations

If/when we remove volume mounts, our cron jobs may need to be re-architected for commands which ssh to hosts (as the code will now be inside docker and we'll need some way to trigger these commands in docker from ol-home0 + ssh)
@cdrini cdrini added Module: Docker Issues related to the configuration or use of Docker. [managed] Needs: Triage This issue needs triage. The team needs to decide who should own it, what to do, by when. [managed] Affects: Configuration Related to the configuration of the dev/staging/prod environments, CI, docker, etc. [managed] Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] labels Dec 12, 2020
@cdrini cdrini added this to the Next (proposed) milestone Dec 12, 2020
@cdrini cdrini added the Type: Subtask of Epic A subtask that is part of the work breakdown of an epic issue (see comments). [managed] label Dec 12, 2020
@cclauss
Copy link
Contributor

cclauss commented Dec 13, 2020 •
edited
Loading

Q: What is the upside of doing all this work? What will www0 give us that www1 does not currently deliver?

A: Being able to build all this static content less often and store those results in an easy-to-serve Docker container.

@cdrini
Copy link
Collaborator Author

cdrini commented Dec 28, 2020

@mekarpeles Note: Logs might need to be longer-lived on www1, and regularly uploaded to IA.

@cdrini
Copy link
Collaborator Author

cdrini commented Dec 28, 2020 •
edited by cclauss
Loading

@cclauss One purpose is being able to build static files at all :) Currently we have to build the static files on ol-home outside of docker; this is going to be deprecated as we work towards #4060 (merged) .

Another important purpose is that we make our nginx confs available to the community, so they can make changes which would currently be impossible for them to do (e.g. add HTTP/2 support).

@cdrini cdrini modified the milestones: Next (proposed), Active Sprint Jan 4, 2021
@cdrini cdrini self-assigned this Jan 4, 2021
@cdrini cdrini added the Priority: 1 Do this week, receiving emails, time sensitive, . [managed] label Jan 4, 2021
@mekarpeles mekarpeles mentioned this issue Jan 14, 2021
Open
24 tasks
@mekarpeles mekarpeles removed the Needs: Triage This issue needs triage. The team needs to decide who should own it, what to do, by when. [managed] label Jan 14, 2021
@mekarpeles mekarpeles mentioned this issue Jan 14, 2021
Closed
8 tasks
@cclauss cclauss mentioned this issue Jan 15, 2021
Closed
@cdrini
Copy link
Collaborator Author

cdrini commented Jan 28, 2021

Note: Might need GeoMind on this :/

@mekarpeles
Copy link
Member

@cclauss please check-in with @cdrini first to coordinate but I see this as a potential opportunity for you to help out with the initial step of provisioning ol-www0 like we we have some of the other VMs (e.g. to get Docker installed, etc)

@cdrini cdrini assigned mekarpeles and unassigned cclauss and cdrini Feb 9, 2021
@cclauss
Copy link
Contributor

cclauss commented Feb 10, 2021 •
edited
Loading

  • Install Docker and create docker group w/ nagios,cclauss,drini,mek as members (run scripts/setup_olserver.sh)
    • Modified scripts/setup_olserver.sh to not install olsystem, openlibrary, infogami, etc.

ol-www0% docker version

Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Fri Dec 18 12:15:19 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

ol-www0% cat /etc/group | grep docker

docker:x:116:nagios,cclauss,drini,me

@cclauss
Copy link
Contributor

cclauss commented Feb 10, 2021

Q: Which repos (olsystem, openlibrary, infogami, etc.) need to be installed on ol-www0? I assume we want a minimal set.

This was referenced Feb 10, 2021
Closed
Open
@cdrini cdrini removed this from the Sprint 2021-02 milestone Mar 1, 2021
@cdrini cdrini reopened this Dec 8, 2021
@cdrini cdrini added the Type: Epic A feature or refactor that is big enough to require subissues. [managed] label Dec 8, 2021
@mekarpeles mekarpeles added Priority: 2 Important, as time permits. [managed] and removed Priority: 1 Do this week, receiving emails, time sensitive, . [managed] labels Dec 13, 2021
This was referenced Dec 22, 2021
Closed
Closed
@cdrini cdrini modified the milestones: Active Sprint, Next (proposed) Dec 22, 2021
@cdrini cdrini self-assigned this Feb 13, 2023
@cdrini cdrini modified the milestones: Active Sprint, 2023 H1 Feb 13, 2023
@cclauss
Copy link
Contributor

cclauss commented Feb 22, 2023 •
edited
Loading

@cclauss cclauss mentioned this issue Mar 12, 2023
Merged
4 tasks
@mekarpeles mekarpeles modified the milestones: 2023 H1, Sprint 2023-03 Mar 15, 2023
@cclauss cclauss mentioned this issue Mar 30, 2023
Merged
@cclauss cclauss closed this as completed May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cclauss cclauss

@cdrini cdrini

Labels
Affects: Configuration Related to the configuration of the dev/staging/prod environments, CI, docker, etc. [managed] Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] Module: Docker Issues related to the configuration or use of Docker. [managed] Priority: 2 Important, as time permits. [managed] Theme: Provisioning Type: Epic A feature or refactor that is big enough to require subissues. [managed] Type: Subtask of Epic A subtask that is part of the work breakdown of an epic issue (see comments). [managed]
Projects
None yet
Milestone
2023
Development

Successfully merging a pull request may close this issue.

dockerfile + compose for ol-www0 web_nginx internetarchive/openlibrary
4 participants
@mekarpeles @cclauss @cdrini @jimchamp