How to update the repositories configured in repos.yaml and addons.yaml using production and test modes?

[pleirb]

In development mode it is easy: we use invoke git-aggregate, but in production and test modes I still can't figure out what the appropriate procedure is. Apparently in this case all the modules remain inside when the image is built. But I don't want to build it again, I just want to update the modules.

[pedrobaeza]

You have to build. There's no other option.

[pleirb]

I understand, so if there is an update to a module or something new that I want to add, should I stop the production environment? Isn't it a little uncomfortable? Wouldn't it be preferable for this functionality to be the same as development mode?

[pedrobaeza]

No, you don't have to stop production environment. Just build while having the instance running, do docker-compose run --rm -l traefik.enable=false odoo click-odoo-update, and then docker-compose up -d to replace the environment with the new one. The downtime is nearly 0.

[pleirb]

Let's see, wait, this is interesting. I have created the image again with:
docker compose build
But even stopping the production environment, the image is built using the repos.yaml and addons.yaml files that are inside my container. I believed that if I changed repos.yaml and addons.yaml on the host side, this would take effect.
Then what should I do? The correct way is edited these files from within the container directly or by copying the host version to the odoo container?
And I can't quite understand this thing about replacing one environment with another.

[pedrobaeza]

You need to make docker compose build --no-cache for forcing to redownload the repositories. An extra --pull will also download images for the rest of the components (mail, database, etc).

[pleirb]

It's true, this had even happened to me before in another environment #399
I'm doing a lot of testing and I didn't realize there is a huge CybroOdoo repository that gives me errors and doesn't let me finish the build. I tried to reduce the load by configuring the number of modules in addons.yaml, but it still seems to insist on downloading everything. I searched for the solution on several sites, but was unsuccessful. I only managed to download completely once and I think it was because I had switched to a high-speed internet provider. The error I get is this:

12.89 (INFO) [18:52:32] git_aggregator.repo  serpent   End aggregation of /opt/odoo/custom/src/serpent
156.9 error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
156.9 error: 8039 bytes of body are still expected
156.9 fetch-pack: unexpected disconnect while reading sideband packet
156.9 fatal: early EOF
156.9 fatal: index-pack failed
156.9 (ERROR) [18:54:56] git_aggregator.repo  design-themes   /opt/odoo/custom/src/design-themes> error calling ('git', 'fetch', '--depth', '100', 'origen', '15.0')
157.5 error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: CANCEL (err 8)
157.5 error: 6761 bytes of body are still expected
157.5 fetch-pack: unexpected disconnect while reading sideband packet
157.5 fatal: early EOF
157.5 fatal: index-pack failed
157.5 (ERROR) [18:54:57] git_aggregator.repo  cybrosys   /opt/odoo/custom/src/cybrosys> error calling ('git', 'fetch', '--depth', '1', 'origen', '15.0')
162.5 error: 7028 bytes of body are still expected
162.5 fetch-pack: unexpected disconnect while reading sideband packet
162.5 fatal: early EOF
162.5 fatal: index-pack failed
162.5 (ERROR) [18:55:02] git_aggregator.repo  odoo   /opt/odoo/custom/src/odoo> error calling ('git', 'fetch', '--depth', '1', 'odoo', '15.0')
162.5 Traceback (most recent call last):
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/main.py", line 219, in aggregate_repo
162.5     repo.aggregate()
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 183, in aggregate
162.5     self.fetch()
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 206, in fetch
162.5     self.log_call(cmd, cwd=self.cwd)
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 160, in log_call
162.5     ret = callwith(cmd, **kw)
162.5   File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
162.5     raise CalledProcessError(retcode, cmd)
162.5 subprocess.CalledProcessError: Command '('git', 'fetch', '--depth', '100', 'origen', '15.0')' returned non-zero exit status 128.
162.5 Traceback (most recent call last):
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/main.py", line 219, in aggregate_repo
162.5     repo.aggregate()
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 183, in aggregate
162.5     self.fetch()
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 206, in fetch
162.5     self.log_call(cmd, cwd=self.cwd)
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 160, in log_call
162.5     ret = callwith(cmd, **kw)
162.5   File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
162.5     raise CalledProcessError(retcode, cmd)
162.5 subprocess.CalledProcessError: Command '('git', 'fetch', '--depth', '1', 'origen', '15.0')' returned non-zero exit status 128.
162.5 Traceback (most recent call last):
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/main.py", line 219, in aggregate_repo
162.5     repo.aggregate()
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 183, in aggregate
162.5     self.fetch()
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 206, in fetch
162.5     self.log_call(cmd, cwd=self.cwd)
162.5   File "/usr/local/lib/python3.8/site-packages/git_aggregator/repo.py", line 160, in log_call
162.5     ret = callwith(cmd, **kw)
162.5   File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
162.5     raise CalledProcessError(retcode, cmd)
162.5 subprocess.CalledProcessError: Command '('git', 'fetch', '--depth', '1', 'odoo', '15.0')' returned non-zero exit status 128.
162.5 Traceback (most recent call last):
162.5   File "/usr/local/bin/autoaggregate", line 170, in <module>
162.5     aggregate(REPOS_YAML)
162.5   File "/usr/local/bin/autoaggregate", line 72, in aggregate
162.5     check_call(
162.5   File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
162.5     raise CalledProcessError(retcode, cmd)
162.5 subprocess.CalledProcessError: Command '['gitaggregate', '--expand-env', '--config', '/opt/odoo/custom/src/repos.yaml', '--log-level', 'INFO', '--jobs', '8', 'aggregate']' returned non-zero exit status 1.
162.5 Traceback (most recent call last):
162.5   File "/opt/odoo/common/build", line 30, in <module>
162.5     subprocess.check_call(command)
162.5   File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
162.5     raise CalledProcessError(retcode, cmd)
162.5 subprocess.CalledProcessError: Command '/opt/odoo/common/build.d/100-repos-aggregate' returned non-zero exit status 1.
------
failed to solve: process "/bin/sh -c /opt/odoo/common/build && sync" did not complete successfully: exit code: 1

[pedrobaeza]

I recommend you to avoid all the modules except OCA ones.

[pleirb]

I have tried removing these third party repositories and manually placing these modules in the private folder, and it works better. Likewise, it has failed even with those from OCA. But by insisting several times I have managed to get it to end completely. Definitely, the GitHub servers are punishing me for making so many downloads, they have probably registered me with the IP

[pleirb]

I have tried this and it works, but I still think it is not practical when every so often you want to add a new repository to production mode or you simply want to update existing modules. Reconstructing the image in my case takes a long time.
Has anyone experimented with this module?:
https://github.com/OCA/interface-github/tree/16.0/github_connector#repositories-synchronization
I haven't been able to get this to work yet, but if it were possible to download modules from the GitHub repositories directly from a graphical interface from Odoo, that would be great! Even more great if it were integrated into the odoo application module so that it allows you to download the modules from the repository you want and allow the installation like any other preloaded module in the image

[pleirb]

Bueno, hay algunas optimizaciones en los archivos Python compilados hechos durante la compilación, pero puedes probar lo que considera correcto.

You mean that during the compilation (docker compose build) the python files that correspond to the oca and third-party modules go through an optimization process that does depend on a volume that is updated dynamically and externally with invoke git-aggretate for example Would this optimization be lost?

[pedrobaeza]

Yes

[pleirb]

Sí,

I understand, I wonder what optimizations these are. I really need to mention all this because in production we usually add modules and update quite often. This involves compiling the image, which takes time, uploading to dockerhub (push) and downloading to the server (pull) another extra time. In total, it is a process of almost 1 hour when there are times when it is only a matter of updating a tiny detail in some module.
That's why I try to see what shorter time options there may be to update the modules in production.

[pedrobaeza]

My builds only takes 2-3 minutes, so that's very weird...

[pleirb]

Mis construcciones sólo tarda 2-3 minutos, así que eso es muy raro...

Wow! , that time would be reasonable, it takes me 40 minutes, then I have to upload and download the image, another 20 minutes (depending on the internet connection)
An image that results in approximately 7GB
I have a Corei5 1135 computer and 16GB of RAM