Persisting workspaces on Dockerized Che

[wernight]

Using Docker to run che-server and persisting /home/user/che/workspaces and /home/user/che/storage. However even though those directories have something, after restarting Che it seems that it lost my workspaces (and associated projects).

(Note: I used CHE_WORKSPACE_STORAGE to change location of the workspaces)

[TylerJewell]

Please be explicit by showing every command you issued. The syntax for running Che CLI vs. Che Launcher vs. Che Docker Container are different. The variables are different and the behaviors are different.

If you are trying to run codenvy/che-server, the syntax for persisting workspaces and projects is different than the CLI.

[wernight]

I can only approximate as I'm using Kubernetes but the equivalent should be:

$ docker run -d --privileged --name docker -v /home/user/che/lib:/home/user/che/lib-copy -v $PWD/storage:/home/user/storage docker:dind
$ docker run --rm --link docker -e DOCKER_HOST=tcp://docker:2375 -e CHE_WORKSPACE_STORAGE=/home/user/che/storage -v $PWD/storage:/home/user/storage codenvy/che-server -s:uid -s:client run

(I need to check the lib mounting)

Would you know which folder normally contains the info that a workspace should exist?

[TylerJewell]

This is way different from the syntax that we support. You will have to explain how the dind container and the linked container running Che are associated. The syntax for running codenvy/che-server is documented here: https://eclipse-che.readme.io/v5.0/docs/usage-docker-server

# Run the Che container with:
docker run --net=host \
           --name che \
           --restart always \
           -p 9001:8080 \
           -v /var/run/docker.sock:/var/run/docker.sock \
           -v /home/user/che/lib:/home/user/che/lib-copy \
           -v /home/user/che/workspaces:/home/user/che/workspaces \
           -v /home/user/che/storage:/home/user/che/storage \
           -v /local:/container \
           -v /home/my_assembly:/home/user/che \
           -e CHE_LOCAL_CONF_DIR=/container \
           -e DOCKER_MACHINE_HOST=172.17.0.1 \
           codenvy/che-server --remote:1.1.1.1

[wernight]

In that example you're mounting /hom/user/che multiple times which shouldn't work. However I didn't try CHE_LOCAL_CONF_DIR; I'll try.

[TylerJewell]

Sure it works - you cannot mount the same exact folder twice, but Docker treats sub-folders differently. You can volume mount sub-folders for /home/user/che as many times as you want.

[wernight]

• CHE_LOCAL_CONF_DIR seems to do nothing (nothing got stored under that folder).
• /home/user/che/storage seems to only contain config and nothing about workspaces.
• /home/user/che/workspaces seems to contain workspaces data, but only that, and Che seems not to use that when restarting.

I tryed to mount and store the entire /home/user/che but outside of making it difficult because it's mounted on two containers but for one lib to mounted as lib-copy, it also fails because it's not initially empty.

[TylerJewell]

Take a look at the source code of github.com/eclipse/Che-dockerfiles/Che-launcher. This is the docker container which launches the Che server and you can see the syntax it uses to manage all of the configuration.

[ghost]

@wernight When set, the value of CHE_LOCAL_CONF_DIR is where Che will look for a custom props file if any.

/home/user/che/storage is where info abut workspaces is stored
/home/user/che/workspaces is where workspace projects are stored

CHE_WORKSPACE_STORAGE will be translated into the right mounts only when launched by the cli.

Also, in your example, you expect storage to be in /home/user/storage but then in another container it's /home/user/che/storage. Perhaps I do not fully understand what linking should do in your particular case.

If you want to run che-server image directory, you should use che.properties to override the defaults.

On the host, crate che.properties file:

che.conf.storage=/your/custom/path

In your run syntax:

-e "CHE_LOCAL_CONF_DIR=/home/user/.che" -v $(pwd)/che.properties:/home/user/.che/che.properties

[wernight]

I looked at default /home/user/che/conf/che.properties and all folders that it defines are:

• workspaces
• stacks
• storage

So I see no reason to use my custom che.properties.

I'm not persisting /home/user/che/lib but that doesn't seem to contain workspace data.

workspaces and storage I'm persisting. Stacks seems to be built-in and I'm not persisting. workspaces contains only one directory per workspace which is even empty initially. storage contains only a preferences.json with only settings and nothing when I just create workspaces.

So I see no place that contains for example the workspace memory or its kind. For example I'd expect it stored somewhere the equivalent of the JSON when exporting a workspace as file from Che. I even ran a grep on the entire file system looking at the workspace name but only found logs matching.

I tried persisting the entire /home/user/che but then che doesn't start properly. Its looks like this is only to run with custom Che binaries (not what I want).

So here are my steps:

1. Start Che
2. Create workspace, blank, any name (optionally start the workspace)
3. Stop Che container
4. Start Che in a new container (while persisting the workspaces and storage directories)

[ghost]

@wernight when in a che-server container, can you see jsons in /home/user/che/storage? Can you confirm this path is mounted into the host when you run docker inspect for this container.

[wernight]

At step 2 above I notice that:

• workspaces only contains an empty dir (equivalent of mkdir workspaces/foo)
• storage is empty

[ghost]

@wernight jsons are created after server is correctly shut down. This isn't the case anymore since we use in memory database. What Che version do you use?

Currently, Che stores database files in /storage/db

[wernight]

Oh.. may be there is an issue in the shutdown then. I'm sending a TERM signal and waiting for 30 sec before possibly sending a KILL signal.

I'll try calling /home/user/che/bin/che.sh stop first.

[TylerJewell]

Take a look at how the che-launcher container stops the che-server container here:
https://github.com/eclipse/che-dockerfiles/blob/master/che-launcher/launcher_cmds.sh#L62

[wernight]

Fixed! Awesome!

[wernight]

It'd however be simpler if this was done on sign TERM.

[wernight]

Sorry re-opening just to clarify: Does it also periodically does the same as che.sh stop? Else if Che crashes (for whatever reason) it seems there could be a large data loss.

[TylerJewell]

In the 4.7.2 release train, we had a persistence model where the persistence was updated periodically. However, starting in the 5.0 release train, we rewrote the persistence model, and now persistence items are updated immediatly when an in-memory object is changed. So, this should prevent any data loss. The only risk for data loss in the 5.0 version is if you have modified non-project files in a workspace and have not snapshot it. There is a workaround for that, too - because evne if you are using a database, you could just volume mount that database in a special volume in Che, so that would be automatically persisted.

[wernight]

I don't remember the DB being mounted by default using Docker. Doesn't sound too bad though at the moment (in the sense it's still important but less than other issues). Cool, then 5.0 would be a lot safer.

[TylerJewell]

With 5.0 - the db is disk-based derby database that runs within the JVM. There is a new directory that it is mounted to, and the tables are stored as files in the /storage folder. In Codenvy, the database is postgres - but it uses a similar JPA driver. And coming soon, we'll make Codenvy deployable as a set of docker compose services - where the postgres container will be a separate container with its contents mounted using a volumes-from reference.

[wernight]

Sounds good, as long as it's not expecting files to exist at that location initially. Should work with an empty directory (saying that because with volumes-from it's possible to point to an image which has already default data).

[roynasser]

Hi, I've been following this thread and another about how to work with volume mounts for bringing in external sourcecode into the che environment and and for some reason i'm having some problems... I was able to set the CHE_DATA_FOLDER to a local directory (using the docker launcher), and I see a storage and a workspaces directory. I would expect the workspaces to contain my workspaces, however, although I do see a workspace directory with the same name as a workspace I launched in che, the directory is empty... the "storage" directory contains some json files and an "images" directory - I assume this is the DB and other stuff...

Am I understanding it wrong? Should the workspace directroy be empty? Is the docker launcher the prefered way to use che? I thought it would be but then there are some more recent suggestions to use CLI? Can someone maybe share a compose file they are working with that shows Che as well as a volume mount on a filecontainer, for example? I'm a bit loss at the exerpts of docker-compose that were shared. thanks!

[TylerJewell]

Yes, when you set that folder and then create a workspace with a project, the files of the project should be there. The database storage is only for internal Che configuration. Please see the PR below with some instructions on a much simplified approach to managing user data in Che. It was a weekend project that works to make sure certain needs are self handled in the new Che server container to more easily support configurations like yours.

It has not been merged yet but it shows the general direction. The current Che launcher should always work for you however so if you are setting that and projects are not written into it then something is wrong.

#2786

[roynasser]

Thats strange then @TylerJewell... the storage directory is there, and so is the workspaces... And adding a workspace definitely added a directory in the workspaces directory, however that directory is empty... I tried adding a second project manually, and still its empty...

I'll take a look at the PR you linked and wait a bit longer to try again... I'm almost ready to "need che" :) We are basically done putting all our dev environment on docker, with different containers for local-development vs "production"/in container files, and soon I'd like to start working on exposing che as a "development server" for our programmers...

I'll keep an eye out for this and will do more testing in the coming weeks.

[TylerJewell]

There are some quirks in the current che-server configuration where if you are running che inside of a VM, like with docker for mac, that we have additional variables that must be set. If you are calling hte che-server directly, you have to set these values. But the che-launcher is designed to handle all of these situations automatically by detecting where it is, and then setting values. So if the che-launcher is not catching that then it's a head scratcher and could be a rare bug given your OS / VM combination.

[roynasser]

Hi @TylerJewell is there anything I can generate in the form of logs or more verbose output that can help diagnose? This is running on a Ubuntu machine with docker 0.12 I believe... Its a baremetal server on scaleway, so its not a VM or vagrant environment... if there is anything I can get from the server or run for diagnose let me know...

edit: correcting, the docker v is 1.12.1 not 0.12.1...apologies, that wouldnt make much sense :)

thks!

[TylerJewell]

This is a closed thread - so I think @RVN-BR - you shoudl open a new thread, provide a complete diagnostic and any errors in the OS + in che info, and then someone can respond over time. Engineers will not be monitoring the closed threads.

[roynasser]

Spot on @TylerJewell I handt even realized this was closed off... I'll try to setup a new environment and reproduce the situation soon...Then I'll open an issue in the correct format if I still cant get past the empty workspaces directories. best!

[wernight]

Remember that for me the main issue was that I had to call "che.sh stop" to
populate the storage files. Normally now Che handles SIGTERM but that's not
tested on my side.

Le mer. 19 oct. 2016 à 15:34, Roy notifications@github.com a écrit :

Spot on @TylerJewell https://github.com/TylerJewell I handt even
realized this was closed off... I'll try to setup a new environment and
reproduce the situation soon...Then I'll open an issue in the correct
format if I still cant get past the empty workspaces directories. best!

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#2741 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAI5A8gHlb4M7O77hK7nDybnBWypQpmXks5q1hxvgaJpZM4KRywi
.