Added preliminary docker image

[ricardbejarano]

As requested in #300, some official Docker image would be great, so this is a start! ✌️

[spullara]

Have you tried this on Docker for Mac? I copy the fdb.cluster file out of the image and attempt to use it to connect to FDB but it says it is unreachable.

$ netstat -an |grep 4500 | grep LISTEN
tcp6       0      0  ::1.4500               *.*                    LISTEN     
tcp4       0      0  *.4500 `

CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS                                            NAMES
949647ee1f1e        spullara/fdb                             "/bin/sh -c /usr/lib…"   10 minutes ago      Up 10 minutes       0.0.0.0:4500->4500/tcp                           practical_murdock

macbookpro2016:fdb sam$ fdbcli
Using cluster file `fdb.cluster'.

The database is unavailable; type `status' for more information.

Welcome to the fdbcli. For help, type `help'.
fdb> status details

Using cluster file `fdb.cluster'.

Could not communicate with a quorum of coordination servers:
  127.0.0.1:4500  (unreachable)

fdb> 

macbookpro2016:fdb sam$ cat fdb.cluster 
pCupvIOd:yB8STNSN@127.0.0.1:4500

[brk0v]

Listening by default on 0.0.0.0 without authentication is not a good idea from my point of view, and could lead to data leaks.

[ricardbejarano]

True! My idea was to enable developers to frictionlessly use FoundationDB thanks to Docker, but we can have the best of both worlds.
I'll follow up with a commit implementing authentication. Thanks for the feedback!

[spullara]

Does it make sense in the case of Docker to not listen on 0.0.0.0? The only use case for FDB is that something is talking to it right?

[ricardbejarano]

Keep in mind it's the container who's listening on 0.0.0.0, not the host system, so unless you bind the host's 4500/tcp to the container's you are safe.

Also, if you are running this on Docker Swarm or Kubernetes, you are either on the same node (so comms do not go outside the host system) or on multiple nodes already connected over TLS and Docker isolates containers at the network level so not every container can communicate with the FDB container.

[umpc]

Does this address unexpected behavior caused by Debian/Ubuntu packages auto-starting fdbmonitor and configuring a new database?

I left my thoughts in a post in that thread if you have a few minutes and would like a few more ideas to discuss potentially adding to this PR.

[ricardbejarano]

Well, from running it manually it looks like what you are saying is true, installing the server package on Ubuntu does, in fact, start up a fdbmonitor process, which is bad.

• We agree in that we could fix this issue with workarounds like stopping the service, this or temporarily changing the runlevel, but that is hacky, doesn't feel elegant and the .deb packages should, at some point, stop auto-starting a process at all, leave choice to the user.

• About docker-compose, I think a successful container (looking at what other dockerized products do) should implement a nice default setup but enable variability to those who want it. A docker-compose.yaml file a developer would expect to work could be:

services:

  db:
    image: foundationdb:latest
    volumes:
      - ./data:/path/to/data

• fdbmonitor > fdbserver, for the reasons you mention, unless we implement it's tasks manually at build time, which is not a bad idea. I'll leave this open to discussion.

• Regarding cluster files (and foundationdb.conf files, too) , my way would be: leave defaults at build and set permissions properly at startup so that users can mount a volume.

• What you mention about name resolution to fdb.cluster files is something I was pretty disappointed with FDB, and I expect it to be implemented in the future.

Pardon me if I missed out on any of your points and let me know what do you think. I'm open to discuss every aspect of the PR.

[ricardbejarano]

Also, I'm discussing Ubuntu here only, I think we should set down first on what we want from a Docker image, regardless of the underlaying OS, and then port it to Debian, CentOS, and if possible, Alpine.

[umpc]

I've responded on the forums, as the core maintainers prefer that this type of discussion happens there, probably because GitHub's Watch feature can get a bit noisy. If you don't have an account, you can login with GitHub.

[ricardbejarano]

Just for future reference, discussion about this PR is held at the forums, and my specific answer to @umpc's previous comment is available at the same thread, here.

[spullara]

What is the best way to automate the extraction of the fdb.cluster file so you can connect to it? I have been exec'ing cat in the container to grab it.

[umpc]

@spullara Since this only affects new clusters, as far as I am aware, the best practice for extracting the file would be to use something like:

docker cp $(docker ps -alq):/etc/foundationdb/fdb.cluster ~/fdb_workspace/fdb.cluster

where $(docker ps -alq) is the ID of the foundationdb container.

In the forums discussion, I had previously suggested including a sample config file and script in a docker directory in the git repo to generate the fdb.cluster file, though this won’t be distributed with the image which will go to Docker Hub and creates additional overhead in ensuring that the included scripts and sample config are always up to date, so adding documentation to use docker cp seems to be the best solution for everyone.

If you have questions or anything that you would like to add, please respond on the forums thread. I don’t like having to ask this, though I want to respect the community guidelines for contributing and often see core maintainers asking this be done for discussions.

Edit: @bjrnio's solution (below) works well to fix this.

[ricardbejarano]

Was working on this commit as you speak, extracted from the commit itself:

We store a backup of fdb.cluster and foundationdb.conf to /etc/foundationdb.default during build, and copy them to /etc/foundationdb during startup if those files do not exist.
This enables the use of a Docker volume at /etc/foundationdb with or without configuration files, so we can both automate pulling of default files and/or use custom files. This implementation also supports mounting a volume with only one of two files, should we want to use a custom configuration for one and leave default for the other one.

Essentially this enables the use of a Docker volume at /etc/foundationdb with or without config files, so that we can use custom files or just pull the defaults if we want, like @spullara requested.
I did it as elegant as I could, let me know if there is some better way!

[umpc]

@bjrnio Your fix is a good idea and makes things easier for new users, even if it's a bit of a hack. Nice work there, imho.

I submitted a few changes here, and I am waiting on the website to be restored before I can test them.

Edit: Just finished making a few changes.

Once they have been merged, and docker-compose is confirmed to work, this initial PR is ready for a review and to be merged, unless the core maintainers would first like to see a new PR merged including the removal of auto-starting fdbmonitor and auto-configuring the initial cluster, which allows for removing the hack that deletes the auto-configured database.

[ricardbejarano]

I'll try to port this image to these other OSs so that users have the freedom of choice:

• Ubuntu 18.04
• Ubuntu 16.04
• CentOS 7.4
• CentOS 6.9
• Debian 9.4 (not officially supported, but should work)
• Alpine Linux 3.7 (official package required or docker-less building instructions)

A great future improvement would be to craft a n-replicated Compose template for Docker Swarm, with on-demand scalability, enabling progressive updates with docker service update.

I'll leave that as a future improvement, right now I don't know enough about FDB to do it.

Note: I'll give priority to the latest versions of supported OSs. I'll put the limit on Friday, I'll commit whatever OSs are ready by then.

[ricardbejarano]

Official FoundationDB Docker image, draft 1

What images did I add?

• Ubuntu 18.04 and 16.04
• CentOS 6.9
• Debian 9.4

All images have been tested to work.

What did I test?

• The image builds
• A container of this image runs a fdbmonitor process, which spawns a fdbserver process
• Can connect from local fdbcli using the container's fdb.cluster file

Why is there no CentOS 7 image?

systemd, it's always DNS systemd.

The official CentOS Docker image requires too much bending of the standards to get systemd up and running, and that means it's a very complicated process to setup a CentOS 7 image for FoundationDB.

Debian is not officially supported, how is it that there is an image for it?

Though not advertised as supported, the Ubuntu .deb packages are compatible with Debian.

Why did I remove UID and GID ARGs?

It caused issues like this, where UID/GIDs collide:

groupmod: GID '20' already exists

Docker handles volume ownership very well:
On the container:

$ docker exec -it foundationdb-dev ls -Flah /etc/foundationdb
-rw-r--r-- 1 foundationdb foundationdb   33 May 15 08:26 fdb.cluster
-rw-r--r-- 1 foundationdb foundationdb 1.2K May 15 08:26 foundationdb.conf

On the host:

$ ls conf/
-rw-r--r-- 1 caretaker  staff    33B May 15 10:26 fdb.cluster
-rw-r--r-- 1 caretaker  staff   1.2K May 15 10:26 foundationdb.conf

Docker lets containers set ownership to files in volumes at will, but at the host system they always belong to the original owner.

Why did I remove rm -rf /var/lib/foundationdb/data/*?

Somehow, if we remove the contents of /var/lib/foundationdb/data/*, not even the server's own client can connect to itself.

At my level of FoundationDB, I can't say why. My take is that because there is no database, there is not database to connect to.

This means we are back to shipping an empty database with the image, which in of itself is not bad, if a user wants to use it's own database he/she can mount a volume at /var/lib/foundationdb/data with said database, or else just use the empty one.

This also means we no longer need to stop the fdbmonitor service at build, so we no longer need the FoundationDB developers to remove service auto-start after installation.

Why did I rename FBD_DEB_REVISION to FBD_REVISION?

Revision number is not just a .deb affair, RHEL/CentOS packages do have a revision number too.

Why did I modify FDB_REVISION from "1" to "-1"?

Future releases of FoundationDB might not have a revision number, which would break the syntax.

If a release has no revision number, FDB_REVISION="" is the only change to make.

Why did I remove chown -R foundationdb:foundationdb /etc/foundationdb /var/log/foundationdb /var/lib/foundationdb?

fdbmonitor already sets permissions how they are supposed to be.

Why do I install initscripts on the CentOS 6.9 image?

FoundationDB requires the service command to work properly, and the official CentOS 6.9 Docker image does not include such package. service is included in initscripts.

Future improvements

• Request the FoundationDB developers to package a binary for Alpine Linux or to simplify the building process
• Provide instuctions to deploy a scalable cluster with Docker Swarm

[umpc]

Now new users must, one way or another, figure out the UID and GID of fdbmonitor and manually apply permissions to their shared directories on the host OS, which makes the image appear to be broken out of the box.

This is the first log message that every new user will see when using bind mounts or volumes:

Time="1526400592.355860" Severity="40" LogGroup="default" Process="backup_agent.1": ERROR: could not create trace log file `/var/log/foundationdb/trace.127.000.000.001.9.1526400586.X3ogOr.1.xml' (13: Permission denied)

I don't see the point of making it more difficult for users to get started from scratch when the fix for this is simple, so I don't support this change.

This may also inadvertently lead to GitHub issues where the same information exclaiming that the 'Docker image is broken' is repeatedly posted.

---

Why did I remove UID and GID ARGs?

It caused issues like this, where UID/GIDs collide:
groupmod: GID '20' already exists

Please note the usage of --non-unique in usermod. I had used groupmod though did not experience an issue without --non-unique, so did not think to add it there as well.

Adding --non-unique to groupmod resolves this issue. See the manpage for more info.

... Docker lets containers set ownership to files in volumes at will, but at the host system they always belong to the original owner.

Notice how all of the files on the host that you give as an example are world-readable, in this case 644. Unless the UID or GID coincidentally matches, as soon as the foundationdb user in the container attempts to write to those files/directories, the write will fail with a permissions error.

---

Why did I remove rm -rf /var/lib/foundationdb/data/*?

Somehow, if we remove the contents of /var/lib/foundationdb/data/*, not even the server's own client can connect to itself.

The reason is because there is no database. It was removed because it confuses people and causes strange/time-consuming errors by having one automatically set up.

If you set up a database with docker exec foundationdb fdbcli --exec "configure new single ssd" or an equivalent configure statement, it works properly afterwards.

---

Why did I rename FBD_DEB_REVISION to FBD_REVISION?

Revision number is not just a .deb affair, RHEL/CentOS packages do have a revision number too.

It is not a revision to FoundationDB itself, it's a deb revision. That is why it was named as such, to distinguish it for what it is rather than implying that FoundationDB itself possibly changed.

From the deb-version manpage:

This format represents the case where a piece of software was written specifically to be turned into a Debian package, and so there is only one “debianisation” of it and therefore no revision indication is required.

You are right, its essence exists in RPM packages, though it isn't a deb revision in those cases, it's a "release" and the Dockerfile variables should be updated to reflect that.

From an RPM.org guide:

• release is the number of times this version of the software has been packaged.

---

Why did I modify FDB_REVISION from "1" to "-1"?

Future releases of FoundationDB might not have a revision number, which would break the syntax.
If a release has no revision number, FDB_REVISION="" is the only change to make.

If the core team drops the deb revision number in later versions, then someone should remove the dash from the URL when that happens and remove the extraneous variable. Using -1 as a deb revision doesn't make sense as it's not a valid deb revision:

From the deb-version manpage:

... It may contain only alphanumerics and the characters + . ~ (plus, full stop, tilde) ...

---

Why did I remove chown -R foundationdb:foundationdb /etc/foundationdb /var/log/foundationdb /var/lib/foundationdb?

fdbmonitor already sets permissions how they are supposed to be.

This is not a failing of fdbmonitor, it is a requirement from the manpages on usermod and groupmod when changing IDs.

---

Formatting

The above are what is important to have to get working. The following are a few nitpicks, though that I believe are important for maintainers:

&&\ contains two different shell actions:

• Start the following command under the condition: if [ $? -eq 0 ]
• Parse the next line as an argument.

This makes &&\ more difficult for casual readers or maintainers to parse, when compared to reading && \.

CMD cp -rn /etc/foundationdb.default/* /etc/foundationdb &&\

In my opinion, uncommon arguments, in this case the --no-clobber arg (-n) to cp, should not be abbreviated in scripts, because it requires that maintainers or casual readers look up the argument definition to understand the behavior, where as --no-clobber on its own explains its behavior. The behavior of -r only explains itself because it is so frequently used in a variety of situations.

---

I hope you can see why the above issues are important to fix. I am not trying to give you a laundry list of changes to rollback, but to make the Docker image easy for everyone to use and maintain, including for the people who have to respond to users who will have very common questions.

[ricardbejarano]

Pull them over!

Just as a side note, my goal with this image is to lower the barrier of entry for newcomers looking for two things: a single-server deployment for testing and learning FDB, and a n-server scalable cluster with Swarm for (maybe not business-level production, but some level of fault-tolerance) production.

So as long as the changes made do not raise the barrier of entry for newcomers willing to do just that, I'll be happy to push new features! Great job!

[umpc]

Sure. They are here. I figured you would get a notification or else I would have said something originally.

Those are all I am doing for now, though I will make the changes to the other distros tomorrow. Let me know your thoughts.

Anyone reading this is welcome to leave feedback as well. I'm currently trying to make the common use-cases easy without bloat or any extra maintenance.

[brk0v]

docker run -d
-e FDB_UID=$(id -u)
-e FDB_GID=$(id -g)
-v $(pwd)/conf:/etc/foundationdb
-v $(pwd)/logs:/var/log/foundationdb
-v /mnt/drive4500:/var/lib/foundationdb/data/4500
-v /mnt/drive4501:/var/lib/foundationdb/data/4501
-p 127.0.0.1:4500:4500
foundationdb:5.1.7-ubuntu-18.04

Correct me if I'm wrong but these docker containers should not work on MacOS X and Windows, because there is no Kernel Async IO for volumes.

[ricardbejarano]

I've got a macOS environment, I'll make sure that what you are pointing out is tested before pulling @umpc's changes, thanks for the feedback!

[umpc]

@brk0v I have not tested this on ether of those OSes as of yet, though it seems that O_DIRECT is at least not supported on macOS.

Their documentation makes no mention of Docker's specific support for that, which is a bit disappointing.

Edit: I will likely switch to doing this in my dev docker branch, as bind mounts at least on Windows seem to be implemented using Samba unfortunately. This might be the same on macOS but I will need to test to confirm.

[ricardbejarano]

As a side note, I'd like to somehow verify the integrity of the binaries downloaded from www.foundationdb.org before installing them.

We could ask the devs to add an official SHA-256 checksum to the webpage, and sha256sum comes preinstalled on all the supported underlaying OS images.

What do you think?

An alternate solution is to build from source, but after trying to do so when I attempted to build an Alpine Linux image I'd rather not to do it myself. As always, I'm open to suggestions.

[ricardbejarano]

As another side note before calling the PR ready, I'd also like to modify the filesystem to accomodate future versions of FoundationDB, such as:

docker/
  centos/
    6.9/
      5.1.7/
        ...
      5.2.0/
        ...
  debian/
  ubuntu/

or

docker/
  5.1.7/
    centos/
    debian/
      9.4/
        ...
    ubuntu/
  5.2.0/
    centos/
    debian/
    ubuntu/
      16.04/
      18.04/
        ...

[umpc]

I am finished making changes to my PR on @bjrnio's branch, aka my branch here.

As far as differences between my branch and the current commit of this PR, I have added:

• Passing signals from bash to fdbmonitor, which fixes unclean shutdowns.
• Organized startup configuration into entrypoint.sh, similarly to other databases using Docker.
• Added the replacement of 127.0.0.1 with the container IP at runtime in fdb.cluster, when environment variable FDB_MAKE_PUBLIC=-a is supplied, and when a user-supplied fdb.cluster file does not exist, for simpler deployments.
• Added detection for changing file ownership, instead of executing chown on each startup.
• Added a Docker Volume mount for /var/lib/foundationdb, to allow the use of fsync on macOS and Windows.
• Added a quick start tutorial to README.md for simulating a fault-tolerant double redundancy setup with 3 coordinators.
• Various README.md and other grammatical/stylistic changes.

What still needs to be done:

• Testing on Windows. (I only have Windows in a VM, which can't run Docker)

Questions to core developers (or anyone else) before requesting a review:

• Should any changes to the directory hierarchy be made?
• Should any base images be removed? @bjrnio would like to provide a choice, though there are many of them, and they solve the same problem, so I wanted to ask.
• Should any changes be made to image versioning? Current example: foundationdb:5.1.7-ubuntu-18.04
• Should any changes be made to better support pushing to Docker Hub?
• Are there any other questions or desired changes?

I did not add versioning to docker/ in my branch because the official repo does not currently use versioning outside of creating specific branches for each release.

I will bump the forum thread with this comment.

[ricardbejarano]

I'd add to @umpc's questions to core devs:

• Can you add a SHA-256 checksum to each binary in www.foundationdb.org?
• Are there any plans to improve the building process? At the current state, it is neither portable nor standardised, so all the Docker images rely on a magic binary. Using a different building process
  could not only enable us to use the actual source during image build time, but would also give us a much wider gamut of underlaying OSs for the Docker images, such as Alpine or Amazonlinux.

[spullara]

I just tried the 16.04 version and fdbserver isn't coming up within the container. Not seeing any obvious reason why it isn't. Running it on Docker for Mac. Some sample events from the trace log:

<Event Severity="10" Time="1527883698.254990" Type="MachineMetrics" Machine="127.0.0.1:20" ID="0000000000000000" Elapsed="5.0001" MbpsSent="0.00159997" MbpsReceived="0.00159997" OutSegs="20" RetransSegs="0" CPUSeconds="0.0100404" TotalMemory="4139307008" CommittedMemory="487510016" AvailableMemory="3651796992" ZoneID="[not set]" MachineID="[not set]" logGroup="default" TrackLatestType="Original"/>
<Event Severity="10" Time="1527883698.255622" Type="TransactionMetrics" Machine="127.0.0.1:20" ID="0000000000000000" ReadVersions="2" CommitStarted="1" CommitCompleted="0" PastVersions="0" FutureVersions="0" NotCommitted="0" MaybeCommitted="0" MeanLatency="0" MedianLatency="0" Latency90="0" Latency98="0" MaxLatency="0" MeanRowReadLatency="0" MedianRowReadLatency="0" MaxRowReadLatency="0" MeanGRVLatency="0" MedianGRVLatency="0" MaxGRVLatency="0" MeanCommitLatency="0" MedianCommitLatency="0" MaxCommitLatency="0" MeanMutationsPerCommit="0" MedianMutationsPerCommit="0" MaxMutationsPerCommit="0" MeanBytesPerCommit="0" MedianBytesPerCommit="0" MaxBytesPerCommit="0" logGroup="default"/>
<Event Severity="10" Time="1527883698.368315" Type="ConnectingTo" Machine="127.0.0.1:20" ID="0000000000000000" PeerAddr="127.0.0.1:4500" SuppressedEventCount="2" logGroup="default"/>
<Event Severity="20" Time="1527883698.368315" Type="N2_ConnectError" Machine="127.0.0.1:20" ID="9b752b6b90d5e74f" Message="111" SuppressedEventCount="2" logGroup="default"/>
<Event Severity="10" Time="1527883698.368315" Type="ConnectionClosed" Machine="127.0.0.1:20" ID="0000000000000000" PeerAddr="127.0.0.1:4500" Error="connection_failed" ErrorDescription="Network connection failed" ErrorCode="1026" SuppressedEventCount="2" logGroup="default"/>
<Event Severity="10" Time="1527883699.376514" Type="ConnectingTo" Machine="127.0.0.1:20" ID="0000000000000000" PeerAddr="127.0.0.1:4500" SuppressedEventCount="1" logGroup="default"/>
<Event Severity="20" Time="1527883699.376514" Type="N2_ConnectError" Machine="127.0.0.1:20" ID="a1c509fefa953e33" Message="111" SuppressedEventCount="1" logGroup="default"/>
<Event Severity="10" Time="1527883699.376514" Type="ConnectionClosed" Machine="127.0.0.1:20" ID="0000000000000000" PeerAddr="127.0.0.1:4500" Error="connection_failed" ErrorDescription="Network connection failed" ErrorCode="1026" SuppressedEventCount="1" logGroup="default"/>
<Event Severity="10" Time="1527883700.400121" Type="ConnectingTo" Machine="127.0.0.1:20" ID="0000000000000000" PeerAddr="127.0.0.1:4500" SuppressedEventCount="1" logGroup="default"/>
<Event Severity="20" Time="1527883700.400121" Type="N2_ConnectError" Machine="127.0.0.1:20" ID="7381099fffef5baa" Message="111" SuppressedEventCount="1" logGroup="default"/>
<Event Severity="10" Time="1527883700.400121" Type="ConnectionClosed" Machine="127.0.0.1:20" ID="0000000000000000" PeerAddr="127.0.0.1:4500" Error="connection_failed" ErrorDescription="Network connection failed" ErrorCode="1026" SuppressedEventCount="1" logGroup="default"/>
<Event Severity="10" Time="1527883700.509411" Type="SomewhatSlowRunLoopTop" Machine="127.0.0.1:20" ID="0000000000000000" Elapsed="0.108965" logGroup="default"/>

[umpc]

@spullara My branch should work.

[spullara]

@umpc just tried the Dockerfile in your branch and it has a similar issue. fdbserver isn't starting up for me.

macbook-pro:fdb sam$ docker run -d   -v $(pwd)/conf:/etc/foundationdb   -v $(pwd)/logs:/var/log/foundationdb   -v $(pwd)/data:/var/lib/foundationdb/data   -p 127.0.0.1:4500:4500  foundationdb:ubuntu-16.04
6779261086d9ecb1d4d006cc4a54e925e34ef4f9c566ac6415f7b25928875466
macbook-pro:fdb sam$ docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                      NAMES
6779261086d9        foundationdb:ubuntu-16.04   "/usr/local/bin/entr…"   2 seconds ago       Up 1 second         127.0.0.1:4500->4500/tcp   focused_aryabhata
macbook-pro:fdb sam$ docker exec -it 6779261086d9 /bin/bash
root@6779261086d9:/# ps axuxwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.3  0.0  18040  2860 ?        Ss   16:30   0:00 bash /usr/local/bin/entrypoint.sh /usr/lib/foundationdb/fdbmonitor
root        31  0.0  0.0  20688  2772 ?        S    16:30   0:00 /usr/lib/foundationdb/fdbmonitor
foundat+    32  0.3  0.2 111836  8108 ?        Sl   16:30   0:00 /usr/lib/foundationdb/backup_agent/backup_agent --cluster_file /etc/foundationdb/fdb.cluster --logdir /var/log/foundationdb
root        41  0.0  0.0  20688   240 ?        S    16:30   0:00 /usr/lib/foundationdb/fdbmonitor
root        42  1.0  0.0  18248  3252 pts/0    Ss   16:30   0:00 /bin/bash
root        52  0.0  0.0  34424  2832 pts/0    R+   16:30   0:00 ps axuxwww
root@6779261086d9:/# exit
macbook-pro:fdb sam$ fdbcli -C conf/fdb.cluster 
Using cluster file `conf/fdb.cluster'.

The database is unavailable; type `status' for more information.

Welcome to the fdbcli. For help, type `help'.
fdb> 

macbook-pro:fdb sam$ 

[umpc]

@spullara Check out the README, as the usage has changed slightly because of how Docker implements mounts on OSes other than Linux distros. In some cases, fsync or an equivalent syscall was not available, leading to unavailability, so that needed to be changed.

Additionally, when using bind mounts and not setting FDB_UID and FDB_GID to appropriate values for your setup, ensure that both UIDs and GIDs of 1000 exist on the host and have read/write permissions set for the shared mount directories on the host.

In this case, I think the -v $(pwd)/data:/var/lib/foundationdb/data argument is causing the bind mount that it specifies to overlay the volume that is specified in the Dockerfile.

If not that, then perhaps mismatched read/write permissions between the container and host on shared bind mount directories.

[ricardbejarano]

IMO, the steps to follow are just too much, raising the barrier of entry to newcomers, and that's exactly why @spullara is having issues.

In @umpc's defence, this is a FoundationDB issue, it's design does not enable the container way.
From the outrageous build process, to how it scales or how instances communicate with each other.

All other databases I've dealt with do containers very well, I don't know why FDB doesn't.

Anyhow, I'll resume my work on this image in a couple weeks and try to Dockerize FoundationDB, but I'm not as excited about it as I was in the beginning.

[umpc]

I am sure that everyone would like better UX.

All other databases I've dealt with do containers very well, I don't know why FDB doesn't.

Many databases do not operate in a distributed fashion, leading to the necessity of the fdb.cluster file and its usability issues. It is possible to use a different method, like doing what etcd does, though that seems to be even less user-friendly in my opinion. This is a difficult problem and all distributed databases solve it in a few ways.

For the other solved issues, postgresql also deals with many of the same problems, such as the fsync issue with bind mounts on Windows and possibly macOS hosts.

As far as bind mount permissions, postgresql solves those in a slightly different way here and here, though unless the user passes --user, I believe it would still run as root. The way that this is solved in my branch is by using FDB_UID and FDB_GID environment variables, which are more explicit in that they accept IDs, allow for using a custom group separate from the user, and in any case, the server does not run as root unless manually specified, except for running as an Administrator on Windows hosts because of how Docker handles permissions.

Just know that while some of the design decisions may seem "outrageous," there are good reasons for many of them. I prefer the core developers work on issues such as writes running proxy servers out of memory or memory leaks, which can impact live production services, rather than retooling.

[ricardbejarano]

Sure! I'm not saying retooling is a priority, but the fact that there is a considerable amount of technical debt makes me question the designers' choices.

[brownleej]

• This is an awesome start. I think that our docker support will continue to improve as we go on, so we don’t need to solve everything in this PR.
• I see a lot of changes to file permissions that seem like they’re not related to the intent of the PR. Could you clean those up?
• The instructions in the README in the PR didn’t work for me on my Mac, but the one @umpc linked did. It looks like there are different volume instructions to support the Mac volume issues folks were discussing above. Can we get those clarified?
• I think that having a docker/5.2.0/ubuntu/16.04 directory structure makes sense
• I don’t have any experience with Docker Hub myself, but if there are changes that would make pushing to Docker Hub easier we can handle them here, or in subsequent PRs
• Yes, we can add SHA-256 checksums to www.foundationdb.org
• I’m not the most familiar with the build process; @alexmiller-apple may be able to shed more light on how we look at those portability concerns

[umpc]

The permission changes are necessary because the user and group are changed to match the host. Removing that means that directories will not be writable in certain cases.

Off the top of my head, I am thinking that it affects the bind-mounted directories when mounts are unspecified, when bind-mounted directories do not exist, or when they otherwise had incorrect permissions.

I am referring to my branch, which fixes the known issues with this one. I figured that my changes would be merged by now into the repo for this PR, though if it would be better to open a new PR, I could do that.

[alexmiller-apple]

If I've skimmed the thread right, I think the build system complaint was that since the supported build environment is an ubuntu image, making non-ubuntu images is hard. I think there might be a few workable directions around this.

1. The FDB team should provide downloads of the binaries not in any distro package. This way you could just wget an fdbserver, and we should indeed provide a hash for them as well.
2. From-source builds for non-ubuntu base images could possibly be done via multi-stage docker builds? It looks pretty reasonable to be able to do a build in ubuntu as step 0, and then have step 1 be an alpine linux image that you copy fdbserver and other needed artifacts into.
3. I think the same multi stage build could be used to install the fdb server .deb into an Ubuntu image, and then cherry-pick out the files you need during stage 1.

As an end result, I think doing (1) by default and providing an option for (2) if a developer wishes to make a change and then build it into a container would be great. Doing (1) by default would be preferred, as all of the QA the FDB team does is done on the released binary. Since (1) isn't available right now, (3) would work as an immediate solution, but isn't pretty. :/

[alexmiller-apple]

The permission changes are necessary because the user and group are changed to match the host. Removing that means that directories will not be writable in certain cases.

As part of the PR right now there's permission modifications like

fdbclient/json_spirit/json_spirit_writer_template.h 100755 → 100644

which is I believe is what @brownleej was asking about, but maybe not what you (@umpc) were replying about?

[umpc]

Okay, I thought he was referring to this in my branch or this from this PR's branch.

[scottlogic-alex]

What still needs to be done:

• Testing on Windows. (I only have Windows in a VM, which can't run Docker)

I did some rudimentary testing of umpc's docker branch on Windows using:

• Docker for Windows (Linux containers)
• MSYS2 (albeit using Git for Windows' modified MSYS2 runtime)
  • MinTTY
  • winpty
  • MinGW64

I followed the instructions in: https://github.com/umpc/foundationdb/blob/1a710525ccf11394eb5a093c542bcb21ca4d506a/docker/README.md

I encountered two gotchas. Neither related to FoundationDB.

Beware CRLF in your working copy of the git repo

I fixed this by doing git clone ... --config core.autocrlf=input to ensure that I grabbed source code using UNIX line-endings.

This is only relevant for people trying to build their own Docker image from source. Pulling from Docker Hub would be fine.

Beware quirks of file mounting on Docker for Windows

I had to preface my bind-mount src paths with C:/ to make them absolute. $(pwd) (which for me expands to /fdb) was not considered absolute enough by the Docker daemon.

I had to preface my mount dst paths with // to avoid Docker's expanding to a Windows-style filepath inside the container.

docker run -i --rm \
-e FDB_UID=$(id -u) \
-e FDB_GID=$(id -g) \
-e FDB_MAKE_PUBLIC=-a \
--mount type=volume,src=fdb4500-data,dst=//var/lib/foundationdb \
--mount type=bind,src=C:/fdb/fdb4500/etc,dst=//etc/foundationdb \
--mount type=bind,src=C:/fdb/fdb4500/log,dst=//var/log/foundationdb \
--net fdb \
--name fdb4500 \
foundationdb:5.1.7-ubuntu-18.04

====

Containers joined the cluster just fine over the Docker network:

# note: I use winpty so that I can enjoy a TTY via -t switch
winpty docker exec -it fdb4500 fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb> status

Using cluster file `/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode        - single
  Storage engine         - memory
  Coordinators           - 1

Cluster:
  FoundationDB processes - 2
  Machines               - 2
  Memory availability    - 1.8 GB per process on machine with least available
                           >>>>> (WARNING: 4.0 GB recommended) <<<<<
  Fault Tolerance        - 0 machines
  Server time            - 07/02/18 14:03:45

Data:
  Replication health     - Healthy
  Moving data            - 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used        - 1 MB

Operating space:
  Storage server         - 58.4 GB free on most full server
  Log server             - 58.4 GB free on most full server

Workload:
  Read rate              - 15 Hz
  Write rate             - 0 Hz
  Transactions started   - 4 Hz
  Transactions committed - 0 Hz
  Conflict rate          - 0 Hz

Backup and DR:
  Running backups        - 0
  Running DRs            - 0

Client time: 07/02/18 14:03:45

Write & read worked:

fdb> writemode on
fdb> set test 123
Committed (1357245611)
fdb> get test
`test' is `123'

Replication worked (see that I can read the key from a different node):

docker exec fdb4501 fdbcli --exec "get test"
`test' is `123'

I don't have a real application to stress-test this with, but vitals seem okay.

[ofek]

What's the status of this? Many of us would really appreciate even alpha-quality official images for testing PoCs 😄

[brownleej]

I've opened another PR on this (#887) to kickstart another round of discussion and get something basically functional merged in.