CoreOS Provisioner does not properly bootstrap etcd

[sylus]

I have been working with the new CoreOS provisioner which is scheduled for 0.5 or perhaps earlier. The new provisioner does indeed work and launches a CoreOS droplet (testing on digitalocean) successfully. The problem has to do with etcd not being configured as the discovery token was not sent via cloud-init. This means native functionality to CoreOS is broken and etcd is returning 404's. I asked the CoreOS folks about possible workarounds but the general opinion was that these parameters need to be entered at provision time.

As a workaround I though I could create CoreOS via cloud-config myself and then attach the docker-machine generic driver. While CoreOS was working and etcd was functional as soon as I attached docker-machine it broke all etcd functionality. Below are the exact steps I followed.

Create CoreOS Droplet with Cloud Init

Curl

curl --request POST "https://api.digitalocean.com/v2/droplets" \
     --header "Content-Type: application/json" \
     --header "Authorization: Bearer $DO_TOKEN" \
     --data '{
        "region":"'"${DO_REGION}"'",
        "image":"coreos-alpha",
        "size":"'"$DO_SIZE"'",
        "private_networking":true,
        "user_data": "'"$(cat conf/digitalocean/cloud-config.yaml)"'",
        "ssh_keys":["'"$DO_KEY_ID"'"],
        "name":"Master"
     }'

Cloud Config

coreos:
  etcd:
    discovery: https://discovery.etcd.io/<token>
    # addr: $private_ipv4:4001
    addr: $public_ipv4:4001
    # peer-addr: $private_ipv4:7001
    peer-addr: $public_ipv4:7001
    peer-heartbeat-interval: 1000
    peer-election-timeout: 5000
  fleet:
    # public-ip: $private_ipv4   # used for fleetctl ssh command
    public-ip: $public_ipv4
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

After these steps CoreOS was launched and everything including etcd were configured properly. Now I attempted to link docker machine (recent dev version with coreos provisioner) to this existing droplet.

docker-machine create -d generic \
--generic-ssh-user core \
--generic-ssh-key ~/.ssh/xxx \
--generic-ip-address xxx.xxx.x.xx \
example

As soon as docker-machine was done its configuration and linked to the droplet etcd was disabled and its configuration lost. I'd welcome any ideas of how to solve this!

[nathanleclaire]

We're doing some limited initialization of cloud-init (https://github.com/docker/machine/blob/master/libmachine/provision/coreos.go#L19) and it would make sense to me to set some sensible defaults (since e.g. I would expect service discovery on my CoreOS box to "just work"), so you're welcome to open a PR for that provisioner, but we're hesitant to expand the scope of provisioning in general without carefully considering the model we use first.