HTTPS via letsencrypt #103

alexlenail · 2016-08-15T19:26:35Z

Just tracking changes I’ve made through the docker-galaxy-stable repo
to add HTTPS. Opening a PR purely for visualization purposes right now.

alexlenail · 2016-08-18T15:55:38Z

@bgruening @martenson @jmchilton
In this PR, I currently use a shell script. That shell script comes from here.

Two questions:

This project is called ansible-galaxy-extras, but I don't really plan on using ansible in this PR. Does that mean it belongs somewhere else? If this repo was renamed "galaxy-extras" I think this PR would fit. I'm not asking for that, I just want to know what you think of this.
The shell script is another repository -- should I think about using that repository as a submodule? I don't think it makes sense, but I'd like to hear what you think.

mvdbeek · 2016-08-18T15:58:28Z

templates/nginx.conf.j2

@@ -34,13 +34,21 @@ http {

    server {
        listen 80 default_server;
-        server_name  localhost;
+        server_name  {{ galaxy_hostname }};  ## default == localhost => fing out where to put this


that would be in defaults/main.yml

mvdbeek · 2016-08-18T16:17:23Z

This project is called ansible-galaxy-extras, but I don't really plan on using ansible in this PR. Does that mean it belongs somewhere else? If this repo was renamed "galaxy-extras" I think this PR would fit. I'm not asking for that, I just want to know what you think of this.

I think that's OK, we have other scripts in here as well.

The shell script is another repository -- should I think about using that repository as a submodule? I don't think it makes sense, but I'd like to hear what you think.

Well, it's a bit of overhead, that would be OK for me, but if you're not modifying the script you could also download it when needed, right?

Have you tried this role https://github.com/thefinn93/ansible-letsencrypt? It looks pretty decent.

alexlenail · 2016-08-18T16:24:19Z

@mvdbeek see thefinn93/ansible-letsencrypt#38

We have unique constraints with this https/letsencrypt problem, because since I'm using @bgruening's docker container, it can't be a regular ansible-role because Bjoern calls the roles at docker build time but letsencrypt certificates are unique to docker runers -- so whatever I end up using needs to be called from startup.sh. I tried to figure out what the best solution to this would be here: http://stackoverflow.com/questions/38978792/what-is-the-easiest-most-correct-way-to-set-up-letsencrypt-programmatically but no answers as you can see.

If you go back through the commit history here a little, you see that I was originally using ansible calling ansible-playbook from startup.sh. At a certain point it doesn't really need to be ansible -- right?

mvdbeek · 2016-08-18T16:28:46Z

regular ansible-role because Bjoern calls the roles at docker build time but letsencrypt certificates are unique to docker runers -- so whatever I end up using needs to be called from startup.sh

you could launch ansible at runtime though, that's what we're doing here:
https://github.com/ARTbio/ansible-artimed/blob/master/startup.sh
Anyway, you would need also need to dynamically update the server name in the nginx config, right?

If you go back through the commit history here a little, you see that I was originally using ansible calling ansible-playbook from startup.sh. At a certain point it doesn't really need to be ansible -- right?

ah yes ... need to read more carefully. Yes, it all has pro's and cons, I agree.

mvdbeek · 2016-08-18T16:49:04Z

templates/nginx.conf.j2


 {% if nginx_use_passwords %}
        auth_basic      "devbox";
        auth_basic_user_file  /etc/nginx/htpasswd;
 {% endif %}

+{% if galaxy_extras_config_letsencrypt %}


I think that would be generally useful, maybe call the variable galaxy_extras_config_ssl ?

I'm sorry -- you want me to change the variable name, yes? Why? Because someone might use something else than letsencrypt? The only small issue with this is that right now, galaxy_extras_config_letsencrypt is the only flag you need to set to get https via letsencrypt -- splitting it into two flags would be kind of annoying...

Because someone might use something else than letsencrypt?

exactly, see https://github.com/galaxyproject/ansible-galaxy-extras/pull/47/files

galaxy_extras_config_letsencrypt is the only flag you need to set to get https via letsencrypt -- splitting it into two flags would be kind of annoying...

so what you can do is {% if galaxy_extras_config_letsencrypt or galaxy_extras_config_ssl %} or

in the defaults.yml set galaxy_extras_config_ssl: "{{ galaxy_extras_config_letsencrypt }}".

Very clever!

mvdbeek · 2016-08-18T20:00:46Z

tasks/nginx.yml

@@ -13,3 +13,12 @@
    - { src: 'nginx_uwsgi.conf.j2',       dest: '{{ nginx_conf_directory }}/uwsgi.conf' }
    - { src: 'nginx.conf.j2',             dest: '{{ nginx_conf_path }}' }
  notify: Restart Nginx
+
+
+- name: Configure HTTPS via Letsencrypt


Can we change the name to 'Copy nginx acme location include' ?

Do you think I can fold this into the above task? Or would making it conditional be difficult?

alexlenail · 2016-08-18T20:22:02Z

Thanks for the close review @mvdbeek! To everyone else: this is currently in testing -- hopefully soon will work and we can merge!

mvdbeek · 2016-08-18T21:00:40Z

What's your plan for providing the user details? I guess email and domain name have to go somewhere ... maybe as variables in letsencrypt.conf.j2 ?

alexlenail · 2016-08-18T21:30:41Z

@mvdbeek what do you mean by providing the user details? I don't think we need email, but we do need domain name -- which needs to be passed in as an --extra-vars right now. I just spoke with Bjoern about how to make this better.

mvdbeek · 2016-08-18T21:39:15Z

I've never used letsencrypt, I was assuming you would need some kind of account.
Yeah, extra-vars is fine (you can just set them to "").

alexlenail · 2016-08-18T21:45:28Z

You mean setting a variable in ansible to "" is enough to suggest that it
should be set with an extra var?

I couldn't figure out how to do this earlier so I asked:
http://stackoverflow.com/questions/38959834/ansible-is-there-a-way-to-have-variables-without-defaults

On Thu, Aug 18, 2016 at 5:39 PM, Marius van den Beek <
notifications@github.com> wrote:

I've never used letsencrypt, I was assuming you would need some kind of
account.
Yeah, extra-vars is fine (you can just set them to "").

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#103 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACojfUDMUmb4aSlfTZzpw1ErbbrvMWe2ks5qhNEEgaJpZM4JksEX
.

Alex Lenail
Tufts University '16

mvdbeek · 2016-08-18T21:53:58Z

You mean setting a variable in ansible to "" is enough to suggest that it
should be set with an extra var?

No, but usually we document all variables in defaults/main.yml, so that people are aware of all variables. So if there is no good default, "" is pretty OK, if you precede it with sth. like "#Set this variable to ... to control ..." .

Anyway, role default variables have the lowest precedence of all variables, so whatever you pass in as --extra-vars or inventory variables or ... will be used.

And that's also nicer for other projects that use this role, where you do know in advance the hostname.

mvdbeek · 2016-08-23T20:08:55Z

templates/letsencrypt_refresh.sh.j2

+	supervisorctl restart nginx:
+fi
+
+/bin/bash letsencrypt.sh --cron -d ${GALAXY_CONFIG_DOMAIN} --config letsencrypt.conf


Does this need cron? I don't think this is available in the docker container

@mvdbeek that command does not require cron, it's meant to be executed from cron. See here:
https://github.com/galaxyproject/ansible-galaxy-extras/pull/103/files/f9d01e2ae4cd57606a0e7b2f6c87a6d4a6095c9d#diff-205c98235541487412061d12953dc4c0R94

alexlenail · 2016-08-25T17:38:27Z

@mvdbeek @bgruening This now works locally and remotely. However, it still uses ansible, which is a problem for people like me using Bjoern's image. So I'm going to try to refactor it to move much of the nginx.yml modifications into startup.sh. I need to ponder this briefly, but I'm hoping to merge this tonight or tomorrow before Bjoern goes on vacation!

alexlenail · 2016-08-26T04:48:15Z

@mvdbeek @bgruening That is now fixed. This is ready for merge.
Note: Please squash this PR into a single commit.

mvdbeek · 2016-08-26T09:24:07Z

tasks/nginx.yml

+  when: galaxy_extras_config_letsencrypt
+
+- name: get the letsencrypt script from github repository
+  get_url:


mvdbeek · 2016-08-26T09:27:51Z

This looks very good, I will do some testing and change the supervisor template and then merge this over the weekend. It looks really nice! Thanks a lot @zfrenchee

bgruening · 2016-08-26T10:37:03Z

@zfrenchee thanks for integrating all my comments and annoying requests - this is really looking great now. I fear that I can only get this in the Docker image when I'm back, in 1-2 weeks. As I already told you this should not put you back, just use your local own build Galaxy image in the meantime.

Moreover, for the Docker integration it would be nice if you can reactivate the self-signed certificate, we need this in the Docker-travis file to test the setup.

I think the following steps needs to be done:

fix the comments from @mvdbeek
get this PR merged
create a PR against galaxy-docker-stable with:
- updated ansible playbook (git submodule foreach git pull origin master)
- two travis tests if possible, one with the self-signed cert and one with fetching a cert from letsencrypt
- it should turn green :)

Thanks again @zfrenchee!

mvdbeek · 2016-08-26T10:43:55Z

two travis tests if possible, one with the self-signed cert and one with fetching a cert from letsencrypt

I don't think you will have incoming connections on travis, so I guess that won't work.
Unless you can simulate this somehow.

alexlenail · 2016-08-26T14:49:09Z

@bgruening As @mvdbeek mentions, I still don't think we can test this in travis. I think I could add back a self-signed certificate and we could test that with travis, but I think that would make this PR more complicated and it wouldn't actually be testing what we want to test.

So it sounds like the only modification I need to make is wrap the cron task in supervisor template in an if block.

alexlenail · 2016-08-26T15:41:51Z

@mvdbeek I cleaned it up a little, including wrapping cron in an if block (please check to make sure I did it right). Looking forward to the merge this weekend if all goes smoothly.

It seems there are some merge conflicts. Would you like to take care of those or should I try?

bgruening · 2016-08-26T15:57:35Z

docs/https.md

+variable called `GALAXY_CONFIG_DOMAIN` which is set to the domain name you would like to certify.
+
+If you're using [Docker-Galaxy-Stable](https://github.com/bgruening/docker-galaxy-stable/) or some
+derivative image, you can specify `GALAXY_CONFIG_DOMAIN` at runtime by running the image


Can you please rename it. GALAXY_CONFIG is a preserved namespace for galaxy.ini changes. Maybe just GALAXY_DOMAIN? You want to have the FQDN (Fully Qualified Domain Name), this could probably also go into the readme.

Oh, sorry! I did not realize GALAXY_CONFIG_* was a reserved namespace! I was hoping to change the name anyways! (It doesn't need to be a fqdn, e.g. for me "oom.io" was all I needed)

bgruening · 2016-09-12T12:44:08Z

templates/nginx.conf.j2

@@ -31,10 +31,17 @@ http {
       server localhost:8080;
    }
 {% endif %}
+    server {
+        listen 80;
+{% if galaxy_extras_config_ssl %}


indentation

We need to cast this to bool isn't it? galaxy_extras_config_ssl|bool

IIRC this is only needed when combined with logic operators in tasks.
Otherwise the tests wouldn't pass.

indentation

I don't see it, what exactly should be indented?

I'm not sure when it is really needed, but it is also mentioned here in when without a logic operator. http://docs.ansible.com/ansible/playbooks_filters.html

- debug: msg=test when: some_string_value | bool

I wouldn't bet my life on it, but I think this is only needed in certain ansible tasks, not for jinja2 templates.
After all, the tests that test port 80 would be redirected to port 443, which would fail. Also we have other if statements in this file that are not using the boolean filter.

I think we should use this only when necessary.

I would err on the side of caution and just add the cast if you're not 100% sure, or, alternatively (more work), test that the false case works as intended without the cast.

I would err on the side of caution and just add the cast if you're not 100% sure, or, alternatively (more work), test that the false case works as intended without the cast.

I am 100% sure, because the tests take the default, which is False, and test that access to galaxy on port 80 works, without exposing port 443. There is no way that in this instance here it is necessary. I am just not sure when exactly it is needed.

bgruening · 2016-09-12T12:58:44Z

Looks great, thanks especially for integrating the self-signed test again!

alexlenail · 2016-09-13T01:16:55Z

@mvdbeek @bgruening what's the latest on this?

mvdbeek · 2016-09-13T06:42:13Z

ready to go from my side

mvdbeek · 2016-09-13T11:17:37Z

rebased to resolve merge conflict.

keys in startup.sh. Remove cron handler (needs to not have autostart enabled, start in startup.sh script). Tag relevant part with https so only subset of tasks runs.

…y-tools

$GALAXY_CONFIG_GALAXY_INFRASTRUCTURE_URL.

mvdbeek · 2016-09-13T12:06:30Z

OK, last test for the boolean issue is here. I changed false to no, just to be sure that this works as well.

bgruening · 2016-09-13T12:12:51Z

Great work @zfrenchee and @mvdbeek! Thanks a lot!

mvdbeek reviewed Aug 18, 2016
View reviewed changes

alexlenail changed the title ~~[WIP] https via lets encrypt~~ [WIP] https via letsencrypt Aug 18, 2016

mvdbeek reviewed Aug 18, 2016
View reviewed changes

mvdbeek reviewed Aug 23, 2016
View reviewed changes

alexlenail changed the title ~~[WIP] https via letsencrypt~~ HTTPS via letsencrypt Aug 25, 2016

mvdbeek reviewed Aug 26, 2016
View reviewed changes

tasks/nginx.yml

when: galaxy_extras_config_letsencrypt

- name: get the letsencrypt script from github repository

get_url:

Copy link

Member

mvdbeek Aug 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

bgruening reviewed Aug 26, 2016
View reviewed changes

bgruening reviewed Sep 12, 2016
View reviewed changes

Alexander Lenail and others added 17 commits September 13, 2016 13:23

HTTPS via lets encrypt

22a4525

Control cron with galaxy_extras_config_cron

0f94c7d

Do not produce empty letsencrypt.conf file

c3d4833

Fix server sections

4f3d242

Separate HTTPS/SSL from letsencrypt

b2f9497

Split ssl into different methods

53d593b

Add possibility to copy own keys

d67d0de

Fire letsencrypt from role

e62b13e

Setup self-signed keys, own keys and letsencrypt

93731ac

keys in startup.sh. Remove cron handler (needs to not have autostart enabled, start in startup.sh script). Tag relevant part with https so only subset of tasks runs.

Change crond to cron, autostart false

7230361

Fix path to playbook and use -a operator

f6ba3a8

Ignore errors and have tasks in main handler file

27e1ae7

Update docs and remove line-endings

ab85082

Go with 'restart nginx', as we have 'restart galaxy' in ansible-galax…

3f955c7

…y-tools

Use CN=selfsigned for self-signed certs

c018e5a

Revert slurm changes

bdcd6ba

Change $GALAXY_DOMAIN to

7999758

$GALAXY_CONFIG_GALAXY_INFRASTRUCTURE_URL.

bgruening merged commit a65f0e3 into galaxyproject:master Sep 13, 2016

This was referenced Sep 13, 2016

HTTPS support bgruening/docker-galaxy-stable#240

Merged

SSL for nginx, safer supervisord task, VIRTUAL_ENV + other env vars for Galaxy handler #47

Closed

alexlenail deleted the https branch September 13, 2016 15:24

HTTPS via letsencrypt #103

HTTPS via letsencrypt #103

Conversation

alexlenail commented Aug 15, 2016 • edited Loading

alexlenail commented Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

mvdbeek commented Aug 18, 2016

alexlenail commented Aug 18, 2016 • edited Loading

mvdbeek commented Aug 18, 2016 • edited Loading

mvdbeek Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexlenail commented Aug 18, 2016

mvdbeek commented Aug 18, 2016

alexlenail commented Aug 18, 2016

mvdbeek commented Aug 18, 2016

alexlenail commented Aug 18, 2016

mvdbeek commented Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexlenail commented Aug 25, 2016

alexlenail commented Aug 26, 2016

Choose a reason for hiding this comment

mvdbeek commented Aug 26, 2016

bgruening commented Aug 26, 2016

mvdbeek commented Aug 26, 2016 • edited Loading

alexlenail commented Aug 26, 2016

alexlenail commented Aug 26, 2016

Choose a reason for hiding this comment

alexlenail Aug 26, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek Sep 12, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvdbeek Sep 13, 2016 • edited Loading

Choose a reason for hiding this comment

bgruening commented Sep 12, 2016

alexlenail commented Sep 13, 2016

mvdbeek commented Sep 13, 2016

mvdbeek commented Sep 13, 2016

mvdbeek commented Sep 13, 2016

bgruening commented Sep 13, 2016 • edited Loading

alexlenail commented Aug 15, 2016 •

edited

Loading

alexlenail commented Aug 18, 2016 •

edited

Loading

alexlenail commented Aug 18, 2016 •

edited

Loading

mvdbeek commented Aug 18, 2016 •

edited

Loading

mvdbeek Aug 18, 2016 •

edited

Loading

mvdbeek commented Aug 18, 2016 •

edited

Loading

mvdbeek commented Aug 26, 2016 •

edited

Loading

alexlenail Aug 26, 2016 •

edited

Loading

mvdbeek Sep 12, 2016 •

edited

Loading

mvdbeek Sep 13, 2016 •

edited

Loading

bgruening commented Sep 13, 2016 •

edited

Loading