Consolidate puppet manifests into 'site.pp' (vagrant)

[jeff1evesque]

We need to consolidate our puppet puppet/environment/vagrant/manifests/*.pp, into a single site.pp. This will decrease the syntax in our Vagrantfile, while also increasing performance, since the transitioning between one manifest, to another (defined within the Vagrantfile), is a noticeable cost performance.

The transition between each puppet instance, can add 1-3+ seconds to the overall vagrant up build:

...
==> default: Notice: Applied catalog in 35.77 seconds
==> default: Running provisioner: puppet...
==> default: Running Puppet with environment vagrant...
==> default: stdin: is not a tty
==> default: Warning: /tmp/vagrant-puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It shold be converted to version 5
==> default:    (in /tmp/vagrant-puppet/hiera.yaml)
==> default: Warning: Unknown variable: 'python::params::provider'. at /tmp/vagrant-puppet/modules-3c0008ae9953309c24252b2dd2bf5cd/python/manifests/init.pp:56:25
==> default: Warning: Unknown variable: 'python::params::source'. at /tmp/vagrant-puppet/modules-3c00084a9953309c24252b2dd2bf5cd/python/manifests/init.pp:61:25
==> default: Notice: Compiled catalog for ubuntu-14.04 in environment vagrant in 1.58 seconds
...

The initial motivation to segregate manifests, is not logical, since each puppet directive has a corresponding standard output in the console, during a vagrant up build, regardless of whether we trigger a single site.pp, or several puppet manifests. But, it is clear the performance benefit of the former.

Note: this issue is motivated by #2924, and the corresponding mongodb_cluster puppet module wrapper being developed. Specifically, it seemed a little odd to add init.pp to the overall Vagrantfile.

[jeff1evesque]

Currently a fresh vagrant up does not render our application within a browser. Only after restarting our application via vagrant halt && vagrant up, are we able to see the application. Common symptoms for this could be related to inconsistent order of our puppet logic. For example, if we properly edit one of our jsx scripts, and save it, the changes would not take effect until successive reboot. Therefore, we may need to investigate along these lines.

[jeff1evesque]

Based on our traceback, it seems our gunicorn processes are starting before our webcompilers execute their initial compilation, which is consistent with our latter comment:

...
==> default: Notice: /Stage[main]/Python::Flask/Package[Flask]/ensure: created
==> default: Notice: /Stage[main]/Webserver::Service/File[/etc/init/start_gunicorn.conf]/ensure: defined content as '{md5}95d861b3c916da7f8e76427279e608af'
==> default: Notice: /Stage[main]/Package::Gunicorn/Package[gunicorn]/ensure: created
==> default: Notice: /Stage[main]/Webserver::Start/Service[start_gunicorn]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/ensure: defined content as '{md5}0962d70c4ec78bbfa6f3544ae0c41974'
==> default: Notice: /Stage[main]/Package::Jsonschema/Package[jsonschema]/ensure: created
==> default: Notice: /Stage[main]/Package::Pyyaml/Package[pyyaml]/ensure: created
==> default: Notice: /Stage[main]/Package::Flask_script/Package[Flask-Script]/ensure: created
==> default: Notice: /Stage[main]/Package::Pytest_flask/Package[pytest-flask]/ensure: created
==> default: Notice: /Stage[main]/Package::Libssl_dev/Package[libssl-dev=1.0.1f-1ubuntu2.22]/ensure: created
==> default: Notice: /Stage[main]/Package::Python_dev/Package[python-dev=2.7.5-5ubuntu3]/ensure: created
==> default: Notice: /Stage[main]/Package::Scrypt/Package[scrypt]/ensure: created
==> default: Notice: /Stage[main]/Database::Database/Exec[create-database-tables]/returns: executed successfully
==> default: Notice: /Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Key[Add key: 9FD3B784BC1C6FC31A8A0A1C1655A0AB68576280 from Apt::Source nodesource]/Apt_key[Add key: 9FD3B784BC1C6FC31A8A0A1C1655A0AB68576280 from Apt::Source nodesource]/ensure: created
==> default: Notice: /Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Setting[list-nodesource]/File[/etc/apt/sources.list.d/nodesource.list]/ensure: defined content as '{md5}5102196d530f84b9443d282b9d8cc8d7'
==> default: Notice: /Stage[main]/Nodejs::Install/File[root_npmrc]/ensure: defined content as '{md5}43cd7c9106474b6fe4b5fc27601d01d0'
==> default: Notice: /Stage[main]/Nginx::Package::Debian/Apt::Source[nginx]/Apt::Key[Add key: 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 from Apt::Source nginx]/Apt_key[Add key: 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 from Apt::Source nginx]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Package::Debian/Apt::Source[nginx]/Apt::Setting[list-nginx]/File[/etc/apt/sources.list.d/nginx.list]/ensure: defined content as '{md5}44b89c7233b93da7e096f3f51334df58'
==> default: Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 events
==> default: Notice: /Stage[main]/Nodejs::Install/Package[nodejs]/ensure: created
==> default: Notice: /Stage[main]/Package::Inotify_tools/Package[inotify-tools=3.14-1ubuntu1]/ensure: created
==> default: Notice: /Stage[main]/Package::React_presets/Exec[install-web-packages]/returns: executed successfully
==> default: Notice: /Stage[main]/Package::Xmltodict/Package[xmltodict]/ensure: created
==> default: Notice: /Stage[main]/Package::Fetch/Package[whatwg-fetch@1.1.0]/ensure: created
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[uglify-js@2.7.5]/ensure: created
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[imagemin@5.2.2]/ensure: created
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[node-sass@4.5.0]/ensure: created
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[babel-core@6.22.1]/ensure: created
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[browserify@14.0.0]/ensure: created
==> default: Notice: /Stage[main]/Package::Webcompilers/Package[babelify@7.3.0]/ensure: created
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/etc/init/browserify.conf]/ensure: defined content as '{md5}685efe0736b4ebb570b72e9a22981f44'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/vagrant/puppet/environment/vagrant/modules/compiler/scripts/browserify]/mode: mode changed '0777' to '0755'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/etc/init/imagemin.conf]/ensure: defined content as '{md5}a4897de3d2afed6d3ae53e98769d46dd'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/vagrant/puppet/environment/vagrant/modules/compiler/scripts/imagemin]/mode: mode changed '0777' to '0755'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/etc/init/sass.conf]/ensure: defined content as '{md5}e5da2c5795131e1ae9c68c3743e2aa84'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/vagrant/puppet/environment/vagrant/modules/compiler/scripts/sass]/mode: mode changed '0777' to '0755'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/etc/init/uglifyjs.conf]/ensure: defined content as '{md5}e987c04f2581502ce0ad36c7b3063234'
==> default: Notice: /Stage[main]/Compiler::Webcompilers/File[/vagrant/puppet/environment/vagrant/modules/compiler/scripts/uglifyjs]/mode: mode changed '0777' to '0755'
==> default: Notice: /Stage[main]/Compiler::Start_sass/Service[sass]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: /Stage[main]/Compiler::Start_uglifyjs/Service[uglifyjs]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: /Stage[main]/Compiler::Start_browserify/Service[browserify]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: /Stage[main]/Compiler::Start_imagemin/Service[imagemin]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: /Stage[main]/Compiler::Initial_compile/Exec[touch-jsx-files]/returns: executed successfully
==> default: Notice: /Stage[main]/Compiler::Initial_compile/Exec[node-sass-initial-compile]/returns: executed successfully
==> default: Notice: /Stage[main]/Nginx::Package::Debian/Package[nginx]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/conf.stream.d]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/conf.mail.d]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/nginx]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/log/nginx]/group: group changed 'adm' to 'root'
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/nginx/client_body_temp]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/nginx/proxy_temp]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/streams-enabled]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/streams-available]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/nginx.conf]/content: content changed '{md5}a85dcbc5367783f3d42d0b7eab22ea50' to '{md5}de983fff1c3d7515a8100b89ed6ab532'
==> default: Notice: /Stage[main]/Webserver::Service/Nginx::Resource::Vhost[machine-learning.com]/Concat[/etc/nginx/sites-available/machine-learning.com.conf]/File[/etc/nginx/sites-available/machine-learning.com.conf]/ensure: defined content as '{md5}5937732bcc4cee3f211f266c2e3ccbb1'
==> default: Notice: /Stage[main]/Webserver::Service/Nginx::Resource::Vhost[machine-learning.com]/File[machine-learning.com.conf symlink]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Service/Service[nginx]: Triggered 'refresh' from 1 events
==> default: Notice: Applied catalog in 573.09 seconds

[jeff1evesque]

We need to cleverly arrange the implementation of include, along with require. This should be done between the compiler, and webserver modules, within both our vagrant, and docker environment. Specifically, we should require that the compiler has properly executed within the webserver module.

[jeff1evesque]

After a fresh initial vagrant up, we are now able to render the application immediately. Additionally, we were able to render the application as expected after vagrant halt && vagrant up. This means our vagrant puppet environment is sufficiently defined. We will later comb through each module, to ensure each module properly defines corresponding relationships via the various include, require, and contain implementations. However, we need to debug why our docker environment does not build successfully.

Note: on our home network we were able to build the current application just under 11 minutes, which is less than the stated README.md (i.e. ~15 minutes):

...
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/conf.stream.d]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/conf.mail.d]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/nginx]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/log/nginx]/group: group changed 'adm' to 'root'
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/nginx/client_body_temp]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/var/nginx/proxy_temp]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/streams-enabled]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/streams-available]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Config/File[/etc/nginx/nginx.conf]/content: content changed '{md5}a85dcbc5367783f3d42d0b7eab22ea50' to '{md5}de983fff1c3d7515a8100b89ed6ab532'
==> default: Notice: /Stage[main]/Webserver::Service/Nginx::Resource::Vhost[machine-learning.com]/Concat[/etc/nginx/sites-available/machine-learning.com.conf]/File[/etc/nginx/sites-available/machine-learning.com.conf]/ensure: defined content as '{md5}5937732bcc4cee3f211f266c2e3ccbb1'
==> default: Notice: /Stage[main]/Webserver::Service/Nginx::Resource::Vhost[machine-learning.com]/File[machine-learning.com.conf symlink]/ensure: created
==> default: Notice: /Stage[main]/Nginx::Service/Service[nginx]: Triggered 'refresh' from 1 events
==> default: Notice: /Stage[main]/Webserver::Start/Service[start_gunicorn]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: Applied catalog in 655.07 seconds

[jeff1evesque]

We are attempting to install nodejs at different particular versions per:

• Reinstalling npm v3 fails on Docker. npm/npm#9863 (comment)
• Making it possible to provide concrete nodejs versions to force upgrade/downgrade (#197) voxpupuli/puppet-nodejs#198 (comment)
• https://deb.nodesource.com/node_5.x/pool/main/n/nodejs/

With the intention of resolving our current traceback error:

• https://travis-ci.org/jeff1evesque/machine-learning/builds/208953776#L4612-L4803
• Reinstalling npm v3 fails on Docker. npm/npm#9863 (comment)

[jeff1evesque]

2f9a456: an engineer from travis confirmed that adding the above services directive, causes the CI build to default to the trusty distribution. They will additionally try to fit this into corresponding comments in the documentation. However, the same engineer provided the below concise solution, which allows us to choose between the precise, or the trusty distribution, from our .travis.yml:

language: python
dist: trusty
sudo: required