windows packaging tests

[andyb-elastic]

Add windows to the set of platforms we verify packaging correctness on.

The BATS tests will be rewritten in java/groovy so they'll be portable on all test platforms. We'd like to build these tests into a jar on the host and then execute the jar in the vm. Depending on how they're written it may make more sense to run them as a gradle task.

Because of licensing issues, we can't make a windows vagrant image publicly available. Users will be able to specify a windows image to use when running these tests, and the tests will include checks for correct java version, etc.

This was from a discussion with @tlrx @rjernst @nik9000 @jasontedor

[jasontedor]

Supersedes #18475

[andyb-elastic]

Here's roughly my roadmap for this since it's probably too much for one pull request

• Add windows images for server 2012-r2 and 2016 for tests to the vagrantfile and vagrant test plugin. Allow an alternate windows image to be used by setting an environment variable. Don't run any tests on these boxes just yet

• Create the groovy project that will be the new portable packaging tests. Add tasks to run these on all boxes, including the windows ones. Don't add all the tests to the new project yet. The bats tests will still run as usual

• In parts, move tests from the bats project to the new groovy project. Once all tests are moved over, remove the bats tasks from the build.

• Add testing windows installer as a service

[rjernst]

Create the groovy project that will be the new portable packaging tests.

May I suggest qa/packaging? qa/vagrant was annoying for many reasons :). After all of them are transitioned, I think we can (and should) remove the remapping we do of the vagrant mount back to /vagrant.

[jasontedor]

Add windows images for server 2012-r2 and 2016 for tests to the vagrantfile and vagrant test plugin. Allow an alternate windows image to be used by setting an environment variable. Don't run any tests on these boxes just yet

We should not enable these by default, and the boxes should not be mentioned in the Vagrantfile. The boxes that we are going to use are "closed source". We do not refer to "closed source" in the open source code base. We should not require open source users to specify an environment variable to get around running these on boxes they can not see. All that we should do is enable these to run if you provide your own box, which we will do on our internal CI.

Add testing windows installer as a service

Can you clarify what this refers to? I think that you mean installing Elasticsearch on Windows as a service, not using the Windows MSI installer but what you write could be interpreted as the latter.

[andyb-elastic]

May I suggest qa/packaging? qa/vagrant was annoying for many reasons :). After all of them are transitioned, I think we can (and should) remove the remapping we do of the vagrant mount back to /vagrant

Sounds good, that seems cleaner 👍

We should not enable these by default, and the boxes should not be mentioned in the Vagrantfile. The boxes that we are going to use are "closed source". We do not refer to "closed source" in the open source code base. We should not require open source users to specify an environment variable to get around running these on boxes they can not see. All that we should do is enable these to run if you provide your own box, which we will do on our internal CI.

Agree, they won't run by default (when vagrant.boxes is unset). I think they should run with vagrant.boxes=all, and I can add options for linux-all (the current behavior of all) and windows-all

I see why we wouldn't want to mention the images explicitly in the Vagrantfile (although it looks like vagrant doesn't complain about not finding the image until you try to start it). They'll be hidden behind an environment variable in the Vagrantfile, something like

Vagrant.configure(2) do |config|
  if ENV['WINDOWS_PACKAGING']
    config.vm.define 'windows-server-2012r2' do |config|
      config.vm.box = ENV['WINDOWS_2012R2_VAGRANT_BOX']
    end
    config.vm.define 'windows-server-2016' do |config|
      config.vm.box = ENV['WINDOWS_2016_VAGRANT_BOX']
    end
  end
  # the other boxes
end

and then gradle will decide whether to set these env variables. I guess it can get the box names either from command line properties or its own environment (I'll favor command line props)

Can you clarify what this refers to? I think that you mean installing Elasticsearch on Windows as a service, not using the Windows MSI installer but what you write could be interpreted as the latter.

Right, the elasticsearch-service script, not the MSI

[jasontedor]

Thanks @andyb-elastic, this sounds good to me now.

I guess it can get the box names either from command line properties or its own environment (I'll favor command line props)

I prefer properties too (we can do the -P thing that we already do.

Right, the elasticsearch-service script, not the MSI

Thank you for clarifying, I wanted to make sure the MSI is out of scope here (and belongs elsewhere, in the MSI repository).

[tlrx]

Thanks @andyb-elastic, this plan looks good. Once the new packaging test project is available, we should make sure to involve many developers to convert the tests from bats to the new format. That will help to parallelize the work and spread the knowledge at the same time (we did this before when migrating settings to a new API or when adding integration test for aggregations).

[andyb-elastic]

We had a discussion about exactly what should be tested with plugins - currently (in bats) we just test that the plugins' jars are installed into the right place but some questioned the value of that. We may want to test basic functionality of the plugins if we can do that in a way that's not too involved.

[andyb-elastic]

Closing this as the new portable test project is created and running in CI, and basic tests have been ported over to it for all distributions we're currently testing. New work on porting and filling out packaging tests will be tracked in #32143