Xpack Foo module on mage build system #9242

sayden · 2018-11-26T19:52:44Z

A merge of the commits that @andrewkroh mention here #9230 with the contents of a dummy foo module.

@ruflin take a look here too, I prefer to do a stop & go, ask and ensure the next step :)

This enables the use of mage to build, test, and package x-pack/metricbeat. This adds it to the test matrix on Travis CI as well. But it does not modify the top-level metricbeat build to stop producing x-pack artifacts. This cut-over needs still needs to be done. (cherry picked from commit d6b1ba1) # Conflicts: # metricbeat/Dockerfile

(cherry picked from commit fd70623)

(cherry picked from commit 0a39283)

houndci-bot · 2018-11-26T19:52:54Z

x-pack/metricbeat/magefile.go

+	return mage.PythonNoseTest(mage.DefaultPythonTestUnitArgs())
+}
+
+// PythonUnitTest executes the python system tests in the integration environment (Docker).


comment on exported function PythonIntegTest should be of the form "PythonIntegTest ..."

My bad. Can you fix this @sayden?

andrewkroh · 2018-11-26T19:54:41Z

x-pack/metricbeat/magefile.go

+	if err != nil {
+		return err
+	}
+	//defer os.Remove(modulesConfigYml)


This should have been uncommented. We want to remove the file when this function finishes.

andrewkroh · 2018-11-26T19:59:49Z

Looks like the foo module needs formatted.

check: Checking for modified files or incorrect permissions
Error: some files are not up-to-date. Run 'mage fmt update' then review and commit the changes. Modified: [x-pack/metricbeat/module/foo/bar/bar.go x-pack/metricbeat/module/foo/doc.go]
make[1]: *** [check] Error 1

andrewkroh · 2018-11-26T21:49:24Z

When running mage integTest it expects to find a docker-compose.yml file. You can copy the one added in the mssql PR, but remove the mssql container defined in that file.

https://travis-ci.org/elastic/beats/jobs/459953763#L599

ruflin · 2018-11-27T05:59:53Z

I checked out the branch locally and the following files are not checked in:

        include/
        metricbeat.reference.yml
        metricbeat.yml
        modules.d/

I think all these should be checked in too?

@andrewkroh I'm a bit surprised that we did not get an error on CI because of the above files.

andrewkroh · 2018-11-27T06:40:11Z

The implementation of mage.Check uses very similar commands as our existing top-level Makefile. Neither will alarm on untracked files from what I understand, probably because this would have been to noisy for local usage. But once they are added make/mage check would alert if they were modified in any way.

beats/Makefile

Lines 88 to 89 in d997a82

    
           @git update-index --refresh 
        
           @git diff-index --exit-code HEAD --

beats/dev-tools/mage/check.go

Line 75 in 4471df5

out, err := sh.Output("git", "diff-index", "HEAD", "--", ".")

But I do think it would be nice to have it fail on this condition in CI.

ruflin · 2018-11-27T06:59:55Z

Got it. We can add such a check in a more generic way later. Thanks

@sayden Can you add the missing files?

…nd include/list.go

ruflin

LGTM.

ruflin · 2018-11-27T08:35:51Z

@kaiyan-sheng Merged. You should rebase your PR on top of this.

@sayden Do you feel comfortable to follow up with a PR for the docs collection?

In general I think we should backport this as soon as we have at least one module in x-pack for 6.x. Also as soon as we have one module in, we should remove the "fake module".

Seems this file was missing in elastic#9242

Seems this file was missing in #9242

ruflin · 2019-03-25T08:20:38Z

metricbeat/Dockerfile

@@ -1,21 +1,16 @@
-FROM golang:1.11.2
-MAINTAINER Nicolas Ruflin <ruflin@elastic.co>
+FROM golang:1.10.3


@sayden Thanks to #11330 we found this change. I wonder if any of the changes here were intended / needed?

As far as I remember this we were using 1.10.3 because of needs in the CI (formatting if I remember well) and probably. I don't remember to directly change this, I think it was a co-authored PR with @andrewkroh who maybe can give us some more light here.

It might be that things changed during the initial opening of the PR and when it get merged. Could you try a PR to get back the "old" Dockerfile again or have a look on what the actual differences and implications could be?

andrewkroh and others added 4 commits November 26, 2018 19:10

Add PROJECTS_XPACK_MAGE to top-level Makefile

d997a82

(cherry picked from commit fd70623)

Add docker to packages.yml for x-pack/metricbeat

a6de97a

(cherry picked from commit 0a39283)

Added Foo module

cffa525

sayden requested review from ruflin and andrewkroh November 26, 2018 19:52

houndci-bot reviewed Nov 26, 2018

View reviewed changes

sayden added review Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels Nov 26, 2018

andrewkroh reviewed Nov 26, 2018

View reviewed changes

sayden added 2 commits November 26, 2018 21:58

updates after mage fmt update command

fd128f8

fix a couple of minor things within magefile

d7eda29

sayden and others added 2 commits November 26, 2018 23:19

Add a near empty docker-compose.yml file

213fe31

Remove mssql env_file

d302141

Added metricbeat/metricbeat.reference.yml metricbeat/metricbeat.yml a…

a2b519f

…nd include/list.go

ruflin approved these changes Nov 27, 2018

View reviewed changes

sayden merged commit 37b67b0 into elastic:master Nov 27, 2018

ruflin added a commit to ruflin/beats that referenced this pull request Nov 27, 2018

Add missing modules.d config file for foo module

7bec110

Seems this file was missing in elastic#9242

ruflin mentioned this pull request Nov 27, 2018

Add missing modules.d config file for foo module #9253

Merged

ruflin added a commit that referenced this pull request Nov 27, 2018

Add missing modules.d config file for foo module (#9253)

6105c08

Seems this file was missing in #9242

sayden mentioned this pull request Dec 27, 2018

Cherry-pick #9802 to 6.x: Remove X-Pack foo Metricbeat module #9803

Merged

ruflin reviewed Mar 25, 2019

View reviewed changes

sayden deleted the xpack-foo-module-on-mage-build-system branch July 30, 2022 05:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Xpack Foo module on mage build system #9242

Xpack Foo module on mage build system #9242

sayden commented Nov 26, 2018

houndci-bot Nov 26, 2018

andrewkroh Nov 26, 2018

sayden Nov 26, 2018

andrewkroh Nov 26, 2018 •

edited

Loading

sayden Nov 26, 2018

andrewkroh commented Nov 26, 2018 •

edited

Loading

andrewkroh commented Nov 26, 2018 •

edited

Loading

ruflin commented Nov 27, 2018

andrewkroh commented Nov 27, 2018 •

edited

Loading

ruflin commented Nov 27, 2018

ruflin left a comment

ruflin commented Nov 27, 2018

ruflin Mar 25, 2019

sayden Mar 25, 2019

ruflin Mar 25, 2019

Xpack Foo module on mage build system #9242

Xpack Foo module on mage build system #9242

Conversation

sayden commented Nov 26, 2018

houndci-bot Nov 26, 2018

Choose a reason for hiding this comment

andrewkroh Nov 26, 2018

Choose a reason for hiding this comment

sayden Nov 26, 2018

Choose a reason for hiding this comment

andrewkroh Nov 26, 2018 • edited Loading

Choose a reason for hiding this comment

sayden Nov 26, 2018

Choose a reason for hiding this comment

andrewkroh commented Nov 26, 2018 • edited Loading

andrewkroh commented Nov 26, 2018 • edited Loading

ruflin commented Nov 27, 2018

andrewkroh commented Nov 27, 2018 • edited Loading

ruflin commented Nov 27, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin commented Nov 27, 2018

ruflin Mar 25, 2019

Choose a reason for hiding this comment

sayden Mar 25, 2019

Choose a reason for hiding this comment

ruflin Mar 25, 2019

Choose a reason for hiding this comment

andrewkroh Nov 26, 2018 •

edited

Loading

andrewkroh commented Nov 26, 2018 •

edited

Loading

andrewkroh commented Nov 26, 2018 •

edited

Loading

andrewkroh commented Nov 27, 2018 •

edited

Loading