feat(gremlin): Add proxy of Gremlin API to retrieve command and target templates #725

mattrjacobs · 2019-02-14T18:19:12Z

Necessary for Deck change: spinnaker/deck#6549
Also related to Orca change: spinnaker/orca#2664

…t templates

spinnakerbot · 2019-02-14T18:20:27Z

We prefer that non-test backend code be written in Java or Kotlin, rather than Groovy. The following files have been added and written in Groovy:

gate-web/src/main/groovy/com/netflix/spinnaker/gate/controllers/GremlinController.groovy

See our server-side commit conventions here.

avram · 2019-02-14T20:42:18Z

gate-web/config/gate.yml

@@ -35,7 +35,8 @@ services:
            packageType: RPM
 # optional services:
  echo:
-    enabled: false
+    baseUrl: http://localhost:8089
+    enabled: true


Why does this PR modify Echo configurations?

I believe it's to improve the local development story ... do these configs get used for anything other than running locally? Does halyard use them @ezimanyi? (we don't use these configs internally so a bit out of the loop).

FWIW echo is no longer an optional service as pipeline triggering flows through it.

@ajordens : Halyard does not use these configs. Halyard uses the ones in the halconfig directory of each service as a base, which it then templates on the settings generated from your halyard config, but doesn't use the ones in service-web/config/...

👍 I thought that's how it worked but thanks for clarifying @ezimanyi!

When I run Gate without Echo being enabled, and execute a pipeline, I get the following stacktrace:

2019-02-19 10:28:35.671 ERROR 5815 --- [nio-8084-exec-4] c.n.s.g.controllers.PipelineController : Unable to trigger pipeline (application: foo, pipelineId: NewTemplate) java.lang.NullPointerException: Cannot invoke method postEvent() on null object at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91) ~[groovy-all-2.4.15.jar:2.4.15] at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47) ~[groovy-all-2.4.15.jar:2.4.15] at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-all-2.4.15.jar:2.4.15] at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34) ~[groovy-all-2.4.15.jar:2.4.15] at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-all-2.4.15.jar:2.4.15] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-all-2.4.15.jar:2.4.15] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128) ~[groovy-all-2.4.15.jar:2.4.15] at com.netflix.spinnaker.gate.services.PipelineService.triggerViaEcho(PipelineService.groovy:114) ~[main/:na]

I had forgotten this was included in this PR. If it makes your life easier, I'm happy to submit this as a separate PR, and also move the echo config out of the optional block in gate.yml

Fixing the local dev config seems alright to me.

avram · 2019-02-14T20:42:43Z

gate-web/config/gate.yml

@@ -48,6 +49,9 @@ services:
    enabled: false
    canaryConfigStore: true
    baseUrl: http://localhost:8090
+  gremlin:
+    enabled: true


It doesn't seem correct that Gremlin would default to on

I'd suggest a new block:

integrations: gremlin: baseUrl: https://api.gremlin.com/v1 enabled: true|false

Makes sense @ajordens and thanks for the review @avram . Is the following what you have in mind?

Create a new block integrations in gate.yml

Move GremlinController and GremlinService into a new top-level module gate-integrations-gremlin.

Fix the dependency injection to respect the integrations.gremlin.enabled flag and only instantiate GremlinController and GremlinService if that flag is true.

I'm guessing similar thoughts apply to the Gremlin work in orca? I'm not as familiar with how we can separate out the UI changes in deck - any thoughts on how that should get modularized? In the meantime, will start working through the gate + orca changes.

ajordens

I think there's an opportunity here to define how external integrations would fit in.

I'd suggest splitting out the grelim-specific code into a new gate-integrations-gremlin artifact rather than putting code into gate-web.

This would provide a better story around ongoing review/maintenance. I'd like to be able to say, Gremlin folks own gate-integrations-gremlin and I'm generally ok with what goes in there vs. having to worry about changes in a core artifact like gate-web.

ajordens · 2019-02-14T23:17:09Z

gate-web/config/gate.yml

@@ -35,7 +35,8 @@ services:
            packageType: RPM
 # optional services:
  echo:
-    enabled: false
+    baseUrl: http://localhost:8089
+    enabled: true


I believe it's to improve the local development story ... do these configs get used for anything other than running locally? Does halyard use them @ezimanyi? (we don't use these configs internally so a bit out of the loop).

FWIW echo is no longer an optional service as pipeline triggering flows through it.

ajordens · 2019-02-14T23:17:47Z

gate-web/config/gate.yml

@@ -48,6 +49,9 @@ services:
    enabled: false
    canaryConfigStore: true
    baseUrl: http://localhost:8090
+  gremlin:
+    enabled: true


I'd suggest a new block:

integrations: gremlin: baseUrl: https://api.gremlin.com/v1 enabled: true|false

ezimanyi · 2019-02-16T04:18:53Z

gate-web/src/main/groovy/com/netflix/spinnaker/gate/controllers/GremlinController.groovy

+import org.springframework.web.bind.annotation.RestController
+
+@RestController
+@RequestMapping('/gremlin')


This also needs to be made conditional on gremlin being enabled, otherwise startup will fail trying to autowire the missing GremlinService.

ajordens

These PRs are ready to merge after the suggest config
and code locations are updated.

ajordens · 2019-02-18T20:16:52Z

gate-web/config/gate.yml

@@ -35,7 +35,8 @@ services:
            packageType: RPM
 # optional services:
  echo:
-    enabled: false
+    baseUrl: http://localhost:8089
+    enabled: true


👍 I thought that's how it worked but thanks for clarifying @ezimanyi!

… wired up the enabled flag to DI

mattrjacobs · 2019-02-19T23:01:07Z

gate-core/src/main/groovy/com/netflix/spinnaker/gate/config/ServiceConfiguration.groovy

@@ -45,6 +46,6 @@ class ServiceConfiguration {
  }

  Service getService(String name) {
-    services[name]
+    (services + integrations)[name]


I kept the integrations structure the same as services in gate.yml so that this code works

mattrjacobs · 2019-02-19T23:02:22Z

...s-gremlin/src/main/groovy/com/netflix/spinnaker/gate/config/gremlin/GremlinGateConfig.groovy

+    createClient('gremlin', GremlinService, okHttpClient)
+  }
+
+  private <T> T createClient(String serviceName,


These helper methods are copied from GateConfig. A more DRY approach would be to have some abstracted-out code that both GateConfig and GremlinGateConfig depend on, but I wanted to get your input before I attempt to refactor one of the core classes

mattrjacobs · 2019-02-19T23:02:48Z

...mlin/src/main/groovy/com/netflix/spinnaker/gate/controllers/gremlin/GremlinController.groovy

+
+@RestController
+@RequestMapping('/gremlin')
+@ConditionalOnExpression('${integrations.gremlin.enabled:true}')


this is new, and I've tested it. only other change in this file is the package (addition of gremlin)

Won't this default it to enabled (whereas the bean will default to disabled)?

When I tested this locally, setting the value of the property in gate.yml to false did not inject this RequestMapping. When the UI made the call, it 404ed, as expected.

When I set it to true, the endpoint was properly mapped and made the appropriate redirection to the Gremlin API.

I'm referring to the case where the config setting is absent entirely---if you don't have the integrations.gremlin.enabled key in your environment at all. Then:

the bean checks @ConditionalOnProperty('integrations.gremlin.enabled') and does not get created

the controller checks @ConditionalOnExpression('${integrations.gremlin.enabled:true}'), falls back to the default of true in the expression, and gets created.

Then the controller tries to autowire the bean, and gate fails to start up.

I misunderstood the syntax of the annotation - thanks for the walkthrough! Changed as you suggested and manually tested all three variants: true / false / missing.

mattrjacobs · 2019-02-19T23:03:37Z

gate-web/gate-web.gradle

@@ -21,6 +21,7 @@ repositories {
 dependencies {
  compile project(":gate-core")
  compile project(":gate-proxy")
+  compile project(":gate-integrations-gremlin")


I didn't see any way around this. I need the @Configuration of GremlinGateConfig to get detected, so I need those classes visible to the webapp.

👍 Yes, there's no way around this right now.

Subsequent integrations we could probably wire something up that would automatically add dependencies on anything matching a *integrations* pattern.

ajordens

Adjust the endpoint and it looks mergeable to me.

ajordens · 2019-02-20T19:56:08Z

gate-core/src/main/groovy/com/netflix/spinnaker/gate/config/ServiceConfiguration.groovy

@@ -45,6 +46,6 @@ class ServiceConfiguration {
  }

  Service getService(String name) {
-    services[name]
+    (services + integrations)[name]


ajordens · 2019-02-20T19:59:09Z

...mlin/src/main/groovy/com/netflix/spinnaker/gate/controllers/gremlin/GremlinController.groovy

+import org.springframework.web.bind.annotation.RestController
+
+@RestController
+@RequestMapping('/gremlin')


Lets make this /integrations/gremlin

ajordens · 2019-02-20T20:00:03Z

gate-web/config/gate.yml

@@ -35,7 +35,8 @@ services:
            packageType: RPM
 # optional services:
  echo:
-    enabled: false
+    baseUrl: http://localhost:8089
+    enabled: true


Fixing the local dev config seems alright to me.

Move the `GremlinController` endpoints to be rooted at `/integrations/gremlin`. `integrations.gremlin.enabled` is `false` when running in local development mode.

ajordens · 2019-02-20T22:01:08Z

@mattrjacobs I pushed some changes to your branch that did a bit of cleanup + groovy -> java conversion ... if it looks good to you I'll merge.

mattrjacobs · 2019-02-21T16:15:03Z

LGTM @ajordens - thanks for doing the conversion. I submitted a PR to deck to adjust the Gate endpoint here: spinnaker/deck#6591. Will begin work to add the integrations pattern to orca as well.

feat(gremlin): Add proxy of Gremlin API to retrieve command and targe…

e51dd90

…t templates

avram reviewed Feb 14, 2019

View reviewed changes

ajordens reviewed Feb 14, 2019

View reviewed changes

ezimanyi reviewed Feb 16, 2019

View reviewed changes

ajordens reviewed Feb 18, 2019

View reviewed changes

feat(gremlin): Move Gremlin into gate-integrations-gremlin module and…

a10edf5

… wired up the enabled flag to DI

mattrjacobs commented Feb 19, 2019

View reviewed changes

Change annotation to not load GremlinController when config is missing

83f0fab

ajordens reviewed Feb 20, 2019

View reviewed changes

ajordens added 3 commits February 20, 2019 12:01

Merge branch 'master' into gremlin-support

383da51

fix(gremlin): Misc code cleanup and build fixing

1a17230

Move the `GremlinController` endpoints to be rooted at `/integrations/gremlin`. `integrations.gremlin.enabled` is `false` when running in local development mode.

fix(gremlin): Groovy -> Java Conversion

4a51623

mattrjacobs mentioned this pull request Feb 21, 2019

feat(gremlin): update to Gremlin Gate endpoint prefix spinnaker/deck#6591

Merged

Merge branch 'master' into gremlin-support

180d9b4

ajordens merged commit 400994b into spinnaker:master Feb 21, 2019

spinnakerbot added the target-release/1.13 label Feb 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gremlin): Add proxy of Gremlin API to retrieve command and target templates #725

feat(gremlin): Add proxy of Gremlin API to retrieve command and target templates #725

mattrjacobs commented Feb 14, 2019 •

edited

Loading

spinnakerbot commented Feb 14, 2019

avram Feb 14, 2019

ajordens Feb 14, 2019

ezimanyi Feb 16, 2019

ajordens Feb 18, 2019

mattrjacobs Feb 19, 2019

ajordens Feb 20, 2019

avram Feb 14, 2019

ajordens Feb 14, 2019

mattrjacobs Feb 19, 2019

ajordens left a comment •

edited

Loading

ajordens Feb 14, 2019

ajordens Feb 14, 2019

ezimanyi Feb 16, 2019

ajordens left a comment

ajordens Feb 18, 2019

mattrjacobs Feb 19, 2019

ajordens Feb 20, 2019

mattrjacobs Feb 19, 2019

mattrjacobs Feb 19, 2019

ezimanyi Feb 19, 2019

mattrjacobs Feb 19, 2019

ezimanyi Feb 20, 2019

mattrjacobs Feb 20, 2019

mattrjacobs Feb 19, 2019

ajordens Feb 20, 2019

ajordens left a comment

ajordens Feb 20, 2019

ajordens Feb 20, 2019

ajordens Feb 20, 2019

ajordens commented Feb 20, 2019

mattrjacobs commented Feb 21, 2019

feat(gremlin): Add proxy of Gremlin API to retrieve command and target templates #725

feat(gremlin): Add proxy of Gremlin API to retrieve command and target templates #725

Conversation

mattrjacobs commented Feb 14, 2019 • edited Loading

spinnakerbot commented Feb 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajordens left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajordens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajordens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajordens commented Feb 20, 2019

mattrjacobs commented Feb 21, 2019

mattrjacobs commented Feb 14, 2019 •

edited

Loading

ajordens left a comment •

edited

Loading