Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture Jenkins job names correctly when the folder plugin is being used #42

Merged
merged 2 commits into from
Dec 23, 2015

Conversation

AMeng
Copy link
Contributor

@AMeng AMeng commented Nov 21, 2015

This is meant to address spinnaker/spinnaker#605.
I'm able to hit http://localhost:8080/jobs/<JenkinsName>/ and see a list of jobs with the expected names:

[
  "Job1",
  "Folder1/job/Job2",
  "Folder1/job/Folder2/job/Job3"
]

Its also likely that a few endpoints will need to be updated since they currently have the job names in the middle of other parameters, such as "/jobs/{jobName}/builds" which could be troublesome.

@@ -73,7 +73,24 @@ class InfoController {
throw new MasterNotFoundException("Master '${master}' does not exist")
}

jenkinsClient.jobs.list.collect { it.name }
def jobList = []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tomaslin
Copy link
Contributor

This is failing for me when I am just trying to get the job details after installing the jenkins folders plugin on a jenkins host. There doesn't seem to be a straightforward way to modify spring paths to accept /

https://github.com/spinnaker/igor/blob/master/igor-web/src/main/groovy/com/netflix/spinnaker/igor/jenkins/InfoController.groovy#L79

I have this job

[
"Jenkins folder/job/hakuna matata"
]

under the host jenkins1. When I try to get the details for it, it doesn't appear to handle the path correctly:

http://localhost:8080/jobs/jenkins1/Jenkins%20folder/job/hakuna%20matata

}

@RequestMapping(value = '/jobs/{master}/{job:.+}')
JobConfig getJobConfig(@PathVariable String master, @PathVariable String job) {
@RequestMapping(value = '/jobs/{master}/**')
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to allow forward slashes in the names of Jenkins jobs, we can manually parse the URL like this. If this looks acceptable, I can do something similar in the BuildController. I'll have to move the job names to the end of all those URLs and manually parse them in this same way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would probably prefer to move job name as a query parameter like some other people have suggested in the comments.

@AMeng
Copy link
Contributor Author

AMeng commented Nov 30, 2015

To be clear, these are the URL changes I've made. Note that the job name has to be at the end of the URL in order to allow any arbitrary number of forward slashes in the name.

Old URL New URL
/jobs/{master}/{job}/{buildNumber} /builds/status/{buildNumber}/{master}/{job}
/jobs/{master}/queue/{item} /builds/queue/{master}/{item}
/jobs/{master}/{job}/builds /builds/all/{master}/{job}
/jobs/{master}/{job}/{buildNumber}/properties/{fileName} /builds/properties/{buildNumber}/{fileName}/{master}/{job}

@ajordens
Copy link
Contributor

ajordens commented Dec 1, 2015

This looks to me like a breaking API change?

If so, we'll need to version the API in here (or perhaps just leave old and new running side-by-side).

@AMeng
Copy link
Contributor Author

AMeng commented Dec 1, 2015

@ajordens, doesn't Gate handle talking to Igor? It seems like users shouldn't be working directly with Igor's API. I guess it makes sense to version it either way. Gate's URLs are changing as well (spinnaker/gate#161), so we'll need to address that issue. But yes, this is a breaking API change. If you'd like to version the API, how would you prefer to handle that? Maybe just prepend the new URLs with /v2 or something? And should we also keep the old endpoints?

@tomaslin
Copy link
Contributor

tomaslin commented Dec 2, 2015

if we leave old and new running side by side, the UI will not be able to handle jobs with the folders plugin. My vote is to change the gate API so that it has v2 attached to it.

Deck would need to be updated as well to support the new url patterns https://github.com/spinnaker/deck/blob/master/app/scripts/modules/core/ci/jenkins/igor.service.js

It also seems that bakery parses jobs incorrectly - https://github.com/spinnaker/orca/blob/master/orca-core/src/main/groovy/com/netflix/spinnaker/orca/pipeline/util/PackageInfo.groovy#L120-L124 so this would have to change as well.

The last bit to sanity check is how things are constructed to link back to the build in the clusters view in deck. I suspect this might be fine but basically we need to wire up all those pieces and test it works correctly.

@tomaslin
Copy link
Contributor

tomaslin commented Dec 2, 2015

might be useful to note that I'm just launching a new jenkins via docker run -p 8080:8080 -p 50000:50000 jenkins and then installing the folders plugin after creating a dummy user. However, since front50 runs in 8080 I change that port in my docker compose config

@gregturn
Copy link
Contributor

gregturn commented Dec 2, 2015

If encoding paths/filenames is an issue, is it possible to switch to making them query parameters? I think that would encode things. Not 100% sure how to run that through retrofit. I know Spring MVC encodes stuff, but I can't say the same for retrofit. If you're breaking the API anyway, this might be the least risky way to get the entire value over the wire.

@AMeng
Copy link
Contributor Author

AMeng commented Dec 2, 2015

Updated gate to prepend the new URLs with /v2: spinnaker/gate#161
Updated orca to fix the Jenkins URL parse issue: spinnaker/orca#669

@tomaslin
Copy link
Contributor

tomaslin commented Dec 3, 2015

Deck code here - spinnaker/deck#1766

@AMeng AMeng force-pushed the jenkins-folders-plugin branch from 97ca888 to da853ce Compare December 3, 2015 19:07
* Jobs created with the Jenkins Folders plugin are nested.
* This query looks for jobs within folders with a depth of 10.
*/
@GET('/api/xml?tree=jobs[name,jobs[name,jobs[name,jobs[name,jobs[name,jobs[name,jobs[name,jobs[name,jobs[name,jobs[name]]]]]]]]]]')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the ideal world this kind of structure and resolution would be configurable at the jenkins masters level.

@AMeng AMeng force-pushed the jenkins-folders-plugin branch from 5274a8b to fec2099 Compare December 9, 2015 17:16
@tomaslin
Copy link
Contributor

tomaslin commented Dec 9, 2015

Getting job configuration from deck UI does not work for folders.

Create a job at the level folder1/folder2/folderjob

Make the job parametrized and add a few parameters to the job.

In deck, create a new application -> new pipeline -> new jenkins stage

Select the master for the folder job and select this job.

Seems like the slashes get encoded when trying to get the job configuration:

http://192.168.99.100:8084/v2/builds/folders/jobs/folder1%2Fjob%2Ffolder2%2Fjob%2Ffolderjob

The result of this is that parameters for the job are not shown

@tomaslin
Copy link
Contributor

tomaslin commented Dec 9, 2015

Polling is broken for jobs coming from the folder plugin.

The BuildMonitor in igor gets the list of jobs via the getProjects endpoint in JenkinsClient. This information is then used to resolve if there is a new build or not and new events get sent to Echo to trigger pipelines.

https://github.com/spinnaker/igor/blob/master/igor-web/src/main/groovy/com/netflix/spinnaker/igor/jenkins/BuildMonitor.groovy#L125

The path is here -

@GET('/api/xml?tree=jobs[name,lastBuild[actions[failCount,skipCount,totalCount,urlName],duration,number,timestamp,result,building,url]]&exclude=/*/*/*/action[not(totalCount)]')
ProjectsList getProjects()

Seems that for jobs with folders attached to it, only the top level folder is displayed as an available job.

The same change to get a list of jobs needs to be made so that new build events from jobs get logged correctly.

@AMeng
Copy link
Contributor Author

AMeng commented Dec 17, 2015

I've addressed the issue with polling jobs. While testing, I also noticed that job names with spaces were causing issues because I had marked the URI as encoded to avoid encoding the forward slashes. I'll need to look over the other PRs to ensure special characters like spaces are correctly encoded.

@AMeng
Copy link
Contributor Author

AMeng commented Dec 17, 2015

Other PRs have been updated to correctly handle spaces in job names. I believe this is again ready for review.

@tomaslin
Copy link
Contributor

do you mind squashing your commits?

@AMeng AMeng force-pushed the jenkins-folders-plugin branch from 9ff17df to d7e5318 Compare December 17, 2015 21:28
@AMeng
Copy link
Contributor Author

AMeng commented Dec 17, 2015

Squashed and rebased all PRs

@ajordens
Copy link
Contributor

I know we talked about this ... but this is a breaking change and the internal Spinnaker callers of igor (gate and orca) are going to break (even if for only an optimal 5 minutes as everything is rolled out).

@AMeng I realize the there are PRs against orca and gate, but we deploy each service independently and strive to avoid dependent rollouts.

@tomaslin What do you figure the LOE would be to re-insert the old igor endpoints? As simple as restoring the code?

@AMeng
Copy link
Contributor Author

AMeng commented Dec 21, 2015

If it is a requirement that these changes be deployable separately, then Gate will have the same issues. I've prepended the new gate URLs with /v2, but did not keep the old ones around. So I would need to allow the original endpoints for both Gate and Igor.

@AMeng
Copy link
Contributor Author

AMeng commented Dec 21, 2015

@ajordens I've re-added the original endpoints for both Igor and Gate. I tried to make it as easy as possible to rip them back out whenever you think its ok to do so (both the methods and the tests are grouped on the bottom).

@tomaslin
Copy link
Contributor

I've tested this PR and with the other changes against a jenkins slave without the folders plugin and one with it. Job names with and without spaces work correctly in both and triggering works as well.. LGTM.

@ajordens can you run through the API changes and merge if you're satisfied?

tomaslin added a commit that referenced this pull request Dec 23, 2015
Capture Jenkins job names correctly when the folder plugin is being used
@tomaslin tomaslin merged commit cc0125a into spinnaker:master Dec 23, 2015
@AMeng AMeng mentioned this pull request Dec 29, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants