From 59104b81640af177b96aa007e78db13964d8cf76 Mon Sep 17 00:00:00 2001 From: Daniel Kirillov Date: Wed, 7 Feb 2018 09:31:19 -0500 Subject: [PATCH] Merge in (#102) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(provider/ecs): ECS Cache base classes (#2065) * Prep classes for the caching agents to come. * Changes based on feedback. * fix(provider/gce):Enable RPS when only RATE or UTILIZATION (#2083) * feat(provider/kubernetes): v2 cache namespaces & relationship (#2087) * feat(provider/kubernetes): V2 enable minimal clusters (#2090) * feat(provider/aws): Enable optional AWS Shield protection on ELB & ALBs (#2089) * feat(provider/amazon): block device config for c5 instance types (#2092) * fix(provider/gce): Ensure referenced instances exist during L4 delete. (#2094) * feat(provider/kubernetes): cache network policies (#2095) * feat(provider/kubernetes): v2 server group manager (#2093) * feat(provider/amazon): Optionally tag server groups w/ app/stack/details (#2096) Set `aws.defaults.addAppStackDetailTags: true` to enable. When enabled, all created server groups will be tagged with: - `spinnaker:application` - `spinnaker:stack` - `spinnaker:details` The tag will only be created if the value is non-null. By default, these tags will be propagated to launched instances. * fix(provider/kubernetes): fix v2 instance lookup (#2098) * feat(provider/ecs): ECS Cluster caching classes and tests. (#2091) * ECS Cluster caching classes and tests. * Base abstraction. * feat(provider/kuberentes): v2 security group provider (#2097) * feat(provider/kubernetes): v2 pod logs (#2099) * fix(provider/kubernetes): security group yaml (#2100) add yaml field to KubernetesV1SecurityGroup so that the "Show YAML" modal works in deck. * feat(kubernetes/artifacts): Support deploying images dynamically (#2102) * fix(provider/kubernetes): docker is not a hard-dependency (#2106) * fix(provider/amazon): Do not delete listeners after updating them. Pretty bad idea. (#2105) * feat(provider/kubernetes): split out k8s providers (#2107) Needed to separate the search endpoints for each provider * feat(provider/kubernetes): v2 search endpoint (#2108) * feat(provider/kubernetes): Allow op to override versioning (#2110) * feat(provider/kubernetes): Add kubectl to Dockerfile. (#2104) * fix(provider/gce): Handle partially-formed L7 LB resources properly. (#2111) * fix(provider/kubernetes): fix application key parsing (#2109) * perf(provider/gce): s/get/list/ in ILB caching agent. (#2112) A few small fixes/guards to the L7 caching agent as well. * feat(provider/kubernetes): remove dependency on api-client (#2113) * feat(provider/kubernetes): reenable kube monitoring (#2115) * feat(core/search): Add parameter for fetching a batch of servergroups by name (#2114) * fix(provider/gce): Fix TCP LB disabled state calculation. (#2116) * perf(provider/gce): s/get/list/ in TCP/SSL LB caching agents. (#2117) * feat(provider/kubernetes): v2 rollback (undo rollout) (#2118) * feat(provider/kubernetes): scale manifest (#2119) * feat(provider/kubernetes): k8s only search provider (#2121) * fix(docker): Use wget instead of curl (#2122) * fix(provider/kubernetes): read namespace as region (#2123) * feat(provider/kuberentes): pause rollout (#2120) * feat(provider/kuberentes): v2 non-enum k8s kinds (#2125) * feat(provider/kubernetes): search result hydrators (#2124) * feat(provider/kubernetes): resume rollout (#2126) * fix(provider/kubernetes): v2 fix cred validation (#2129) * fix(perf): Avoid manually copying bytes between streams (#2127) * fix(provider/kubernetes): autowire correct objectmapper (#2133) * fix(provider/kubernetes): attach correct op decorators (#2130) * fix(provider/kubernetes): switch access modifiers for KubernetesKind constructors (#2128) * feat(provider/kubernetes): runJob multi-container (#2135) support multiple containers in the run job description. we have to support both `container` and `containers` until we can deprecate the `container` field since the current defs are all using `container`. * feat(entitytags): Support indexing / searching by application (#2132) If `entityRef.application` is not explicitly provided, it will be parsed from `entityRef.entityId` using frigga conventions. Searching by application can be accomplished by `/tags?application=my_application` * fix(provider/gce): Tolerate named port values as Doubles. (#2137) Pipelines are being persisted with 'port' values of NamedPorts as Doubles. This is an issue. This change tolerates that behavior in deploys since users will have pipelines with Double ports in the wild. * fix(provider/dcos): Use createApp instead of updateApp for deploy (#2136) * fix(provider/dcos): Set container type appropriately. * fix(provider/dcos): use createApp instead of updateApp for deploy * fix(provider/dcos): Remove dcos client proxy exception wrapping * Added tests for Keys. Changed separator. (#2101) * fix(provider/kubernetes): fix security group loading by name (#2134) * fix(kubectl): --to-revision (#2140) * feat(provider/kubernetes): allow OAuth token authentication (#2147) * refactor(provider/openstack): Updated openstack4j to 3.1.0 and removed components that are available upstream (#2103) * fix(elasticsearch): Force refresh when fetching entity tags from Front50 (#2149) `elasticsearch_index_template.json` is the index template that should be applied in elastic search. * fix(provider/aws): Fix updating ALB listeners (#2148) * feat(provider/google): Label instances so they can easily be grouped later. (#2142) * feat(provider/kubernetes): more robust status (#2152) * feat(provider/kubernetes): default to frigga (#2156) * feat(provider/kubernetes): v2 configmap support (#2157) * feat(provider/kuberentes): send deployed artifact to orca (#2158) * feat(provider/kubernetes): v2 support secrets (#2159) * feat(provider/kubernetes): register docker image replacers (#2160) * feat(provider/kubernetes): v1 pod logs (#2162) support pod logs for v1 provider. going to be useful when working on run job logs as well. * feat(provider/kubernetes): v2 support daemonset (#2161) * fix(provider/gce): Tolerate failed backend service getHealth() calls. (#2153) * fix(provider/gce): Break out labels. (#2163) * fix(provider/aws): Should use edda to lookup target groups by name (#2165) * ECS Container Instance caching classes and tests. (#2143) * fix(provider/aws): Remove invalid `spinnaker:` tags (#2168) Remove any `spinnaker:*` tags that are no longer valid (ie. cloning across stacks and `spinnaker:stack` no longer points at the correct stack). This is necessary as `deck` will _now_ supply existing tags when cloning. * feat(provider/kubernetes): runJob logs (#2164) add support for capturing logs as part of `collectJob`. logs are added to the job status. i'm not sure how well this scales as log size grows so this may not be the best implementation, but it may work for the short term. since execution details are persisted to volitile storage storing them with the jobs status is best since the reference to the job pod is also stored there. * refactor(cf): Stop building CF provider until maintainer is found (#1942) BREAKING CHANGE: Stops support of CF until maintainer is found * bug(provider/openstack) - LoadBalancer security groups are optional. (#2154) * chore(dependencies): Bump to 0.123.0 (#2169) * feat(provider/kubernetes): load artifacts alongside manifest (#2166) * refactor(provider/kubernetes): allow api version subclassing (#2167) * chore(jobs): turn down log-level (#2171) * fix(web): Do not throw exception when sg not found (#2172) * feat(provider/ecs): ECS Task Definition caching classes and tests. (#2145) * ECS Task Definition caching classes and tests. * ObjectMapper is now injected. * refactor(entitytags): Support for alerts/notices on any entity type (#2175) * feat(cats): Support for selectively enabling agents (#2177) This is an assist to local development when you do not want _all_ agents to run by default. ``` redis: agent: enabledPattern: .*reservation.* ``` By default `enabledPattern` is `.*` and all agents will be scheduled. * Use setApiKey when using token from service account (#2178) * fix(provider/kubernetes): v1 add docker registries in all cases (#2176) * bug(provider/openstack) - Display VIP for LBs that do not have Floating IP instead of null.. (#2170) * fix(provider/kubernetes): don't throw on invalid lb op (#2184) * fix(provider/aws): Reservation caching agent, now with less RxJava (#2180) * feat(provider/kubernetes): replace deployed configmap volumes (#2183) * feat(xenial_debians): Add systemd service configuration. (#2182) * feat(xenial_debians): Add systemd service configuration. * feat(xenial_builds): Fix typo. * feat(provider/ecs): ECS Service caching classes and tests. (#2144) * ECS Service caching classes and tests. * Objectmapper is now injected. * feat(provider/aws): Support expected capacity constraint when resizing (#2179) Support a constraint wherein the server groups' current capacity must match expected prior to a resize. There are scenarios wherein a user initiates a resize via the UI at the same time that an autoscaling event is happening behind the scenes. In such a scenario, the resize via UI != autoscaling and some churn will occur. `deck` will be changed to supply the capacity constraint. * fix(provider/aws): Retry createAutoScalingGroup() / updateAutoScalingGroup() (#2185) * feat(provider/kubernetes): cache controller revisions (#2138) * fix(provider/kubernetes): fix registry init (#2188) if using the `namespaces` key for kubernetes accounts, reconfigure registries using those namespaces. * feat(provider/kubernetes): v2 support labels on delete & scale (#2190) * Xenial builds (#2189) * feat(xenial_debians): Add systemd service configuration. * fix(provider/kubernetes): fix onDemand cache update in controller agent. (#2131) * chore(dependencies): update to latest spinnaker-dependencies Also updates gradle to 3.5 * fix(provider/kubernetes): fix registry init 2 (#2191) the previous fix caused sever performance issues. this fix will only configure hard coded registries once, only reconfiguring failed registries on the next loop, if any. * fix(google): Retry on all 5xx errors from the platform. (#2193) * fix(core): fix bean mismatch due to Spring update removes unnecessary corsfilter bean - clouddriver should not be hit by browser agents * chore(warnings): fix compiler warnings * feat(provider/ecs): ECS Task caching classes and tests. (#2146) * ECS Task caching classes and tests. * ObjectMapper is now injected. * fix(cats): fix conditional instantiation of EurekaNodeStatusProvider * fix(build): fix spring-boot plugin usage * refactor(kubernetes): change OAuth token configuration (#2196) * fix(provider/aws): s/contraints/constraints (#2198) * fix(provider/kubernetes): fix envvar source (#2199) the bump in client library broke the API for configMap and secret envvar sources. adds an extra property for optional. defaults to true to preserve backwards compatibility. shoutout major version bumps. * chore(logging): remove extra logging (#2200) * feat(provider/gce): Support for RMIG zone selection. (#2201) * chore(systemd_logs): Cleanup unneeded logging in systemd config. (#2202) * feat(provider/ecs): ECS IAM Role caching classes and tests. (#2150) * ECS IAM Role caching classes and tests. * Moved Role and the associated classes into clouddriver-aws. * AWS' default region is being used. Cleanup. * refactor(clouddriver-core) Fixed to use const value (#2186) * fix(provider/kubernetes): registry init fix (#2205) for real this time. the previous fix caused registry namespaces to get set to an empty list the second time the initialization occurred. moving it down into this method allows us to still use the full list of namespaces (`allNamespaces`) and setup the affected ones to be initialized. This would occur if a new namespace was added, for instance. * fix(amazon/loadBalancers): Be smarter about updating ALB listeners (#2203) * fix(provider/dcos) Catch the correct exception for the proxy. (#2207) * chore(dependencies): update to 0.128.0 (#2208) * fix(core/search): Fix search query which matches non-conforming cache keys (#2209) Certain cache keys were failing due to Keys.parse() returning null. Turns out there is no need to parse the cache key and then reconstruct it. * feat(provider/kubernetes): support envFrom (#2213) adds support for envFrom for the v1 provider. envFrom can be either a secret or config map reference and will be injected into the pod as environment variables. * fix(provider/gce): Tolerate null instance template lists. (#2216) * feat(aws/loadBalancer): Support on demand caching for target groups (#2214) * refactor(provider/kubernetes): refactor delete pods operation (#2210) * feat(artifacts): sets up artifact module (#2218) My plan was to reuse the artifact work in appengine, but it would have been too messy to refactor. It didn't expose any artifact-centric interface or allow easy integration with other artifact sources without a fair bit of extra work, so I rewrote in a separate module. T he appengine GCS integration was left unchanged (with the small change of moving the storage utils into a 'AritifactUtils' class). * chore(dependencies): Bump spinnaker-dependencies version. (#2220) Picks up the new compute library versions. * fix(provider/amazon): render IPv6 ingress security group rules (#2212) * feat(provider/appengine): allow gcloud version to be configurable (#2217) * feat(provider/kubernetes): init containers (#2219) adds support for initContainer property of the Pod spec * fix(provider/kubernetes): s/name/manifestName (#2222) * fix(provider/kubernetes):Server Group is disable in Daemonset when replic set to 0 (#2206) * feat(provider/kubernetes): deploy from artifact (#2223) * fix(provider/appengine): fix monikers for app engine server groups (#2224) * fix(provider/amazon): correctly propagate ipv6 ingress security group rules on upsert (#2225) * fix(provider/aws): block device configs for h1.* and i3.* (#2227) * fix(registry): Handle situations where tag creation date cannot be fetched (#2228) * fix(provider/amazon): ignore description when upserting security group rules (#2229) * bug(provider/openstack) - filter images by region if provided. (#2230) * fix(entitytags): Retry support when fetching entity tags from Front50 (#2232) * fix(core): Create the 'RetrySupport' bean (#2233) * feat(provider/kubernetes): return modified manifests (#2234) * fix(dockerRegistry): handle 400 for token refresh (#2215) By the spec https://docs.docker.com/registry/spec/api/#base, 401 should be used to indicate an expired token, but it seems 400 is used as well. * feat(provider/kubernetes): ds & ss stability conditions (#2238) * fix(appengine): remove batch calls, revert client library (#2221) This reverts commit 8db951459adfd0259d1a2ca95323099ad4509e6b. * feat(provider/kubernetes): return deployed manifests (#2235) * feat(provider/kubernetes): v2 make rollback dynamic (#2237) * fix(provider/kubernetes): disable v1 controller caching (#2239) This had a chance of deleting k8s resources from the cache when used with daemonsets & statefulsets, since there is no type information provided with a resource name. * chore(dependencies): bump spinnaker-dependencies to 0.131.0 (#2242) - newer aws sdk * feat(artifacts): Download artifacts from GitHub (#2231) * feat(provider/gcp): Add ability for default app creds to impersonate … (#2240) * feat(provider/gcp): Add ability for default app creds to impersonate a service account * duftler comments * fix(provider/aws): Avoid being marked unhealthy after initialization (#2245) Avoid being marked unhealthy should there be a temporary blip in connectivity to one or more accounts. This frequently occurs as a result of an AWS rate limit being hit. The `health.amazon.errors` metric can be used to alert on possible errors that would have previously caused an instance to go unhealthy. * feat(provider/appengine): enable flex deployments (#2241) Allow App Engine deployments with a built container image and app.yaml. * fix(startup): make constructor bean optional (#2247) * fix(clusters): Return all matching server groups for a provider (#2249) * fix(provider/kubernetes): v2 Fix kubectl label selector flag (#2252) Currently we set `-l=` when `labelSelectors` is null or is not empty, causing all commands that pass a null value for `labelSelectors` to fail. The `-l=somelabel` flag should be set if `labelSelectors` is not null and is not empty. * fix(provider/aws): handle spotPrice == in rollingpush (#2251) * debug(provider/aws): Avoid thread pool when building reservation report (#2253) Attempting to track down an issue wherein the reservation report caching agents _stop_ and are never rescheduled after some period of time. Will revisit and remove the thread pool configuration if this change happens to improve the stop/reschedule situation. * debug(provider/aws): Additional logging for `ReservationReportCachingAgent` (#2254) * fix(provider/aws): Specify `connectionRequestTimeout` for edda (#2255) Establishes an upper bounds on the amount of time to spend waiting on an available connection from the `edda` connection pool. Previously, a caching agent could wait _indefinitely_ for an available connection. The default is now 10000 (milliseconds) and can be overridden with `aws.edda.connectionRequestTimeout`. * fix(provider/amazon): do not try to create reserved tags (#2256) * fix(provider/aws): Ensure that `HttpEntity` is closed on edda failure (#2258) Also log an error when edda retries have been exhausted. * bug(provider/openstack) - allow for binding to multiple loadbalancers with the same port mappings. (#2257) * fix(provider/aws): block device config for missing instance types (#2266) This should provide coverage for all currently available instance types. Will look for better ways to manage this moving forward. * fix(provider/kubernetes): allow empty serviceName in statefulset (#2261) * chore(provider/kubernetes): log malformed cache entries (#2262) * feat(provider/kubernetes): v2 ignore caching flag (#2268) * feat(provider/kubernetes): return resource create time (#2269) * feat(provider/kubernetes): fail deploy if artifacts don't bind (#2271) * feat(authz): requiredGroupMemberships to permissions migration (#2272) update AllowedAccountsValidator to use permissions config in addition to requiredGroupMemberships * fix(provider/kubernetes): Replace artifact only if target found (#2259) Do not attempt artifact replacement if no target was found. Reading a non-existing path can return an empty ArrayNode object, in which case we should not attempt any replacement. Add checks for bound artifacts in artifact replacement tests. * feat(provider/kubernetes): split out required & optional artifacts (#2273) * fix(provider/docker): update bearer_token to access_token (#2274) * fix(provider/kubernetes): Fixed statefulset and daemonset volumesource bugs. (#2276) feat(provider/kubernetes): split out required & optional artifacts (#2273) fix(provider/docker): update bearer_token to access_token (#2274) * fix(provider/kubernetes): annotate artifacts with reference (#2277) * feat(provider/kubernetes): only redeploy versioned artifacts on changes (#2278) * fix(jobs): Fix bottleneck around JobLocalExecutor.startJob (#2280) The `startJob` method submits your request to a threadpool, which it immediately starts polling. This isn't useful, and limits the number of jobs that can be started to the number of threads supported by your scheduler. As a result, submitting a lot of jobs at once causes (needlessly) a lot of these jobs to timeout. There is still a `sleep(500)` which doesn't seem to add anything in the function, but didn't feel confident removing since other spots in clouddriver make use of it. * fix(provider/kubernetes): v2 Set default namespace dynamically (#2279) Set the default namespace according to the `kubectl` rules defined in https://github.com/kubernetes/kubernetes/blob/bd4d511a40e142ee65edff3b286e57de502aa790/pkg/kubectl/cmd/util/factory_client_access.go#L127 In short, the default namespace is set in this way: - If `/var/run/secrets/kubernetes.io/serviceaccount/namespace` exists, assume we are running in a Kubernetes container, and set the default namespace to the contents of this file. - Set the default namespace to the namespace of current `kubectl` config context, if it exists. `kubectl config view` is used to get this namespace, which takes care of the complicated merges and overrides. - Otherwise, set the default namespace to "default". This fixes the issue of manifests without a namespace specification being deployed in the namespace of clouddriver when Spinnaker has been deployed in Kubernetes. Previously manifests without a namespace were assumed to belong to the "default" namespace, but clouddriver would deploy them in the "spinnaker" namespace, where it is deployed by default by Halyard. Deploying such manifests would cause Spinnaker to get stuck waiting for a deployment to show up in the "default" namespace, since clouddriver would instead have created a deployment in the "spinnaker" namespace. * Added EcsCloudMetricAlarmCaching - Agent/Client/Tests. (#2263) * Added TaskHealthCaching - Agent/Client/Tests. (#2260) * feat(provider/ecs): ECS Loadbalancer cache client and tests. (#2264) * Added EcsLoadbalancerCacheClient with a cache model and test. * Object mapper configuration moved to declaration line. * Clean up. * feat(provider/ecs): ECS Target Group cache client and tests. (#2265) * Added EcsTargetGroupCacheClient with a model and tests. * Changes based on feedback. * feat(artifacts): add artifact download endpoint (#2282) * refactor(provider/kubernetes): remove obsolete source check (#2283) * fix(core): Allow for versioned providers in lb controller (#2284) * perf(provider/gce): De-dupe getHealth() calls in LB caching agents. (#2281) * fix(provider/kubernetes): load replicas for non replicaset server groups (#2287) * feat(provider/kubernetes): look up namer by class name (#2288) * fix(core): Support an empty NamerRegistry (#2297) * poc(provider/aws): Index a subset of server group / instance details (#2204) Mo bettah search! Specifically this PR introduces a brand new agent that fetches all server groups and instances in each aws account / region. The current breakdown of caching agents (per account and per region) are potentially too granular to create useful elastic search indexes off of. * fix(provider/kubernetes): prevent NPE if resource does not have creation timestamp (#2299) * fix(provider/kubernetes): fix cast from double in replica count (#2298) * fix(provider/aws): Handle ApplicationLoadBalancers with invalid actions (#2300) * feat(provider/kubernetes): allow artifact types to be extended (#2302) * fix(provider/kubernetes): v2 cache clusters for workloads only (#2303) This ensures that only workload based resources are associated with a spinnaker cluster, reducing 'application' screen clutter * feat(provider/openstack): Add support for availability zones (#2304) * fix(provider/kubernetes): only include app & cluster frigga details (#2306) * feat(provider/ecs): Ecs instance cache client (#2289) * Added EC2 Instance cache client. * Removed unneeded import. * Removed @Qualifier from cache client constructors, and updated existing ones. * feat(provider/ecs): ECS Scalable Targets caching classes and tests (#2290) * Added ScalableTarget cache - caching agent and client. * Removed @Qualifier. * Added EcsClusterProvider. (#2291) * Added EcsRoleProvider. (#2292) * Added EcsApplicationProvider, EcsApplication model, and TestCredential.. (#2293) * Added EcsCloudMetricProvider and EcsCloudMetricController. (#2294) * poc(provider/aws): Index a subset of instance details (#2204) (#2308) Support searching for instances by ip address (and a handful of other attributes). Specifically this PR introduces a brand new agent that fetches all instances in each aws account / region. * feat(provider/kubernetes) - adds configMap replacer support for replicasets (#2310) * fix(provider/kubernetes): v2 Make workloads authoritative for apps (#2307) Let workload deployments have full control over (generated) applications coming and going. This is in line with https://github.com/spinnaker/clouddriver/pull/2303 and fixes https://github.com/spinnaker/spinnaker/issues/2259 In particular, applications will not be shown in the deck "Applications" view if the replicaset/daemonset/deployment/statefulset of the application has the `caching.spinnaker.io/ignore="true"` annotation, and will be removed from the list if the annotation added to an existing workload. * refactor(provider/appengine): artifact logic from orca (#2305) Moves responsibility for artifact hydration during appengine deploys from orca into clouddriver. * fix(provider/kubernetes) : disable v1 controller caching #2239 (#2301) * fix(provider/kubernetes) : disable v1 controller caching #2239 * fix(provider/kubernetes) : disable v1 controller caching #2239) * feat(provider/kubernetes): v1 volume mount subpath support (#2316) * fix(provider/kubernetes): v1 fix red/black for svgs without lbs (#2317) * feat(provider/kubernetes): default artifact replacers (#2318) * fix(artifacts): improve github artifact downloader error handling (#2319) * fix(provider/gce): Fix typo in metric tag. (#2320) * fix(provider/kubernetes): v1 eventual consistency bug in disable (#2321) It turns out it was possible that we would 1. apply an annotation 2. toggle replica set labels 3. fail to wait for the label change to be applied Part 3. could happen because the generation of the replica set returned from the "toggle label" operation was that of the "apply annotation" operation. The "wait for consistency" step we usually do here was then skipped causing us to edit pod labels before the replica set owning them was finised updating. This doesn't happen often, but enough to be a real problem. * fix(gcs): Prevent NPE when gcs artifact account has no credentials path (#2324) * Corrected capitalization for cloudMetrics endpoint. (#2312) * feat(provider/ecs): ECR Image Provider (#2309) * Added EcrImageProvider. * Moved EcrImageProvider to the correct package. * Made a filtering method. * Error switched to IllegalArgumentException. Grammar fixed. * Updated tests. * feat(provider/ecs): ECS Cluster Controller (#2313) * Added EcsClusterController. * Camel-cased the endpoint. * fix(provider/amazon): propagate serverGroupNamesByRegion result on copyLastAsg (#2270) * Added ContainerInformationService. (#2314) * fix(provider/ecs): ECS Caching Agents - Account/Region awareness (#2315) * Caching agents are now account and region aware. * getCluster account aware. * Formatting fixed. * EcsCredentialsInitializer now initializes NetflixAssumeRoleEcsCredentials. (#2323) * fix(rrb): Disable percentage calculation should use Math.ceil() (#2327) This aligns the calculation with what is also happening within `orca`. * fix(provider/kubernetes): v1 oom killed job has exit code 0 (#2328) * Added EcsImagesController. (#2329) * feat(provider/ecs): ECS Instance Provider (#2330) * Added EcsInstanceProvider. Updated ContainerInstance and the caching client to include the availability zone. * Corrected variable names. * Added EcsCloudMetricService. (#2331) * Added EcsServerClusterProvider. (#2332) * feature(provider/openstack) implementing senlin zone policy and scheduler hints (#2325) * feat(credentials): add expand parameter to /accounts endpoint (#2326) * fix(entitytags): Increase default 'maxResults' from 100 -> 2000 (#2335) * feat(provider/ecs): ECS Load Balancer Provider (#2333) * Added EcsLoadBalancerProvider. * Feedback changes. * Changed x's to it's. * feat(provider/ecs): Destroy Service Atomic Operation (#2334) * Added DestroyServiceAtomicOperation along with base deploy classes. * Removed invalid import. Formatting fixed. * Service changed to server group in the output messages. * Copyright year updated. * feat(provider/kubernetes): make kubectl configurable per-account (#2337) * fix(provider/appengine): dont prepend "gs" to all repository URLs (#2346) * fix(provider/kubernetes): v2 fix caching of malformed annotations (#2340) * fix(provider/kubernetes): v2 gracefully handle unknown kind/version (#2342) * fix(provider/kubernetes): v2 reduce noise when polling older clusters (#2344) * feat(provider/kubernetes): v2 hpa support (#2347) * Added TerminateInstancesAtomicOperation. (#2339) * Added CreateServerGroup AO. (#2338) * fix(*): Force using hashtags for dynomite connections (#2348) * Added ResizeServiceAtomicOperation. (#2341) * feat(provider/ecs): Enable Service Atomic Operation (#2343) * Added EnableServiceAtomicOperation. * Deleted an unnecessary description. * feat(provider/ecs): Disable Service Atomic Operation (#2345) * Added DisableServiceAtomicOperation. * Deleted an unnecessary validator. * refactor(provider/ecs): Caching agents enabled, code cleanup. (#2349) * EcsProviderConfig now adds all of the caching agents. `expand` parameter being used properly in `EcsApplicationProvider`. Added missing copyrights. Added missing @Component annotation. Minor syntax changes. * Lombok dependency added in clouddriver-ecs.gradle. * bug(provider/openstack) - cleanly handle errors when caching OST server groups. (#2350) * fix(provider/kubernetes): concurrent modification of kind list (#2353) Seems the iterator wasn't closed * fix(provider/kubernetes): Add top level `kind` to make Deck happy (#2355) * feat(provider/kubernetes): v2 Extract artifact name from reference (#2322) Allow for for using a regex to set the artifact name after finding artifacts in a manifest. This is used to add support in orca for https://github.com/spinnaker/spinnaker/issues/2267 Currently only done for Docker images, but should be possible for configmaps and secrets as well. * Missing files. * fix(provider/appengine): surface timeouts to frontend during job execution (#2356) * Added missing file back. Deleted a duplicate method. --- build.gradle | 6 +- .../appengine/AppengineJobExecutor.groovy | 5 +- ...oyAppengineAtomicOperationConverter.groovy | 28 +- .../DeployAppengineDescription.groovy | 2 + ...ngineDescriptionConversionException.groovy | 22 + .../ops/DeployAppengineAtomicOperation.groovy | 28 +- ...DeployAppengineDescriptionValidator.groovy | 22 +- ...pengineAtomicOperationConverterSpec.groovy | 22 + .../artifacts/gcs/GcsArtifactCredentials.java | 3 +- .../github/GitHubArtifactCredentials.java | 43 +- .../ops/CopyLastAsgAtomicOperation.groovy | 4 + ...ApplicationLoadBalancerCachingAgent.groovy | 8 + .../CopyLastAsgAtomicOperationUnitSpec.groovy | 9 +- .../cache/DynomiteCacheConfig.groovy | 12 +- .../clouddriver/core/CloudDriverConfig.groovy | 8 + .../clouddriver/core/DynomiteConfig.groovy | 2 +- .../EnableDisablePercentageCategorizer.groovy | 2 +- .../jobs/local/JobExecutorLocal.groovy | 2 +- .../clouddriver/names/NamerRegistry.java | 13 + .../clouddriver/names/NamingStrategy.java | 7 + ...bleDisablePercentageCategorizerSpec.groovy | 44 +- .../clouddriver/ecs/EcsCloudProvider.java | 2 +- .../clouddriver/ecs/EcsConfiguration.java | 2 +- .../ecs/EcsConfigurationProperties.java | 2 +- .../clouddriver/ecs/EcsOperation.java | 2 +- .../spinnaker/clouddriver/ecs/cache/Keys.java | 46 +- .../client/ScalableTargetCacheClient.java | 9 +- .../ecs/cache/client/ServiceCacheClient.java | 11 +- .../ecs/cache/client/TaskCacheClient.java | 9 +- .../client/TaskDefinitionCacheClient.java | 9 +- .../controllers/EcsCloudMetricController.java | 2 +- .../ecs/controllers/EcsClusterController.java | 4 +- ...estroyServiceAtomicOperationConverter.java | 2 +- ...isableServiceAtomicOperationConverter.java | 2 +- ...teServerGroupAtomicOperationConverter.java | 2 +- ...EnableServiceAtomicOperationConverter.java | 2 +- ...ResizeServiceAtomicOperationConverter.java | 2 +- ...nateInstancesAtomicOperationConverter.java | 4 +- .../description/AbstractECSDescription.java | 2 +- .../CreateServerGroupDescription.java | 2 +- .../description/ModifyServiceDescription.java | 2 +- .../description/ResizeServiceDescription.java | 2 +- .../TerminateInstancesDescription.java | 2 +- .../ops/AbstractEcsAtomicOperation.java | 12 +- .../ops/CreateServerGroupAtomicOperation.java | 3 +- .../ops/DestroyServiceAtomicOperation.java | 10 +- .../ops/DisableServiceAtomicOperation.java | 6 +- .../ops/EnableServiceAtomicOperation.java | 6 +- .../ops/ResizeServiceAtomicOperation.java | 7 +- .../TerminateInstancesAtomicOperation.java | 3 +- .../deploy/validators/CommonValidator.java | 34 +- ...estroyServiceAtomicOperationValidator.java | 4 +- .../DisableServiceDescriptionValidator.java | 4 +- ...CreateServerGroupDescriptionValidator.java | 2 +- .../EnableServiceDescriptionValidator.java | 4 +- .../ResizeServiceDescriptionValidator.java | 4 +- .../ServerGroupDescriptionValidator.java | 2 +- ...erminateInstancesDescriptionValidator.java | 9 +- .../loadbalancer/EcsLoadBalancerSummary.java | 2 +- .../EcsLoadBalancerSummaryByRegion.java | 6 +- .../clouddriver/ecs/provider/EcsProvider.java | 4 +- .../agent/AbstractEcsCachingAgent.java | 25 +- .../agent/ContainerInstanceCachingAgent.java | 24 +- .../EcsCloudMetricAlarmCachingAgent.java | 2 +- .../ecs/provider/agent/IamPolicyReader.java | 4 +- .../provider/agent/IamRoleCachingAgent.java | 5 +- .../ecs/provider/agent/TaskCachingAgent.java | 36 +- .../ecs/provider/view/EcrImageProvider.java | 20 +- .../ecs/provider/view/EcsClusterProvider.java | 3 - .../view/EcsLoadBalancerProvider.java | 8 +- .../ecs/services/EcsCloudMetricService.java | 8 +- .../ecs/view/EcsInstanceProvider.java | 8 +- .../cache/EcsInstanceCacheClientSpec.groovy | 2 +- .../EcsCloudMetricControllerSpec.groovy | 1 - ...ServiceAtomicOperationConverterSpec.groovy | 2 +- ...ServiceAtomicOperationConverterSpec.groovy | 2 +- ...erGroupAtomicOperationConverterSpec.groovy | 2 +- ...ServiceAtomicOperationConverterSpec.groovy | 2 +- ...ServiceAtomicOperationConverterSpec.groovy | 5 +- ...ServiceAtomicOperationConverterSpec.groovy | 2 +- ...nstanceAtomicOperationConverterSpec.groovy | 2 +- .../deploy/ops/CommonAtomicOperation.groovy | 4 +- ...reateServerGroupAtomicOperationSpec.groovy | 2 +- .../DestroyServiceAtomicOperationSpec.groovy | 3 +- .../DisableServiceAtomicOperationSpec.groovy | 2 +- .../EnableServiceAtomicOperationSpec.groovy | 2 +- .../ResizeServiceAtomicOperationSpec.groovy | 2 +- ...erminateInstanceAtomicOperationSpec.groovy | 3 +- .../validators/AbstractValidatorSpec.groovy | 13 +- ...ServergroupDescriptionValidatorSpec.groovy | 2 +- .../ResizeDescriptionValidatorSpec.groovy | 3 +- ...ServerGroupDescriptionValidatorSpec.groovy | 5 +- ...ateInstanceDescriptionValidatorSpec.groovy | 5 +- ...EcsCloudMetricAlarmCachingAgentSpec.groovy | 1 - .../provider/view/EcrImageProviderSpec.groovy | 10 +- .../view/EcsClusterProviderSpec.groovy | 2 +- .../view/EcsLoadBalancerProviderSpec.groovy | 16 +- .../view/EcsServerClusterProviderSpec.groovy | 17 +- .../provider/agent/CommonCachingAgent.java | 25 +- .../ecs/provider/agent/IamRoleCacheTest.java | 2 + .../ecs/provider/agent/ServiceCacheTest.java | 3 + .../agent/ServiceCachingAgentTest.java | 1 + .../ecs/provider/agent/TaskCacheTest.java | 2 + .../agent/TaskDefinitionCacheTest.java | 8 +- .../agent/TaskDefinitionCachingAgentTest.java | 5 +- .../clouddriver-elasticsearch-aws.gradle | 10 + .../elasticsearch/ElasticSearchClient.kt | 105 +++++ ...ElasticSearchAmazonCachingAgentProvider.kt | 73 +++ ...ElasticSearchAmazonInstanceCachingAgent.kt | 130 ++++++ ...sticSearchAmazonServerGroupCachingAgent.kt | 173 +++++++ .../model/ElasticSearchModels.kt | 59 +++ .../config/ElasticSearchAmazonConfig.kt | 48 ++ .../clouddriver/google/deploy/GCEUtil.groovy | 4 +- .../KubernetesConfigurationProperties.groovy | 3 +- .../KubernetesNamedAccountCredentials.java | 26 +- ...sNamedAccountCredentialsInitializer.groovy | 3 + .../v1/api/KubernetesApiAdaptor.groovy | 16 + .../v1/api/KubernetesApiConverter.groovy | 8 +- .../api/KubernetesClientApiConverter.groovy | 8 +- .../v1/deploy/KubernetesUtil.groovy | 2 + ...ubernetesAtomicOperationDescription.groovy | 1 + ...bleDisableKubernetesAtomicOperation.groovy | 16 +- .../v1/model/KubernetesJobStatus.groovy | 5 + .../v1/model/KubernetesV1ServerGroup.groovy | 7 +- .../KubernetesControllersCachingAgent.groovy | 424 ++++++++++++++++++ .../KubernetesServerGroupCachingAgent.groovy | 22 +- .../KubernetesV1CachingAgentDispatcher.groovy | 1 + .../v2/artifact/ArtifactReplacer.java | 18 + .../v2/artifact/ArtifactReplacerFactory.java | 49 ++ .../kubernetes/v2/artifact/ArtifactTypes.java | 9 +- .../agent/KubernetesCacheDataConverter.java | 44 +- .../KubernetesDaemonSetCachingAgent.java | 7 +- .../KubernetesDeploymentCachingAgent.java | 7 +- ...esHorizontalPodAutoscalerCachingAgent.java | 59 +++ .../agent/KubernetesPodCachingAgent.java | 7 +- .../KubernetesReplicaSetCachingAgent.java | 7 +- .../KubernetesStatefulSetCachingAgent.java | 7 +- .../agent/KubernetesV2CachingAgent.java | 15 +- .../KubernetesV2OnDemandCachingAgent.java | 9 +- .../view/model/KubernetesV2ServerGroup.java | 16 +- .../view/model/ManifestBasedModel.java | 5 + .../KubernetesV2ApplicationProvider.java | 4 + .../KubernetesV2ManifestProvider.java | 6 +- .../manifest/KubernetesApiVersion.java | 12 +- .../description/manifest/KubernetesKind.java | 13 +- .../manifest/KubernetesManifest.java | 4 +- .../manifest/KubernetesManifestAnnotater.java | 11 +- .../v2/names/KubernetesManifestNamer.java | 11 +- .../deployer/KubernetesDeploymentHandler.java | 19 +- ...ernetesHorizontalPodAutoscalerHandler.java | 54 +++ .../deployer/KubernetesReplicaSetHandler.java | 19 +- .../v2/op/job/KubectlJobExecutor.java | 22 +- .../KubernetesDeployManifestOperation.java | 2 - .../v2/security/KubernetesV2Credentials.java | 14 +- .../KubernetesCacheDataConvertSpec.groovy | 55 +-- .../client/OpenstackComputeProvider.groovy | 8 + .../client/OpenstackComputeV2Provider.groovy | 10 +- .../servergroup/ServerGroupParameters.groovy | 9 +- .../deploy/ops/StackPoolMemberAware.groovy | 7 +- ...StackUpdateOpenstackAtomicOperation.groovy | 47 +- .../CloneOpenstackAtomicOperation.groovy | 1 - .../DeployOpenstackAtomicOperation.groovy | 94 +++- .../servergroup/ServerGroupConstants.groovy | 6 +- .../OpenstackServerGroupCachingAgent.groovy | 130 +++--- .../OpenstackCredentialsInitializer.groovy | 31 +- .../OpenstackNamedAccountCredentials.groovy | 11 +- .../src/main/resources/servergroup.yaml | 18 +- .../main/resources/servergroup_resource.yaml | 5 + .../main/resources/servergroup_server.yaml | 5 + ...ckOrchestrationV1ClientProviderSpec.groovy | 12 +- .../ServerGroupParametersSpec.groovy | 53 ++- .../CloneOpenstackAtomicOperationSpec.groovy | 3 - .../DeployOpenstackAtomicOperationSpec.groovy | 76 +++- ...penstackNamedAccountCredentialsSpec.groovy | 31 ++ .../deploy/ops/servergroup/servergroup.yaml | 184 ++++++++ .../ops/servergroup}/servergroup_float.yaml | 18 +- .../ops/servergroup/servergroup_resource.yaml | 56 +++ .../servergroup_resource_float.yaml | 5 + .../ops/servergroup/servergroup_server.yaml | 46 ++ .../servergroup_server_float.yaml | 5 + clouddriver-web/clouddriver-web.gradle | 1 + .../controllers/CredentialsController.groovy | 39 +- .../controllers/EntityTagsController.java | 2 +- gradle.properties | 0 .../kotlin.gradle | 32 +- .../spek.gradle | 31 +- gradle/wrapper/gradle-wrapper.properties | 4 +- settings.gradle | 1 + 188 files changed, 2745 insertions(+), 722 deletions(-) create mode 100644 clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/exception/AppengineDescriptionConversionException.groovy create mode 100644 clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamingStrategy.java create mode 100644 clouddriver-elasticsearch-aws/clouddriver-elasticsearch-aws.gradle create mode 100644 clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/ElasticSearchClient.kt create mode 100644 clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonCachingAgentProvider.kt create mode 100644 clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonInstanceCachingAgent.kt create mode 100644 clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonServerGroupCachingAgent.kt create mode 100644 clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/model/ElasticSearchModels.kt create mode 100644 clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/config/ElasticSearchAmazonConfig.kt create mode 100644 clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesControllersCachingAgent.groovy create mode 100644 clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacerFactory.java create mode 100644 clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesHorizontalPodAutoscalerCachingAgent.java create mode 100644 clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesHorizontalPodAutoscalerHandler.java create mode 100644 clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup.yaml rename clouddriver-openstack/src/{main/resources => test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup}/servergroup_float.yaml (93%) create mode 100644 clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource.yaml rename clouddriver-openstack/src/{main/resources => test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup}/servergroup_resource_float.yaml (91%) create mode 100644 clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server.yaml rename clouddriver-openstack/src/{main/resources => test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup}/servergroup_server_float.yaml (90%) delete mode 100644 gradle.properties rename clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~HEAD => gradle/kotlin.gradle (58%) rename clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~4441780d8c55c8e7e5dca9a1a26378fc72e0c299 => gradle/spek.gradle (61%) diff --git a/build.gradle b/build.gradle index 4ee21409968..327a1d4d8d3 100644 --- a/build.gradle +++ b/build.gradle @@ -17,6 +17,8 @@ buildscript { ext { springBootVersion = "1.5.7.RELEASE" + kotlinVersion = "1.2.0" + junitPlatformVersion = "1.0.2" } repositories { jcenter() @@ -26,6 +28,8 @@ buildscript { dependencies { classpath 'com.netflix.spinnaker.gradle:spinnaker-gradle-project:3.17.0' classpath "org.springframework.boot:spring-boot-gradle-plugin:${springBootVersion}" + classpath "org.junit.platform:junit-platform-gradle-plugin:${junitPlatformVersion}" + classpath "com.netflix.nebula:nebula-kotlin-plugin:${kotlinVersion}" } } @@ -35,7 +39,7 @@ allprojects { apply plugin: 'groovy' ext { - spinnakerDependenciesVersion = project.hasProperty('spinnakerDependenciesVersion') ? project.property('spinnakerDependenciesVersion') : '0.131.0' + spinnakerDependenciesVersion = project.hasProperty('spinnakerDependenciesVersion') ? project.property('spinnakerDependenciesVersion') : '0.134.0' } def checkLocalVersions = [spinnakerDependenciesVersion: spinnakerDependenciesVersion] diff --git a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/AppengineJobExecutor.groovy b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/AppengineJobExecutor.groovy index 50f241edf7b..3c47929aa39 100644 --- a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/AppengineJobExecutor.groovy +++ b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/AppengineJobExecutor.groovy @@ -41,10 +41,13 @@ class AppengineJobExecutor { void waitForJobCompletion(String jobId) { sleep(sleepMs) JobStatus jobStatus = jobExecutor.updateJob(jobId) - while (jobStatus.state == JobStatus.State.RUNNING) { + while (jobStatus != null && jobStatus.state == JobStatus.State.RUNNING) { sleep(sleepMs) jobStatus = jobExecutor.updateJob(jobId) } + if (jobStatus == null) { + throw new RuntimeException("job timed out or was cancelled") + } if (jobStatus.result == JobStatus.Result.FAILURE && jobStatus.stdOut) { throw new IllegalArgumentException("$jobStatus.stdOut + $jobStatus.stdErr") } diff --git a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/converters/DeployAppengineAtomicOperationConverter.groovy b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/converters/DeployAppengineAtomicOperationConverter.groovy index 86452c20338..456586df6d0 100644 --- a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/converters/DeployAppengineAtomicOperationConverter.groovy +++ b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/converters/DeployAppengineAtomicOperationConverter.groovy @@ -16,22 +16,48 @@ package com.netflix.spinnaker.clouddriver.appengine.deploy.converters +import com.fasterxml.jackson.databind.ObjectMapper import com.netflix.spinnaker.clouddriver.appengine.AppengineOperation import com.netflix.spinnaker.clouddriver.appengine.deploy.description.DeployAppengineDescription +import com.netflix.spinnaker.clouddriver.appengine.deploy.exception.AppengineDescriptionConversionException import com.netflix.spinnaker.clouddriver.appengine.deploy.ops.DeployAppengineAtomicOperation import com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation import com.netflix.spinnaker.clouddriver.orchestration.AtomicOperations import com.netflix.spinnaker.clouddriver.security.AbstractAtomicOperationsCredentialsSupport +import com.netflix.spinnaker.kork.artifacts.model.Artifact +import org.springframework.beans.factory.annotation.Autowired import org.springframework.stereotype.Component @AppengineOperation(AtomicOperations.CREATE_SERVER_GROUP) @Component class DeployAppengineAtomicOperationConverter extends AbstractAtomicOperationsCredentialsSupport { + @Autowired + ObjectMapper objectMapper + AtomicOperation convertOperation(Map input) { new DeployAppengineAtomicOperation(convertDescription(input)) } DeployAppengineDescription convertDescription(Map input) { - AppengineAtomicOperationConverterHelper.convertDescription(input, this, DeployAppengineDescription) + DeployAppengineDescription description = AppengineAtomicOperationConverterHelper.convertDescription(input, this, DeployAppengineDescription) + + if (input.artifact) { + description.artifact = objectMapper.convertValue(input.artifact, Artifact) + switch (description.artifact.type) { + case 'gcs/object': + description.repositoryUrl = description.artifact.reference; + if (!description.repositoryUrl.startsWith('gs://')) { + description.repositoryUrl = "gs://${description.repositoryUrl}" + } + break + case 'docker/image': + description.containerImageUrl = description.artifact.name; + break + default: + throw new AppengineDescriptionConversionException("Invalid artifact type for Appengine deploy: ${description.artifact.type}") + } + } + + return description } } diff --git a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/description/DeployAppengineDescription.groovy b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/description/DeployAppengineDescription.groovy index 572629739ce..707c06355fe 100644 --- a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/description/DeployAppengineDescription.groovy +++ b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/description/DeployAppengineDescription.groovy @@ -18,12 +18,14 @@ package com.netflix.spinnaker.clouddriver.appengine.deploy.description import com.netflix.spinnaker.clouddriver.appengine.gitClient.AppengineGitCredentialType import com.netflix.spinnaker.clouddriver.deploy.DeployDescription +import com.netflix.spinnaker.kork.artifacts.model.Artifact import groovy.transform.AutoClone import groovy.transform.Canonical @AutoClone @Canonical class DeployAppengineDescription extends AbstractAppengineCredentialsDescription implements DeployDescription { + Artifact artifact String accountName String application String stack diff --git a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/exception/AppengineDescriptionConversionException.groovy b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/exception/AppengineDescriptionConversionException.groovy new file mode 100644 index 00000000000..c23b4b9e3a3 --- /dev/null +++ b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/exception/AppengineDescriptionConversionException.groovy @@ -0,0 +1,22 @@ +/* + * Copyright 2018 Google, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.appengine.deploy.exception + +import groovy.transform.InheritConstructors + +@InheritConstructors +class AppengineDescriptionConversionException extends RuntimeException { } diff --git a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/ops/DeployAppengineAtomicOperation.groovy b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/ops/DeployAppengineAtomicOperation.groovy index 269c244aa92..632e5a2d5b7 100644 --- a/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/ops/DeployAppengineAtomicOperation.groovy +++ b/clouddriver-appengine/src/main/groovy/com/netflix/spinnaker/clouddriver/appengine/deploy/ops/DeployAppengineAtomicOperation.groovy @@ -17,22 +17,19 @@ package com.netflix.spinnaker.clouddriver.appengine.deploy.ops import com.netflix.spinnaker.clouddriver.appengine.AppengineJobExecutor -import com.netflix.spinnaker.clouddriver.appengine.config.AppengineConfigurationProperties +import com.netflix.spinnaker.clouddriver.appengine.artifacts.GcsStorageService +import com.netflix.spinnaker.clouddriver.appengine.artifacts.config.StorageConfigurationProperties import com.netflix.spinnaker.clouddriver.appengine.deploy.AppengineMutexRepository import com.netflix.spinnaker.clouddriver.appengine.deploy.AppengineServerGroupNameResolver import com.netflix.spinnaker.clouddriver.appengine.deploy.description.DeployAppengineDescription import com.netflix.spinnaker.clouddriver.appengine.deploy.exception.AppengineOperationException import com.netflix.spinnaker.clouddriver.appengine.gcsClient.AppengineGcsRepositoryClient -import com.netflix.spinnaker.clouddriver.appengine.artifacts.GcsStorageService -import com.netflix.spinnaker.clouddriver.appengine.artifacts.config.StorageConfigurationProperties import com.netflix.spinnaker.clouddriver.data.task.Task import com.netflix.spinnaker.clouddriver.data.task.TaskRepository import com.netflix.spinnaker.clouddriver.deploy.DeploymentResult import com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation -import org.springframework.beans.factory.annotation.Autowired - import java.nio.file.Paths - +import org.springframework.beans.factory.annotation.Autowired import static com.netflix.spinnaker.clouddriver.appengine.config.AppengineConfigurationProperties.ManagedAccount.GcloudReleaseTrack class DeployAppengineAtomicOperation implements AtomicOperation { @@ -57,8 +54,23 @@ class DeployAppengineAtomicOperation implements AtomicOperation { def thisResult = basicAmazonDeployHandler.handle(newDescription, priorOutputs) result.serverGroupNames.addAll(thisResult.serverGroupNames) + result.deployedNames.addAll(thisResult.deployedNames) result.messages.addAll(thisResult.messages) + thisResult.serverGroupNameByRegion.entrySet().each { result.serverGroupNameByRegion[it.key] = it.value } + thisResult.deployedNamesByLocation.entrySet().each { result.deployedNamesByLocation[it.key] = it.value } + task.updateStatus BASE_PHASE, "Deployment complete in $targetRegion. New ASGs = ${result.serverGroupNames}" } task.updateStatus BASE_PHASE, "Finished copying last ASG for ${cluster}. New ASGs = ${result.serverGroupNames}." diff --git a/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/provider/agent/AmazonApplicationLoadBalancerCachingAgent.groovy b/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/provider/agent/AmazonApplicationLoadBalancerCachingAgent.groovy index b001b737283..c6a16391cc7 100644 --- a/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/provider/agent/AmazonApplicationLoadBalancerCachingAgent.groovy +++ b/clouddriver-aws/src/main/groovy/com/netflix/spinnaker/clouddriver/aws/provider/agent/AmazonApplicationLoadBalancerCachingAgent.groovy @@ -413,6 +413,10 @@ class AmazonApplicationLoadBalancerCachingAgent extends AbstractAmazonLoadBalanc listenerAttributes.loadBalancerName = ArnUtils.extractLoadBalancerName((String)listenerAttributes.loadBalancerArn).get() listenerAttributes.remove('loadBalancerArn') for (Map action : (List>)listenerAttributes.defaultActions) { + if (!action.targetGroupArn) { + continue + } + String targetGroupName = ArnUtils.extractTargetGroupName(action.targetGroupArn as String).get() action.targetGroupName = targetGroupName action.remove("targetGroupArn") @@ -425,6 +429,10 @@ class AmazonApplicationLoadBalancerCachingAgent extends AbstractAmazonLoadBalanc for (Rule rule : listenerAssociations.listenerToRules.get(listener)) { Map ruleAttributes = objectMapper.convertValue(rule, ATTRIBUTES) for (Map action : (List>)ruleAttributes.actions) { + if (!action.targetGroupArn) { + continue + } + String targetGroupName = ArnUtils.extractTargetGroupName(action.targetGroupArn).get() action.targetGroupName = targetGroupName action.remove("targetGroupArn") diff --git a/clouddriver-aws/src/test/groovy/com/netflix/spinnaker/clouddriver/aws/deploy/ops/CopyLastAsgAtomicOperationUnitSpec.groovy b/clouddriver-aws/src/test/groovy/com/netflix/spinnaker/clouddriver/aws/deploy/ops/CopyLastAsgAtomicOperationUnitSpec.groovy index 526a776d73f..e496ec5f60d 100644 --- a/clouddriver-aws/src/test/groovy/com/netflix/spinnaker/clouddriver/aws/deploy/ops/CopyLastAsgAtomicOperationUnitSpec.groovy +++ b/clouddriver-aws/src/test/groovy/com/netflix/spinnaker/clouddriver/aws/deploy/ops/CopyLastAsgAtomicOperationUnitSpec.groovy @@ -90,9 +90,12 @@ class CopyLastAsgAtomicOperationUnitSpec extends Specification { } when: - op.operate([]) + def result = op.operate([]) then: + result.serverGroupNameByRegion['us-east-1'] == 'asgard-stack-v001' + result.serverGroupNameByRegion['us-west-1'] == 'asgard-stack-v001' + result.serverGroupNames == ['asgard-stack-v001', 'asgard-stack-v001'] 2 * mockAutoScaling.describeLaunchConfigurations(_) >> { DescribeLaunchConfigurationsRequest request -> assert request.launchConfigurationNames == ['foo'] def mockLaunch = Mock(LaunchConfiguration) @@ -114,8 +117,8 @@ class CopyLastAsgAtomicOperationUnitSpec extends Specification { } 2 * serverGroupNameResolver.resolveLatestServerGroupName("asgard-stack") >> { "asgard-stack-v000" } 0 * serverGroupNameResolver._ - 1 * deployHandler.handle(expectedDeployDescription('us-east-1'), _) >> new DeploymentResult(serverGroupNameByRegion: ['us-east-1': 'asgard-stack-v001']) - 1 * deployHandler.handle(expectedDeployDescription('us-west-1'), _) >> new DeploymentResult(serverGroupNameByRegion: ['us-west-1': 'asgard-stack-v001']) + 1 * deployHandler.handle(expectedDeployDescription('us-east-1'), _) >> new DeploymentResult(serverGroupNames: ['asgard-stack-v001'], serverGroupNameByRegion: ['us-east-1': 'asgard-stack-v001']) + 1 * deployHandler.handle(expectedDeployDescription('us-west-1'), _) >> new DeploymentResult(serverGroupNames: ['asgard-stack-v001'], serverGroupNameByRegion: ['us-west-1': 'asgard-stack-v001']) where: requestSpotPrice | ancestorSpotPrice || expectedSpotPrice diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/cache/DynomiteCacheConfig.groovy b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/cache/DynomiteCacheConfig.groovy index a4cb18ef2b6..5a245488250 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/cache/DynomiteCacheConfig.groovy +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/cache/DynomiteCacheConfig.groovy @@ -65,7 +65,7 @@ class DynomiteCacheConfig { @Bean @ConfigurationProperties("dynomite.connectionPool") ConnectionPoolConfigurationImpl connectionPoolConfiguration(DynomiteConfigurationProperties dynomiteConfigurationProperties) { - new ConnectionPoolConfigurationImpl(dynomiteConfigurationProperties.applicationName) + new ConnectionPoolConfigurationImpl(dynomiteConfigurationProperties.applicationName).withHashtag("{}") } @Bean @@ -91,13 +91,13 @@ class DynomiteCacheConfig { .withCPConfig(connectionPoolConfiguration) }).orElseGet({ connectionPoolConfiguration - .withTokenSupplier(new StaticTokenMapSupplier(dynomiteConfigurationProperties.dynoHostTokens)) - .setLocalDataCenter(dynomiteConfigurationProperties.localDataCenter) - .setLocalRack(dynomiteConfigurationProperties.localRack) + .withTokenSupplier(new StaticTokenMapSupplier(dynomiteConfigurationProperties.dynoHostTokens)) + .setLocalDataCenter(dynomiteConfigurationProperties.localDataCenter) + .setLocalRack(dynomiteConfigurationProperties.localRack) builder - .withHostSupplier(new StaticHostSupplier(dynomiteConfigurationProperties.dynoHosts)) - .withCPConfig(connectionPoolConfiguration) + .withHostSupplier(new StaticHostSupplier(dynomiteConfigurationProperties.dynoHosts)) + .withCPConfig(connectionPoolConfiguration) }).build() } diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/CloudDriverConfig.groovy b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/CloudDriverConfig.groovy index 01ecdddfc91..b649b44c756 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/CloudDriverConfig.groovy +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/CloudDriverConfig.groovy @@ -56,6 +56,8 @@ import com.netflix.spinnaker.clouddriver.model.SecurityGroupProvider import com.netflix.spinnaker.clouddriver.model.ServerGroupManager import com.netflix.spinnaker.clouddriver.model.ServerGroupManagerProvider import com.netflix.spinnaker.clouddriver.model.SubnetProvider +import com.netflix.spinnaker.clouddriver.names.NamerRegistry +import com.netflix.spinnaker.clouddriver.names.NamingStrategy import com.netflix.spinnaker.clouddriver.orchestration.AtomicOperationConverter import com.netflix.spinnaker.clouddriver.search.ApplicationSearchProvider import com.netflix.spinnaker.clouddriver.search.NoopSearchProvider @@ -66,6 +68,7 @@ import com.netflix.spinnaker.clouddriver.security.AccountCredentialsRepository import com.netflix.spinnaker.clouddriver.security.DefaultAccountCredentialsProvider import com.netflix.spinnaker.clouddriver.security.MapBackedAccountCredentialsRepository import com.netflix.spinnaker.kork.core.RetrySupport +import com.netflix.spinnaker.moniker.Namer import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean import org.springframework.boot.context.properties.ConfigurationProperties @@ -256,4 +259,9 @@ class CloudDriverConfig { public RetrySupport retrySupport() { return new RetrySupport(); } + + @Bean + NamerRegistry namerRegistry(Optional> namingStrategies) { + new NamerRegistry(namingStrategies.orElse([])) + } } diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/DynomiteConfig.groovy b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/DynomiteConfig.groovy index 9fe5e763817..b2fe05aa10c 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/DynomiteConfig.groovy +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/core/DynomiteConfig.groovy @@ -56,7 +56,7 @@ class DynomiteConfig { @Bean @ConfigurationProperties("dynomite.connectionPool") ConnectionPoolConfigurationImpl connectionPoolConfiguration(DynomiteConfigurationProperties dynomiteConfigurationProperties) { - new ConnectionPoolConfigurationImpl(dynomiteConfigurationProperties.applicationName) + new ConnectionPoolConfigurationImpl(dynomiteConfigurationProperties.applicationName).withHashtag("{}") } @Bean(destroyMethod = "stopClient") diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizer.groovy b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizer.groovy index 8f8977038ad..58e81961dc6 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizer.groovy +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizer.groovy @@ -35,7 +35,7 @@ class EnableDisablePercentageCategorizer { } int totalSize = modified.size() + unmodified.size() - int newSize = (int) (totalSize * (float) (desiredPercentage / 100)) + int newSize = (int) Math.ceil(totalSize * (float) (desiredPercentage / 100)) int returnSize = modified.size() > newSize ? 0 : newSize - modified.size() diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/jobs/local/JobExecutorLocal.groovy b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/jobs/local/JobExecutorLocal.groovy index 2cb4ae2ff73..62618d7f242 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/jobs/local/JobExecutorLocal.groovy +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/jobs/local/JobExecutorLocal.groovy @@ -146,7 +146,7 @@ class JobExecutorLocal implements JobExecutor { jobStatus.stdErr = errors return jobStatus } else { - // This instance is not managing the job + // This instance is not managing the job, it has timed out, or it was cancelled. return null } } catch (Exception e) { diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamerRegistry.java b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamerRegistry.java index c6b6d5142e0..2c1ad51f67e 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamerRegistry.java +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamerRegistry.java @@ -21,6 +21,7 @@ import com.netflix.spinnaker.moniker.frigga.FriggaReflectiveNamer; import lombok.extern.slf4j.Slf4j; +import java.util.List; import java.util.concurrent.ConcurrentHashMap; /** @@ -31,6 +32,7 @@ * must happen within Spinnaker. */ public class NamerRegistry { + final private List namingStrategies; private static Namer defaultNamer = new FriggaReflectiveNamer(); private static ProviderLookup providerLookup = new ProviderLookup(); @@ -42,6 +44,17 @@ public static ProviderLookup lookup() { return providerLookup; } + public NamerRegistry(List namingStrategies) { + this.namingStrategies = namingStrategies; + } + + public Namer getNamingStrategy(String strategyName) { + return this.namingStrategies.stream() + .filter(strategy -> strategy.getName().equalsIgnoreCase(strategyName)) + .findFirst() + .orElseThrow(() -> new IllegalArgumentException("Could not find naming strategy '" + strategyName + "'")); + } + @Slf4j public static class ResourceLookup { private ConcurrentHashMap map = new ConcurrentHashMap<>(); diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamingStrategy.java b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamingStrategy.java new file mode 100644 index 00000000000..96a8562793e --- /dev/null +++ b/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/names/NamingStrategy.java @@ -0,0 +1,7 @@ +package com.netflix.spinnaker.clouddriver.names; + +import com.netflix.spinnaker.moniker.Namer; + +public interface NamingStrategy extends Namer { + String getName(); +} diff --git a/clouddriver-core/src/test/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizerSpec.groovy b/clouddriver-core/src/test/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizerSpec.groovy index a5ec8097cc8..713acbf8f8d 100644 --- a/clouddriver-core/src/test/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizerSpec.groovy +++ b/clouddriver-core/src/test/groovy/com/netflix/spinnaker/clouddriver/helpers/EnableDisablePercentageCategorizerSpec.groovy @@ -17,9 +17,11 @@ package com.netflix.spinnaker.clouddriver.helpers import spock.lang.Specification +import spock.lang.Unroll class EnableDisablePercentageCategorizerSpec extends Specification { + @Unroll void "should pick the right number of elements to move between lists"() { when: def output = EnableDisablePercentageCategorizer.getInstancesToModify(modified, unmodified, percent) @@ -31,24 +33,28 @@ class EnableDisablePercentageCategorizerSpec extends Specification { (output.size() + modified.size()) / (modified.size() + unmodified.size()) >= (float) percent / 1000 } + output.size() == expectedInstancesToDisableCount + where: - modified || unmodified || percent - [1] * 0 || [1] * 4 || 100 - [1] * 0 || [1] * 4 || 25 - [1] * 0 || [1] * 7 || 10 - - [1] * 4 || [1] * 0 || 100 - [1] * 3 || [1] * 0 || 10 - [1] * 9 || [1] * 0 || 60 - - [1] * 0 || [1] * 0 || 0 - [1] * 0 || [1] * 0 || 100 - [1] * 0 || [1] * 0 || 40 - - [1] * 9 || [1] * 9 || 60 - [1] * 7 || [1] * 3 || 10 - [1] * 4 || [1] * 4 || 100 - [1] * 5 || [1] * 10 || 0 - [1] * 9 || [1] * 10 || 90 -} + modified || unmodified || percent || expectedInstancesToDisableCount + [1] * 0 || [1] * 4 || 100 || 4 + [1] * 0 || [1] * 4 || 25 || 1 + [1] * 0 || [1] * 7 || 10 || 1 + + [1] * 4 || [1] * 0 || 100 || 0 + [1] * 3 || [1] * 0 || 10 || 0 + [1] * 9 || [1] * 0 || 60 || 0 + + [1] * 0 || [1] * 0 || 0 || 0 + [1] * 0 || [1] * 0 || 100 || 0 + [1] * 0 || [1] * 0 || 40 || 0 + + [1] * 9 || [1] * 9 || 60 || 2 + [1] * 7 || [1] * 3 || 10 || 0 + [1] * 4 || [1] * 4 || 100 || 4 + [1] * 5 || [1] * 10 || 0 || 0 + [1] * 9 || [1] * 10 || 90 || 9 + + [1] * 0 || [1] * 3 || 33 || 1 + } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsCloudProvider.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsCloudProvider.java index dc72c8b4275..cd6e225c4d0 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsCloudProvider.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsCloudProvider.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfiguration.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfiguration.java index 8203c441225..f7f50336104 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfiguration.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfiguration.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfigurationProperties.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfigurationProperties.java index 0f332f49120..d2f2c898d69 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfigurationProperties.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsConfigurationProperties.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsOperation.java index 81b1889186f..71713cb986f 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/EcsOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/Keys.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/Keys.java index 51fa76f82e8..c275e4164a4 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/Keys.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/Keys.java @@ -49,22 +49,7 @@ public String toString() { public static final String SEPARATOR = ";"; - @Override - public String getCloudProvider() { - return ID; - } - - @Override - public Map parseKey(String key) { - return parse(key); - } - - @Override - public Boolean canParseType(String type) { - return canParse(type); - } - - private static Boolean canParse(String type){ + private static Boolean canParse(String type) { for (Namespace key : Namespace.values()) { if (key.toString().equals(type)) { return true; @@ -85,7 +70,7 @@ public static Map parse(String key) { result.put("type", parts[1]); result.put("account", parts[2]); - if(!canParse(parts[1]) && parts[1].equals(HEALTH.getNs())){ + if (!canParse(parts[1]) && parts[1].equals(HEALTH.getNs())) { result.put("region", parts[3]); result.put("taskId", parts[4]); return result; @@ -130,11 +115,6 @@ public static Map parse(String key) { return result; } - @Override - public Boolean canParseField(String type) { - return false; - } - public static String getServiceKey(String account, String region, String serviceName) { return buildKey(Namespace.SERVICES.ns, account, region, serviceName); } @@ -171,7 +151,27 @@ public static String getIamRoleKey(String account, String iamRoleName) { return ID + SEPARATOR + Namespace.IAM_ROLE + SEPARATOR + account + SEPARATOR + iamRoleName; } - private static String buildKey(String namespace,String account, String region, String identifier){ + private static String buildKey(String namespace, String account, String region, String identifier) { return ID + SEPARATOR + namespace + SEPARATOR + account + SEPARATOR + region + SEPARATOR + identifier; } + + @Override + public String getCloudProvider() { + return ID; + } + + @Override + public Map parseKey(String key) { + return parse(key); + } + + @Override + public Boolean canParseType(String type) { + return canParse(type); + } + + @Override + public Boolean canParseField(String type) { + return false; + } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ScalableTargetCacheClient.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ScalableTargetCacheClient.java index 6ed09efe070..6e8f59ca67e 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ScalableTargetCacheClient.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ScalableTargetCacheClient.java @@ -21,25 +21,24 @@ import com.netflix.spinnaker.cats.cache.Cache; import com.netflix.spinnaker.cats.cache.CacheData; import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.stereotype.Component; import static com.netflix.spinnaker.clouddriver.ecs.cache.Keys.Namespace.SCALABLE_TARGETS; @Component public class ScalableTargetCacheClient extends AbstractCacheClient { - private final ObjectMapper mapper; + private final ObjectMapper objectMapper; @Autowired - public ScalableTargetCacheClient(Cache cacheView, @Qualifier("objectMapper") ObjectMapper mapper) { + public ScalableTargetCacheClient(Cache cacheView, ObjectMapper objectMapper) { super(cacheView, SCALABLE_TARGETS.toString()); - this.mapper = mapper; + this.objectMapper = objectMapper; } @Override protected ScalableTarget convert(CacheData cacheData) { ScalableTarget scalableTarget; - scalableTarget = mapper.convertValue(cacheData.getAttributes(), ScalableTarget.class); + scalableTarget = objectMapper.convertValue(cacheData.getAttributes(), ScalableTarget.class); return scalableTarget; } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ServiceCacheClient.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ServiceCacheClient.java index bc9c609a3b9..e879ac7eba8 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ServiceCacheClient.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/ServiceCacheClient.java @@ -22,7 +22,6 @@ import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.model.Service; import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.stereotype.Component; import java.util.ArrayList; @@ -34,12 +33,12 @@ @Component public class ServiceCacheClient extends AbstractCacheClient { - private ObjectMapper mapper; + private ObjectMapper objectMapper; @Autowired - public ServiceCacheClient(Cache cacheView, @Qualifier("objectMapper") ObjectMapper mapper) { + public ServiceCacheClient(Cache cacheView, ObjectMapper objectMapper) { super(cacheView, SERVICES.toString()); - this.mapper = mapper; + this.objectMapper = objectMapper; } @Override @@ -65,8 +64,8 @@ protected Service convert(CacheData cacheData) { List deserializedLoadbalancers = new ArrayList<>(loadBalancers.size()); for (Map serializedLoadbalancer : loadBalancers) { - if(serializedLoadbalancer!=null) { - deserializedLoadbalancers.add(mapper.convertValue(serializedLoadbalancer, LoadBalancer.class)); + if (serializedLoadbalancer != null) { + deserializedLoadbalancers.add(objectMapper.convertValue(serializedLoadbalancer, LoadBalancer.class)); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskCacheClient.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskCacheClient.java index f07d5022f67..b02c1da5307 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskCacheClient.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskCacheClient.java @@ -22,7 +22,6 @@ import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.model.Task; import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.stereotype.Component; import java.util.ArrayList; @@ -34,12 +33,12 @@ @Component public class TaskCacheClient extends AbstractCacheClient { - private ObjectMapper mapper; + private ObjectMapper objectMapper; @Autowired - public TaskCacheClient(Cache cacheView, @Qualifier("objectMapper") ObjectMapper mapper) { + public TaskCacheClient(Cache cacheView, ObjectMapper objectMapper) { super(cacheView, TASKS.toString()); - this.mapper = mapper; + this.objectMapper = objectMapper; } @Override @@ -61,7 +60,7 @@ protected Task convert(CacheData cacheData) { for (Map serializedContainer : containers) { if (serializedContainer != null) { - deserializedLoadbalancers.add(mapper.convertValue(serializedContainer, Container.class)); + deserializedLoadbalancers.add(objectMapper.convertValue(serializedContainer, Container.class)); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskDefinitionCacheClient.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskDefinitionCacheClient.java index ce20ef2bcf7..5c775bcce81 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskDefinitionCacheClient.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/cache/client/TaskDefinitionCacheClient.java @@ -22,7 +22,6 @@ import com.netflix.spinnaker.cats.cache.Cache; import com.netflix.spinnaker.cats.cache.CacheData; import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.stereotype.Component; import java.util.ArrayList; @@ -34,12 +33,12 @@ @Component public class TaskDefinitionCacheClient extends AbstractCacheClient { - private ObjectMapper mapper; + private ObjectMapper objectMapper; @Autowired - public TaskDefinitionCacheClient(Cache cacheView, @Qualifier("objectMapper") ObjectMapper mapper) { + public TaskDefinitionCacheClient(Cache cacheView, ObjectMapper objectMapper) { super(cacheView, TASK_DEFINITIONS.toString()); - this.mapper = mapper; + this.objectMapper = objectMapper; } @Override @@ -56,7 +55,7 @@ protected TaskDefinition convert(CacheData cacheData) { for (Map serializedContainerDefinitions : containerDefinitions) { if (serializedContainerDefinitions != null) { - deserializedContainerDefinitions.add(mapper.convertValue(serializedContainerDefinitions, ContainerDefinition.class)); + deserializedContainerDefinitions.add(objectMapper.convertValue(serializedContainerDefinitions, ContainerDefinition.class)); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricController.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricController.java index cec52c2cbda..96833392e80 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricController.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricController.java @@ -25,7 +25,7 @@ import java.util.Collection; @RestController -@RequestMapping("/ecs/cloudmetrics") +@RequestMapping("/ecs/cloudMetrics") public class EcsCloudMetricController { private final EcsCloudMetricProvider provider; diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsClusterController.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsClusterController.java index c8181cf5917..5eeca9d9a47 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsClusterController.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsClusterController.java @@ -19,12 +19,10 @@ import com.netflix.spinnaker.clouddriver.ecs.cache.model.EcsCluster; import com.netflix.spinnaker.clouddriver.ecs.provider.view.EcsClusterProvider; import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RestController; import java.util.Collection; -import java.util.List; @RestController public class EcsClusterController { @@ -37,7 +35,7 @@ public EcsClusterController(EcsClusterProvider ecsClusterProvider) { } - @RequestMapping(value = {"/ecs/ecsclusters"}) + @RequestMapping(value = {"/ecs/ecsClusters"}) public Collection getAllEcsClusters() { return ecsClusterProvider.getAllEcsClusters(); } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverter.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverter.java index 653558b9669..859f23ba795 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverter.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverter.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverter.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverter.java index a167e043776..e8584988299 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverter.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverter.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverter.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverter.java index 1eaa93b49c1..60073451007 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverter.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverter.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverter.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverter.java index cb1e72228ae..4e1db41a1a0 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverter.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverter.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverter.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverter.java index 8bed3e9992d..1f74c6f06bb 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverter.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverter.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstancesAtomicOperationConverter.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstancesAtomicOperationConverter.java index c1ffee3e6b5..b4344e68da5 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstancesAtomicOperationConverter.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstancesAtomicOperationConverter.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -42,7 +42,7 @@ public TerminateInstancesDescription convertDescription(Map input) { converted.setCredentials(getCredentialsObject(input.get("credentials").toString())); converted.setRegion(input.get("region").toString()); List ecsTaskIds = new ArrayList<>(); - for (Object id: (List) input.get("instanceIds")) { + for (Object id : (List) input.get("instanceIds")) { ecsTaskIds.add(id.toString()); } converted.setEcsTaskIds(ecsTaskIds); diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/AbstractECSDescription.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/AbstractECSDescription.java index eee2f117162..504ae69a342 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/AbstractECSDescription.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/AbstractECSDescription.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/CreateServerGroupDescription.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/CreateServerGroupDescription.java index 75d378872ce..0c82ef35f56 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/CreateServerGroupDescription.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/CreateServerGroupDescription.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ModifyServiceDescription.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ModifyServiceDescription.java index 31d5a01e712..97052a5a4d0 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ModifyServiceDescription.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ModifyServiceDescription.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ResizeServiceDescription.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ResizeServiceDescription.java index 986db6aa77e..84d8a0584f6 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ResizeServiceDescription.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/ResizeServiceDescription.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/TerminateInstancesDescription.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/TerminateInstancesDescription.java index 105d946b1ab..83e78b8e5fc 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/TerminateInstancesDescription.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/description/TerminateInstancesDescription.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/AbstractEcsAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/AbstractEcsAtomicOperation.java index 2e1afb8d306..fcbabea0ca6 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/AbstractEcsAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/AbstractEcsAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -29,18 +29,16 @@ import org.springframework.beans.factory.annotation.Autowired; public abstract class AbstractEcsAtomicOperation implements AtomicOperation { + private final String basePhase; @Autowired AmazonClientProvider amazonClientProvider; @Autowired AccountCredentialsProvider accountCredentialsProvider; @Autowired ContainerInformationService containerInformationService; - - private final String basePhase; - T description; - AbstractEcsAtomicOperation(T description, String basePhase){ + AbstractEcsAtomicOperation(T description, String basePhase) { this.description = description; this.basePhase = basePhase; @@ -63,7 +61,7 @@ AmazonECS getAmazonEcsClient() { return amazonClientProvider.getAmazonEcs(credentialAccount, credentialsProvider, region); } - protected String getRegion(){ + protected String getRegion() { return description.getRegion(); } @@ -71,7 +69,7 @@ AmazonCredentials getCredentials() { return (AmazonCredentials) accountCredentialsProvider.getCredentials(description.getCredentialAccount()); } - void updateTaskStatus(String status) { + void updateTaskStatus(String status) { getTask().updateStatus(basePhase, status); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java index 28bbe209531..2a7409c6d33 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -57,7 +57,6 @@ import java.util.Arrays; import java.util.Collection; import java.util.HashMap; -import java.util.HashSet; import java.util.LinkedList; import java.util.List; import java.util.Map; diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperation.java index 04400c10c70..40fc9a0ba8f 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -37,7 +37,7 @@ public DestroyServiceAtomicOperation(ModifyServiceDescription description) { @Override public Void operate(List priorOutputs) { - updateTaskStatus("Initializing Destroy Amazon ECS Server Group (Service) Operation..."); + updateTaskStatus("Initializing Destroy Amazon ECS Server Group Operation..."); AmazonECS ecs = getAmazonEcsClient(); String ecsClusterName = containerInformationService.getClusterName(description.getServerGroupName(), description.getAccount(), description.getRegion()); @@ -51,17 +51,17 @@ public Void operate(List priorOutputs) { updateServiceRequest.setDesiredCount(0); updateServiceRequest.setCluster(ecsClusterName); - updateTaskStatus("Scaling " + description.getServerGroupName() + " service down to 0."); + updateTaskStatus("Scaling " + description.getServerGroupName() + " server group down to 0."); ecs.updateService(updateServiceRequest); DeleteServiceRequest deleteServiceRequest = new DeleteServiceRequest(); deleteServiceRequest.setService(description.getServerGroupName()); deleteServiceRequest.setCluster(ecsClusterName); - updateTaskStatus("Deleting " + description.getServerGroupName() + " service."); + updateTaskStatus("Deleting " + description.getServerGroupName() + " server group."); DeleteServiceResult deleteServiceResult = ecs.deleteService(deleteServiceRequest); - updateTaskStatus("Deleting " + deleteServiceResult.getService().getTaskDefinition() + " task definition belonging to the service."); + updateTaskStatus("Deleting " + deleteServiceResult.getService().getTaskDefinition() + " task definition belonging to the server group."); ecs.deregisterTaskDefinition(new DeregisterTaskDefinitionRequest().withTaskDefinition(deleteServiceResult.getService().getTaskDefinition())); return null; diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperation.java index 38a42de46bc..4970938d7cf 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -43,12 +43,12 @@ private void disableService() { String account = description.getCredentialAccount(); String cluster = getCluster(service, account); - updateTaskStatus(String.format("Disabling %s service for %s.", service, account)); + updateTaskStatus(String.format("Disabling %s server group for %s.", service, account)); UpdateServiceRequest request = new UpdateServiceRequest() .withCluster(cluster) .withService(service) .withDesiredCount(0); ecs.updateService(request); - updateTaskStatus(String.format("Service %s disabled for %s.", service, account)); + updateTaskStatus(String.format("Server group %s disabled for %s.", service, account)); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperation.java index aece1740fa1..e84435cdd27 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -55,9 +55,9 @@ private void enableService() { .withService(service) .withDesiredCount(getMaxCapacity(cluster)); - updateTaskStatus(String.format("Enabling %s service for %s.", service, account)); + updateTaskStatus(String.format("Enabling %s server group for %s.", service, account)); ecsClient.updateService(request); - updateTaskStatus(String.format("Service %s enabled for %s.", service, account)); + updateTaskStatus(String.format("Server group %s enabled for %s.", service, account)); } private Integer getMaxCapacity(String cluster) { diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperation.java index a7eda13524e..e826921c7f9 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -24,14 +24,9 @@ import com.amazonaws.services.ecs.AmazonECS; import com.amazonaws.services.ecs.model.Service; import com.amazonaws.services.ecs.model.UpdateServiceRequest; -import com.netflix.spinnaker.clouddriver.aws.security.AmazonClientProvider; -import com.netflix.spinnaker.clouddriver.aws.security.AmazonCredentials; -import com.netflix.spinnaker.clouddriver.data.task.Task; -import com.netflix.spinnaker.clouddriver.data.task.TaskRepository; import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ResizeServiceDescription; import com.netflix.spinnaker.clouddriver.ecs.services.ContainerInformationService; import com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation; -import com.netflix.spinnaker.clouddriver.security.AccountCredentialsProvider; import org.springframework.beans.factory.annotation.Autowired; import java.util.List; diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstancesAtomicOperation.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstancesAtomicOperation.java index b9ca7a395c2..f761dc26dc5 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstancesAtomicOperation.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstancesAtomicOperation.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -42,5 +42,4 @@ public Void operate(List priorOutputs) { return null; } - } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/CommonValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/CommonValidator.java index 63beafec1cd..0ebb9dc7a83 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/CommonValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/CommonValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -55,42 +55,42 @@ boolean validateCredentials(AbstractAmazonCredentialsDescription credentialsDesc return true; } - void validateCapacity(Errors errors, ServerGroup.Capacity capacity){ - if(capacity != null){ + void validateCapacity(Errors errors, ServerGroup.Capacity capacity) { + if (capacity != null) { boolean desiredNotNull = capacity.getDesired() != null; boolean minNotNull = capacity.getMin() != null; boolean maxNotNull = capacity.getMax() != null; - if(!desiredNotNull){ + if (!desiredNotNull) { rejectValue(errors, "capacity.desired", "not.nullable"); } - if(!minNotNull){ + if (!minNotNull) { rejectValue(errors, "capacity.min", "not.nullable"); } - if(!maxNotNull){ + if (!maxNotNull) { rejectValue(errors, "capacity.max", "not.nullable"); } - positivityCheck(desiredNotNull,capacity.getDesired(), "desired", errors); - positivityCheck(minNotNull,capacity.getMin(), "min", errors); - positivityCheck(maxNotNull,capacity.getMax(), "max", errors); + positivityCheck(desiredNotNull, capacity.getDesired(), "desired", errors); + positivityCheck(minNotNull, capacity.getMin(), "min", errors); + positivityCheck(maxNotNull, capacity.getMax(), "max", errors); - if(minNotNull && maxNotNull){ - if(capacity.getMin() > capacity.getMax()){ + if (minNotNull && maxNotNull) { + if (capacity.getMin() > capacity.getMax()) { rejectValue(errors, "capacity.min.max.range", "invalid"); } - if(desiredNotNull && capacity.getDesired() > capacity.getMax()){ + if (desiredNotNull && capacity.getDesired() > capacity.getMax()) { rejectValue(errors, "capacity.desired", "exceeds.max"); } - if(desiredNotNull && capacity.getDesired() < capacity.getMin()){ + if (desiredNotNull && capacity.getDesired() < capacity.getMin()) { rejectValue(errors, "capacity.desired", "less.than.min"); } } - }else{ + } else { rejectValue(errors, "capacity", "not.nullable"); } } @@ -99,9 +99,9 @@ void rejectValue(Errors errors, String field, String reason) { errors.rejectValue(field, errorKey + "." + field + "." + reason); } - private void positivityCheck(boolean isNotNull, Integer capacity, String fieldName, Errors errors){ - if(isNotNull && capacity < 0){ - rejectValue(errors, "capacity."+fieldName, "invalid"); + private void positivityCheck(boolean isNotNull, Integer capacity, String fieldName, Errors errors) { + if (isNotNull && capacity < 0) { + rejectValue(errors, "capacity." + fieldName, "invalid"); } } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DestroyServiceAtomicOperationValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DestroyServiceAtomicOperationValidator.java index d37e93da43f..d2dc8b3ed34 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DestroyServiceAtomicOperationValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DestroyServiceAtomicOperationValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,7 +23,7 @@ @EcsOperation(AtomicOperations.DESTROY_SERVER_GROUP) @Component("destroyServiceAtomicOperationValidator") public class DestroyServiceAtomicOperationValidator extends ServerGroupDescriptionValidator { - public DestroyServiceAtomicOperationValidator(){ + public DestroyServiceAtomicOperationValidator() { super("destroyServiceAtomicOperation"); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DisableServiceDescriptionValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DisableServiceDescriptionValidator.java index d7de36e3bf7..8611ec3928a 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DisableServiceDescriptionValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/DisableServiceDescriptionValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,7 +23,7 @@ @EcsOperation(AtomicOperations.DISABLE_SERVER_GROUP) @Component("disableServiceAtomicOperationValidator") public class DisableServiceDescriptionValidator extends ServerGroupDescriptionValidator { - public DisableServiceDescriptionValidator(){ + public DisableServiceDescriptionValidator() { super("disableServiceDescription"); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServerGroupDescriptionValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServerGroupDescriptionValidator.java index db6d637cc5e..e6a81ed89ed 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServerGroupDescriptionValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServerGroupDescriptionValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EnableServiceDescriptionValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EnableServiceDescriptionValidator.java index 5eaf9794767..f8bb3476ab8 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EnableServiceDescriptionValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EnableServiceDescriptionValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -23,7 +23,7 @@ @EcsOperation(AtomicOperations.ENABLE_SERVER_GROUP) @Component("enableServiceAtomicOperationValidator") public class EnableServiceDescriptionValidator extends ServerGroupDescriptionValidator { - public EnableServiceDescriptionValidator(){ + public EnableServiceDescriptionValidator() { super("enableServiceDescription"); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeServiceDescriptionValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeServiceDescriptionValidator.java index ceb1267a00a..dbfe67251b3 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeServiceDescriptionValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeServiceDescriptionValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -43,7 +43,7 @@ public void validate(List priorDescriptions, Object description, Errors errors) validateRegions(typedDescription, Collections.singleton(typedDescription.getRegion()), errors, "region"); } - if(typedDescription.getServerGroupName() == null){ + if (typedDescription.getServerGroupName() == null) { rejectValue(errors, "serverGroupName", "not.nullable"); } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidator.java index ae57bf13180..26b6c2fa4ca 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstancesDescriptionValidator.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstancesDescriptionValidator.java index 668f1529ec1..091e3ab9d56 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstancesDescriptionValidator.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstancesDescriptionValidator.java @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -46,11 +46,10 @@ public void validate(List priorDescriptions, Object description, Errors errors) if (typedDescription.getEcsTaskIds() != null) { typedDescription.getEcsTaskIds().forEach(taskId -> { - if (!TASK_ID_PATTERN.matcher(taskId).find()) { - rejectValue(errors, "ecsTaskIds." + taskId, "invalid"); - } + if (!TASK_ID_PATTERN.matcher(taskId).find()) { + rejectValue(errors, "ecsTaskIds." + taskId, "invalid"); } - ); + }); } else { rejectValue(errors, "ecsTaskIds", "not.nullable"); } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummary.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummary.java index 7157e53dc60..b2e667a6ef7 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummary.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummary.java @@ -33,7 +33,7 @@ public class EcsLoadBalancerSummary implements Item { private String name; private Map byAccounts = new HashMap<>(); - public EcsLoadBalancerSummary withName(String name){ + public EcsLoadBalancerSummary withName(String name) { setName(name); return this; } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummaryByRegion.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummaryByRegion.java index 5d0330d94c6..c28994e62f3 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummaryByRegion.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/model/loadbalancer/EcsLoadBalancerSummaryByRegion.java @@ -18,21 +18,17 @@ import lombok.Data; -import java.util.HashMap; import java.util.LinkedList; import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; import static com.netflix.spinnaker.clouddriver.model.LoadBalancerProvider.ByRegion; -import static com.netflix.spinnaker.clouddriver.model.LoadBalancerProvider.Details; @Data public class EcsLoadBalancerSummaryByRegion implements ByRegion { private String name; private List loadBalancers = new LinkedList<>(); - public EcsLoadBalancerSummaryByRegion withName(String name){ + public EcsLoadBalancerSummaryByRegion withName(String name) { setName(name); return this; } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/EcsProvider.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/EcsProvider.java index 1aa6fd2bf3d..ca1153d31a4 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/EcsProvider.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/EcsProvider.java @@ -28,8 +28,6 @@ import java.util.Collections; import java.util.HashMap; import java.util.HashSet; -import java.util.LinkedList; -import java.util.List; import java.util.Map; import java.util.Set; import java.util.stream.Collectors; @@ -74,7 +72,7 @@ public Map getUrlMappingTemplates() { } @Override - public Map getSearchResultHydrators() { + public Map getSearchResultHydrators() { //TODO: Implement if needed - see InstanceSearchResultHydrator as an example. return Collections.emptyMap(); } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/AbstractEcsCachingAgent.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/AbstractEcsCachingAgent.java index 18e71e8aff5..d65a45e96f2 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/AbstractEcsCachingAgent.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/AbstractEcsCachingAgent.java @@ -46,12 +46,11 @@ import static com.netflix.spinnaker.clouddriver.ecs.cache.Keys.Namespace.IAM_ROLE; abstract class AbstractEcsCachingAgent implements CachingAgent { - private final Logger log = LoggerFactory.getLogger(getClass()); - final AmazonClientProvider amazonClientProvider; final AWSCredentialsProvider awsCredentialsProvider; final String region; final String accountName; + private final Logger log = LoggerFactory.getLogger(getClass()); AbstractEcsCachingAgent(String accountName, String region, AmazonClientProvider amazonClientProvider, AWSCredentialsProvider awsCredentialsProvider) { this.accountName = accountName; @@ -62,7 +61,8 @@ abstract class AbstractEcsCachingAgent implements CachingAgent { /** * Fetches items from the ECS service. - * @param ecs The AmazonECS client that will be used to make the queries. + * + * @param ecs The AmazonECS client that will be used to make the queries. * @param providerCache A ProviderCache that is used to access already existing cache. * @return A list of generic type objects. */ @@ -70,6 +70,7 @@ abstract class AbstractEcsCachingAgent implements CachingAgent { /** * Generates a map of CacheData collections associated to a key namespace from a given collection of generic type objects. + * * @param cacheableItems A collection of generic type objects. * @return A map of CacheData collections belonging to a key namespace. */ @@ -92,14 +93,15 @@ public CacheResult loadData(ProviderCache providerCache) { /** * Provides a set of ECS cluster ARNs. * Either uses the cache, or queries the ECS service. - * @param ecs The AmazonECS client to use for querying. + * + * @param ecs The AmazonECS client to use for querying. * @param providerCache The ProviderCache to retrieve clusters from. * @return A set of ECS cluster ARNs. */ Set getClusters(AmazonECS ecs, ProviderCache providerCache) { Set clusters = providerCache.getAll(ECS_CLUSTERS.toString()).stream() - .filter(cacheData -> cacheData.getAttributes().get("region").equals(region) && - cacheData.getAttributes().get("account").equals(accountName)) + .filter(cacheData -> cacheData.getAttributes().get("region").equals(region) && + cacheData.getAttributes().get("account").equals(accountName)) .map(cacheData -> (String) cacheData.getAttributes().get("clusterArn")) .collect(Collectors.toSet()); @@ -124,6 +126,7 @@ Set getClusters(AmazonECS ecs, ProviderCache providerCache) { /** * Provides the key namespace that the caching agent is authoritative of. * Currently only supports the caching agent being authoritative over one key namespace. + * * @return Key namespace. */ String getAuthoritativeKeyName() { @@ -160,6 +163,7 @@ CacheResult buildCacheResult(String authoritativeKeyName, List items, Provide /** * Evicts cache that does not belong to an entity on the ECS service. * This is done by evicting old keys that are no longer found in the new keys provided by the new data. + * * @param newData New data that contains new keys. * @param oldKeys Old keys. * @return Key collection associated to the key namespace the the caching agent is authoritative of. @@ -183,17 +187,18 @@ private Map> computeEvictableData(Collection keyParts = Keys.parse(key); return keyParts != null && - keyParts.get("account").equals(accountName) && - //IAM role keys are not region specific, so it will be true. The region will be checked of other keys. - (authoritativeKeyName.equals(IAM_ROLE.ns) || keyParts.get("region").equals(region)); + keyParts.get("account").equals(accountName) && + //IAM role keys are not region specific, so it will be true. The region will be checked of other keys. + (authoritativeKeyName.equals(IAM_ROLE.ns) || keyParts.get("region").equals(region)); } /** * This method is to be overridden in order to add extra evictions. + * * @param evictions The existing eviction map. * @return Eviction map with addtional keys. */ - protected Map> addExtraEvictions(Map> evictions){ + protected Map> addExtraEvictions(Map> evictions) { return evictions; } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ContainerInstanceCachingAgent.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ContainerInstanceCachingAgent.java index a6d04dc4aaa..ef1df6bf2d3 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ContainerInstanceCachingAgent.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ContainerInstanceCachingAgent.java @@ -55,6 +55,18 @@ public ContainerInstanceCachingAgent(String accountName, String region, AmazonCl super(accountName, region, amazonClientProvider, awsCredentialsProvider, registry); } + public static Map convertContainerInstanceToAttributes(ContainerInstance containerInstance) { + Map attributes = new HashMap<>(); + attributes.put("containerInstanceArn", containerInstance.getContainerInstanceArn()); + attributes.put("ec2InstanceId", containerInstance.getEc2InstanceId()); + for (Attribute containerAttribute : containerInstance.getAttributes()) { + if (containerAttribute.getName().equals("ecs.availability-zone")) { + attributes.put("availabilityZone", containerAttribute.getValue()); + } + } + return attributes; + } + @Override public String getAgentType() { return accountName + "/" + region + "/" + getClass().getSimpleName(); @@ -111,16 +123,4 @@ protected Map> generateFreshData(Collection convertContainerInstanceToAttributes(ContainerInstance containerInstance){ - Map attributes = new HashMap<>(); - attributes.put("containerInstanceArn", containerInstance.getContainerInstanceArn()); - attributes.put("ec2InstanceId", containerInstance.getEc2InstanceId()); - for(Attribute containerAttribute:containerInstance.getAttributes()){ - if(containerAttribute.getName().equals("ecs.availability-zone")){ - attributes.put("availabilityZone", containerAttribute.getValue()); - } - } - return attributes; - } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgent.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgent.java index 4e2dc904efa..c33a1081dad 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgent.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgent.java @@ -155,7 +155,7 @@ private boolean keyAccountRegionFilter(String key) { @Override public String getAgentType() { - return accountName + "/" + region + "/" + getClass().getSimpleName(); + return accountName + "/" + region + "/" + getClass().getSimpleName(); } @Override diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamPolicyReader.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamPolicyReader.java index 39c498a102a..1f235763e7a 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamPolicyReader.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamPolicyReader.java @@ -43,11 +43,11 @@ public Set getTrustedEntities(String urlEncodedPolicyDocum try { policyDocument = mapper.readValue(decodedPolicyDocument, Map.class); List> statementItems = (List>) policyDocument.get("Statement"); - for (Map statementItem: statementItems) { + for (Map statementItem : statementItems) { if ("sts:AssumeRole".equals(statementItem.get("Action"))) { Map principal = (Map) statementItem.get("Principal"); - for (Map.Entry principalEntry: principal.entrySet()) { + for (Map.Entry principalEntry : principal.entrySet()) { if (principalEntry.getValue() instanceof List) { ((List) principalEntry.getValue()).stream() .forEach(o -> trustedEntities.add(new IamTrustRelationship(principalEntry.getKey(), o.toString()))); diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCachingAgent.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCachingAgent.java index 8f48f9239a0..ca722de5517 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCachingAgent.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCachingAgent.java @@ -117,7 +117,9 @@ private void logUpcomingActions(Map> newDataMap, M private Map> computeEvictableData(Collection newData, Collection oldKeys) { - Set newKeys = newData.stream().map(CacheData::getId).collect(Collectors.toSet()); + Set newKeys = newData.stream() + .map(CacheData::getId) + .collect(Collectors.toSet()); Set evictedKeys = new HashSet<>(); for (String oldKey : oldKeys) { @@ -176,6 +178,7 @@ Set fetchIamRoles(AmazonIdentityManagement iam, String accountName) { return cacheableRoles; } + private boolean keyAccountFilter(String key) { Map keyParts = Keys.parse(key); return keyParts != null && diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCachingAgent.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCachingAgent.java index 6ab7098153e..a55fe2d04fa 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCachingAgent.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCachingAgent.java @@ -62,6 +62,23 @@ public TaskCachingAgent(String accountName, String region, AmazonClientProvider super(accountName, region, amazonClientProvider, awsCredentialsProvider, registry); } + public static Map convertTaskToAttributes(Task task) { + String taskId = StringUtils.substringAfterLast(task.getTaskArn(), "/"); + + Map attributes = new HashMap<>(); + attributes.put("taskId", taskId); + attributes.put("taskArn", task.getTaskArn()); + attributes.put("clusterArn", task.getClusterArn()); + attributes.put("containerInstanceArn", task.getContainerInstanceArn()); + attributes.put("group", task.getGroup()); + attributes.put("containers", task.getContainers()); + attributes.put("lastStatus", task.getLastStatus()); + attributes.put("desiredStatus", task.getDesiredStatus()); + attributes.put("startedAt", task.getStartedAt().getTime()); + + return attributes; + } + @Override public Collection getProvidedDataTypes() { return types; @@ -105,7 +122,7 @@ public Collection pendingOnDemandRequests(ProviderCache providerCache) { Map parsedKey = Keys.parse(onDemand.getId()); if (parsedKey != null && parsedKey.get("type") != null && (parsedKey.get("type").equals(SERVICES.toString()) || parsedKey.get("type").equals(TASKS.toString()) && - parsedKey.get("account").equals(accountName) && parsedKey.get("region").equals(region))) { + parsedKey.get("account").equals(accountName) && parsedKey.get("region").equals(region))) { parsedKey.put("type", "serverGroup"); parsedKey.put("serverGroup", parsedKey.get("serviceName")); @@ -165,21 +182,4 @@ protected Map> generateFreshData(Collection return dataMap; } - - public static Map convertTaskToAttributes(Task task){ - String taskId = StringUtils.substringAfterLast(task.getTaskArn(), "/"); - - Map attributes = new HashMap<>(); - attributes.put("taskId", taskId); - attributes.put("taskArn", task.getTaskArn()); - attributes.put("clusterArn", task.getClusterArn()); - attributes.put("containerInstanceArn", task.getContainerInstanceArn()); - attributes.put("group", task.getGroup()); - attributes.put("containers", task.getContainers()); - attributes.put("lastStatus", task.getLastStatus()); - attributes.put("desiredStatus", task.getDesiredStatus()); - attributes.put("startedAt", task.getStartedAt().getTime()); - - return attributes; - } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProvider.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProvider.java index dbe7ff63ffe..af657cf99d5 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProvider.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProvider.java @@ -84,7 +84,7 @@ public List findImage(String url) { NetflixAmazonCredentials credentials = getCredentials(accountId); if (!isValidRegion(credentials, region)) { - throw new Error("The repository URI provided does not belong to a region that the credentials have access to or is not a valid region."); + throw new IllegalArgumentException("The repository URI provided does not belong to a region that the credentials have access to or the region is not valid."); } AmazonECR amazonECR = amazonClientProvider.getAmazonEcr(credentials.getName(), credentials.getCredentialsProvider(), region); @@ -94,15 +94,13 @@ public List findImage(String url) { // TODO - what is the user interface we want to have here? We should discuss with Lars and Ethan from the community as this whole thing will undergo a big refactoring List imagesWithThisIdentifier = imagesResult.getImageDetails().stream() - .filter(imageDetail -> isTag ? - imageDetail.getImageTags() != null && imageDetail.getImageTags().contains(identifier) : - imageDetail.getImageDigest().equals(identifier)) + .filter(imageDetail -> imageFilter(imageDetail, identifier, isTag)) .collect(Collectors.toList()); if (imagesWithThisIdentifier.size() > 1) { - throw new Error("More than 1 image has this " + (isTag ? "tag" : "digest") + "! We can't handle this in the POC!"); + throw new IllegalArgumentException("More than 1 image has this " + (isTag ? "tag" : "digest") + "! This is currently not supported."); } else if (imagesWithThisIdentifier.size() == 0) { - throw new Error(String.format("No image with the " + (isTag ? "tag" : "digest") + " %s was found.", identifier)); + throw new IllegalArgumentException(String.format("No image with the " + (isTag ? "tag" : "digest") + " %s was found.", identifier)); } ImageDetail matchedImage = imagesWithThisIdentifier.get(0); @@ -140,6 +138,12 @@ public List findImage(String url) { return Collections.singletonList(ecsDockerImage); } + private boolean imageFilter(ImageDetail imageDetail, String identifier, boolean isTag) { + return isTag ? + imageDetail.getImageTags() != null && imageDetail.getImageTags().contains(identifier) : + imageDetail.getImageDigest().equals(identifier); + } + private NetflixAmazonCredentials getCredentials(String accountId) { for (AccountCredentials credentials : accountCredentialsProvider.getAll()) { if (credentials instanceof NetflixAmazonCredentials) { @@ -183,7 +187,7 @@ private String extractAwsRegion(String imageUrl) { private String extractString(Pattern pattern, String imageUrl, int group, String error) { Matcher matcher = pattern.matcher(imageUrl); if (!matcher.find()) { - throw new Error(error); + throw new IllegalArgumentException(error); } return matcher.group(group); } @@ -192,7 +196,7 @@ private String extractEcrIdentifier(String repository, String imageUrl) { final Pattern identifierPatter = Pattern.compile(repository + IDENTIFIER_PATTERN); Matcher matcher = identifierPatter.matcher(imageUrl); if (!matcher.find()) { - throw new Error("The repository URI provided does not contain a proper tag or sha256 digest."); + throw new IllegalArgumentException("The repository URI provided does not contain a proper tag or sha256 digest."); } return matcher.group(1).startsWith(":") ? matcher.group(2) : diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProvider.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProvider.java index 11b80813b67..f7d3689aa1d 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProvider.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProvider.java @@ -23,9 +23,6 @@ import org.springframework.stereotype.Component; import java.util.Collection; -import java.util.Collections; -import java.util.List; -import java.util.stream.Collectors; @Component public class EcsClusterProvider { diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProvider.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProvider.java index 0e41acbf804..77ddfcbea56 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProvider.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProvider.java @@ -42,7 +42,7 @@ public List list() { for (EcsLoadBalancerCache lb : loadBalancers) { String account = getEcsAccountName(lb.getAccount()); - if(account == null){ + if (account == null) { continue; } @@ -112,9 +112,9 @@ public Set getApplicationLoadBalancers(String application) { return null; //TODO - Implement this. This is used to show load balancers and reveals other buttons } - private String getEcsAccountName(String awsAccountName){ - for(ECSCredentialsConfig.Account ecsAccount: ecsCredentialsConfig.getAccounts()){ - if(ecsAccount.getAwsAccount().equals(awsAccountName)){ + private String getEcsAccountName(String awsAccountName) { + for (ECSCredentialsConfig.Account ecsAccount : ecsCredentialsConfig.getAccounts()) { + if (ecsAccount.getAwsAccount().equals(awsAccountName)) { return ecsAccount.getName(); } } diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/services/EcsCloudMetricService.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/services/EcsCloudMetricService.java index f5da20b9475..a12da56e21b 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/services/EcsCloudMetricService.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/services/EcsCloudMetricService.java @@ -142,10 +142,10 @@ private void deregisterScalableTargets(Set resources, String account, St } private PutMetricAlarmRequest buildPutMetricAlarmRequest(MetricAlarm metricAlarm, - String serviceName, - Set insufficientActionPolicyArns, - Set okActionPolicyArns, - Set alarmActionPolicyArns) { + String serviceName, + Set insufficientActionPolicyArns, + Set okActionPolicyArns, + Set alarmActionPolicyArns) { return new PutMetricAlarmRequest() .withAlarmName(metricAlarm.getAlarmName() + "-" + serviceName) .withEvaluationPeriods(metricAlarm.getEvaluationPeriods()) diff --git a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/view/EcsInstanceProvider.java b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/view/EcsInstanceProvider.java index 600b0bc45c9..d4acd22192e 100644 --- a/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/view/EcsInstanceProvider.java +++ b/clouddriver-ecs/src/main/java/com/netflix/spinnaker/clouddriver/ecs/view/EcsInstanceProvider.java @@ -88,10 +88,10 @@ public String getConsoleOutput(String account, String region, String id) { } private boolean isValidId(String id, String region) { - String id_regex = "[\\da-f]{8}-[\\da-f]{4}-[\\da-f]{4}-[\\da-f]{4}-[\\da-f]{12}"; - String id_only = String.format("^%s$", id_regex); - String arn = String.format("arn:aws:ecs:%s:\\d*:task/%s", region, id_regex); - return id.matches(id_only) || id.matches(arn); + String idRegex = "[\\da-f]{8}-[\\da-f]{4}-[\\da-f]{4}-[\\da-f]{4}-[\\da-f]{12}"; + String idOnly = String.format("^%s$", idRegex); + String arn = String.format("arn:aws:ecs:%s:\\d*:task/%s", region, idRegex); + return id.matches(idOnly) || id.matches(arn); } } diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/cache/EcsInstanceCacheClientSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/cache/EcsInstanceCacheClientSpec.groovy index 5d0783146e3..3b8e8670943 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/cache/EcsInstanceCacheClientSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/cache/EcsInstanceCacheClientSpec.groovy @@ -21,9 +21,9 @@ import com.fasterxml.jackson.databind.DeserializationFeature import com.fasterxml.jackson.databind.ObjectMapper import com.netflix.spinnaker.cats.cache.Cache import com.netflix.spinnaker.cats.cache.DefaultCacheData +import com.netflix.spinnaker.clouddriver.aws.data.Keys import com.netflix.spinnaker.clouddriver.ecs.cache.client.EcsInstanceCacheClient import spock.lang.Specification -import com.netflix.spinnaker.clouddriver.aws.data.Keys import spock.lang.Subject import static com.netflix.spinnaker.clouddriver.core.provider.agent.Namespace.INSTANCES diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricControllerSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricControllerSpec.groovy index 61f75876667..d62f0876af0 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricControllerSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/controllers/EcsCloudMetricControllerSpec.groovy @@ -16,7 +16,6 @@ package com.netflix.spinnaker.clouddriver.ecs.controllers -import com.amazonaws.services.cloudwatch.model.MetricAlarm import com.netflix.spinnaker.clouddriver.ecs.cache.model.EcsMetricAlarm import com.netflix.spinnaker.clouddriver.ecs.provider.view.EcsCloudMetricProvider import spock.lang.Specification diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverterSpec.groovy index 1067db245fa..bc4321288fa 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DestroyServiceAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverterSpec.groovy index 1d584a0170b..abe4b8c6da9 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/DisableServiceAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverterSpec.groovy index e56f10a86fb..042525e8a55 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EcsCreateServerGroupAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverterSpec.groovy index b21163e77c5..5f5f57bc544 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/EnableServiceAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ModifyServiceAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ModifyServiceAtomicOperationConverterSpec.groovy index e7e9d61b6aa..629f95593ec 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ModifyServiceAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ModifyServiceAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -16,12 +16,9 @@ package com.netflix.spinnaker.clouddriver.ecs.deploy.converters -import com.fasterxml.jackson.databind.ObjectMapper import com.netflix.spinnaker.clouddriver.ecs.TestCredential import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ModifyServiceDescription import com.netflix.spinnaker.clouddriver.ecs.deploy.ops.AbstractEcsAtomicOperation -import com.netflix.spinnaker.clouddriver.ecs.deploy.ops.EnableServiceAtomicOperation -import com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation import com.netflix.spinnaker.clouddriver.security.AbstractAtomicOperationsCredentialsSupport import com.netflix.spinnaker.clouddriver.security.AccountCredentialsProvider import spock.lang.Specification diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverterSpec.groovy index 814092d9011..c8f74f50681 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/ResizeServiceAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstanceAtomicOperationConverterSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstanceAtomicOperationConverterSpec.groovy index 1de48d2ef01..ab79349e1e8 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstanceAtomicOperationConverterSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/converters/TerminateInstanceAtomicOperationConverterSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CommonAtomicOperation.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CommonAtomicOperation.groovy index a0214279618..99d7fdd01fd 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CommonAtomicOperation.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CommonAtomicOperation.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -24,7 +24,7 @@ import com.netflix.spinnaker.clouddriver.ecs.services.ContainerInformationServic import com.netflix.spinnaker.clouddriver.security.AccountCredentialsProvider import spock.lang.Specification -class CommonAtomicOperation extends Specification{ +class CommonAtomicOperation extends Specification { def amazonClientProvider = Mock(AmazonClientProvider) def accountCredentialsProvider = Mock(AccountCredentialsProvider) def containerInformationService = Mock(ContainerInformationService) diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperationSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperationSpec.groovy index 175b37fd119..71fe38b21c2 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperationSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/CreateServerGroupAtomicOperationSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperationSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperationSpec.groovy index 8b3a2ab6b85..c50bdd74824 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperationSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DestroyServiceAtomicOperationSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -19,7 +19,6 @@ package com.netflix.spinnaker.clouddriver.ecs.deploy.ops import com.amazonaws.services.ecs.model.DeleteServiceResult import com.amazonaws.services.ecs.model.Service import com.netflix.spinnaker.clouddriver.ecs.TestCredential -import com.netflix.spinnaker.clouddriver.ecs.cache.client.EcsCloudWatchAlarmCacheClient import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ModifyServiceDescription import com.netflix.spinnaker.clouddriver.ecs.services.EcsCloudMetricService diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperationSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperationSpec.groovy index 2fea39d7135..ce19a52ce75 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperationSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/DisableServiceAtomicOperationSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperationSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperationSpec.groovy index a671bac7a34..d83b28c19b9 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperationSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/EnableServiceAtomicOperationSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperationSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperationSpec.groovy index 167c375a7f0..a93be8980cc 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperationSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/ResizeServiceAtomicOperationSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstanceAtomicOperationSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstanceAtomicOperationSpec.groovy index a6cca7126f7..2e1c3e09cc0 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstanceAtomicOperationSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/ops/TerminateInstanceAtomicOperationSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,7 +17,6 @@ package com.netflix.spinnaker.clouddriver.ecs.deploy.ops import com.netflix.spinnaker.clouddriver.ecs.TestCredential -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ModifyServiceDescription import com.netflix.spinnaker.clouddriver.ecs.deploy.description.TerminateInstancesDescription class TerminateInstanceAtomicOperationSpec extends CommonAtomicOperation { diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/AbstractValidatorSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/AbstractValidatorSpec.groovy index 885ebf40d62..3fa2a82f561 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/AbstractValidatorSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/AbstractValidatorSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -21,7 +21,6 @@ import com.netflix.spinnaker.clouddriver.data.task.TaskRepository import com.netflix.spinnaker.clouddriver.deploy.DescriptionValidator import com.netflix.spinnaker.clouddriver.ecs.TestCredential import com.netflix.spinnaker.clouddriver.ecs.deploy.description.AbstractECSDescription -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.CreateServerGroupDescription import org.springframework.validation.Errors import spock.lang.Specification import spock.lang.Subject @@ -50,7 +49,7 @@ abstract class AbstractValidatorSpec extends Specification { setTestRegion() } - def setTestRegion(){ + def setTestRegion() { testRegion = true } @@ -62,12 +61,12 @@ abstract class AbstractValidatorSpec extends Specification { def errors = Mock(Errors) when: - if(testRegion) { + if (testRegion) { validator.validate([], description, errors) } then: - if(testRegion) { + if (testRegion) { 1 * errors.rejectValue('region', _) } } @@ -83,7 +82,7 @@ abstract class AbstractValidatorSpec extends Specification { validator.validate([], description, errors) then: - for(def nullProperty:nullProperties){ + for (def nullProperty : nullProperties) { 1 * errors.rejectValue(nullProperty, "${descriptionName}.${nullProperty}.not.nullable") } } @@ -100,7 +99,7 @@ abstract class AbstractValidatorSpec extends Specification { validator.validate([], description, errors) then: - for(String invalidField:invalidFields) { + for (String invalidField : invalidFields) { 1 * errors.rejectValue(invalidField, "${descriptionName}.${invalidField}.invalid") } } diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServergroupDescriptionValidatorSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServergroupDescriptionValidatorSpec.groovy index 89ad29e3a4d..a3bb90363a5 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServergroupDescriptionValidatorSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/EcsCreateServergroupDescriptionValidatorSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeDescriptionValidatorSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeDescriptionValidatorSpec.groovy index 31882a4f1a9..f011f1db6a9 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeDescriptionValidatorSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ResizeDescriptionValidatorSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -20,7 +20,6 @@ import com.netflix.spinnaker.clouddriver.deploy.DescriptionValidator import com.netflix.spinnaker.clouddriver.ecs.TestCredential import com.netflix.spinnaker.clouddriver.ecs.deploy.description.AbstractECSDescription import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ResizeServiceDescription -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.TerminateInstancesDescription import com.netflix.spinnaker.clouddriver.model.ServerGroup import org.springframework.validation.Errors diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidatorSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidatorSpec.groovy index 2860a3f939f..4a8f732b811 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidatorSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/ServerGroupDescriptionValidatorSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -20,9 +20,6 @@ import com.netflix.spinnaker.clouddriver.deploy.DescriptionValidator import com.netflix.spinnaker.clouddriver.ecs.TestCredential import com.netflix.spinnaker.clouddriver.ecs.deploy.description.AbstractECSDescription import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ModifyServiceDescription -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ResizeServiceDescription -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.TerminateInstancesDescription -import org.springframework.validation.Errors class ServerGroupDescriptionValidatorSpec extends AbstractValidatorSpec { diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstanceDescriptionValidatorSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstanceDescriptionValidatorSpec.groovy index d4f077e71c3..5af2d597e50 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstanceDescriptionValidatorSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/deploy/validators/TerminateInstanceDescriptionValidatorSpec.groovy @@ -1,5 +1,5 @@ /* - * Copyright 2017 Lookout, Inc. + * Copyright 2018 Lookout, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -19,10 +19,7 @@ package com.netflix.spinnaker.clouddriver.ecs.deploy.validators import com.netflix.spinnaker.clouddriver.deploy.DescriptionValidator import com.netflix.spinnaker.clouddriver.ecs.TestCredential import com.netflix.spinnaker.clouddriver.ecs.deploy.description.AbstractECSDescription -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ModifyServiceDescription -import com.netflix.spinnaker.clouddriver.ecs.deploy.description.ResizeServiceDescription import com.netflix.spinnaker.clouddriver.ecs.deploy.description.TerminateInstancesDescription -import org.springframework.validation.Errors class TerminateInstanceDescriptionValidatorSpec extends AbstractValidatorSpec { def invalidTaskId = 'some-invalid-task-id' diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgentSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgentSpec.groovy index 849ab8e261c..92cf1fcd42e 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgentSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/agent/EcsCloudMetricAlarmCachingAgentSpec.groovy @@ -68,7 +68,6 @@ class EcsCloudMetricAlarmCachingAgentSpec extends Specification { def cacheData = agent.generateFreshData(metricAlarms) then: - cacheData.size() == 1 cacheData.get(ALARMS.ns).size() == metricAlarms.size() metricAlarms*.alarmName.containsAll(cacheData.get(ALARMS.ns)*.getAttributes().alarmName) metricAlarms*.alarmArn.containsAll(cacheData.get(ALARMS.ns)*.getAttributes().alarmArn) diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProviderSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProviderSpec.groovy index 3919d50b2e9..efef272c016 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProviderSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcrImageProviderSpec.groovy @@ -145,7 +145,7 @@ class EcrImageProviderSpec extends Specification { provider.findImage(url) then: - final Error error = thrown() + final IllegalArgumentException error = thrown() error.message == "The repository URI provided does not contain a proper account ID." } @@ -161,7 +161,7 @@ class EcrImageProviderSpec extends Specification { provider.findImage(url) then: - final Error error = thrown() + final IllegalArgumentException error = thrown() error.message == "The repository URI provided does not contain a proper repository name." } @@ -177,7 +177,7 @@ class EcrImageProviderSpec extends Specification { provider.findImage(url) then: - final Error error = thrown() + final IllegalArgumentException error = thrown() error.message == "The repository URI provided does not contain a proper region." } @@ -195,7 +195,7 @@ class EcrImageProviderSpec extends Specification { provider.findImage(url) then: - final Error error = thrown() - error.message == "The repository URI provided does not belong to a region that the credentials have access to or is not a valid region." + final IllegalArgumentException error = thrown() + error.message == "The repository URI provided does not belong to a region that the credentials have access to or the region is not valid." } } diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProviderSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProviderSpec.groovy index 5e702c5ed8f..1b368ea12db 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProviderSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsClusterProviderSpec.groovy @@ -89,7 +89,7 @@ class EcsClusterProviderSpec extends Specification { cacheView.getAll(_) >> cacheData when: - List ecsClusters = ecsClusterProvider.getAllEcsClusters() + Collection ecsClusters = ecsClusterProvider.getAllEcsClusters() then: ecsClusters.size() == numberOfClusters diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProviderSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProviderSpec.groovy index ec69d3f798e..c0c8d41d047 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProviderSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsLoadBalancerProviderSpec.groovy @@ -43,10 +43,10 @@ class EcsLoadBalancerProviderSpec extends Specification { def expectedNumberOfLoadbalancers = 2 def givenList = [] def accounts = [] - for (int x = 0; x < expectedNumberOfLoadbalancers; x++) { + (1..expectedNumberOfLoadbalancers).forEach() { givenList << new EcsLoadBalancerCache( - account: 'test-account-' + x, - region: 'us-west-' + x, + account: 'test-account-' + it, + region: 'us-west-' + it, loadBalancerArn: 'arn', loadBalancerType: 'always-classic', cloudProvider: EcsCloudProvider.ID, @@ -54,20 +54,20 @@ class EcsLoadBalancerProviderSpec extends Specification { scheme: 'scheme', availabilityZones: [], ipAddressType: 'ipv4', - loadBalancerName: 'load-balancer-' + x, + loadBalancerName: 'load-balancer-' + it, canonicalHostedZoneId: 'zone-id', - vpcId: 'vpc-id-' + x, + vpcId: 'vpc-id-' + it, dnsname: 'dns-name', createdTime: System.currentTimeMillis(), subnets: [], securityGroups: [], - targetGroups: ['target-group-' + x], + targetGroups: ['target-group-' + it], serverGroups: [] ) accounts << new ECSCredentialsConfig.Account( - name: 'test-account-' + x, - awsAccount: 'test-account-' + x + name: 'test-account-' + it, + awsAccount: 'test-account-' + it ) } ecsCredentialsConfig.getAccounts() >> accounts diff --git a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsServerClusterProviderSpec.groovy b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsServerClusterProviderSpec.groovy index df1294e06ce..3b513c4c2a7 100644 --- a/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsServerClusterProviderSpec.groovy +++ b/clouddriver-ecs/src/test/groovy/com/netflix/spinnaker/clouddriver/ecs/provider/view/EcsServerClusterProviderSpec.groovy @@ -24,7 +24,8 @@ import com.fasterxml.jackson.databind.DeserializationFeature import com.fasterxml.jackson.databind.ObjectMapper import com.netflix.spinnaker.cats.cache.Cache import com.netflix.spinnaker.cats.cache.DefaultCacheData -import com.netflix.spinnaker.clouddriver.ecs.TestCredential +import com.netflix.spinnaker.clouddriver.aws.security.AmazonCredentials +import com.netflix.spinnaker.clouddriver.ecs.cache.Keys import com.netflix.spinnaker.clouddriver.ecs.cache.client.* import com.netflix.spinnaker.clouddriver.ecs.cache.model.EcsLoadBalancerCache import com.netflix.spinnaker.clouddriver.ecs.model.EcsServerCluster @@ -38,9 +39,6 @@ import com.netflix.spinnaker.clouddriver.security.AccountCredentialsProvider import spock.lang.Specification import spock.lang.Subject -import static com.netflix.spinnaker.clouddriver.ecs.cache.Keys.Namespace.SERVICES -import static com.netflix.spinnaker.clouddriver.ecs.cache.Keys.Namespace.TASKS - class EcsServerClusterProviderSpec extends Specification { def cacheView = Mock(Cache) def objectMapper = new ObjectMapper() @@ -76,7 +74,12 @@ class EcsServerClusterProviderSpec extends Specification { def serviceName = "${familyName}-v007" def startedAt = new Date() - def creds = TestCredential.named('test', [CLOUD_PROVIDER: 'ecs']) + def creds = Mock(AmazonCredentials) + creds.getCloudProvider() >> 'ecs' + creds.getName() >> 'test' + creds.getRegions() >> [new AmazonCredentials.AWSRegion('us-east-1', ['us-east-1b', 'us-east-1c', 'us-east-1d']), + new AmazonCredentials.AWSRegion('us-west-1', ['us-west-1b', 'us-west-1c', 'us-west-1d'])] + def cachedService = new Service( serviceName: serviceName, @@ -154,8 +157,8 @@ class EcsServerClusterProviderSpec extends Specification { ecsCloudWatchAlarmCacheClient.getMetricAlarms(_, _, _) >> [] cacheView.filterIdentifiers(_, _) >> ['key'] - cacheView.getAll(SERVICES.ns, _) >> [serviceCacheData] - cacheView.getAll(TASKS.ns, _) >> [taskCacheData] + cacheView.getAll(Keys.Namespace.SERVICES.ns, _) >> [serviceCacheData] + cacheView.getAll(Keys.Namespace.TASKS.ns, _) >> [taskCacheData] when: def retrievedCluster = provider.getCluster("myapp", creds.getName(), familyName) diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/CommonCachingAgent.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/CommonCachingAgent.java index 6a361c7b10d..de41091c0cd 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/CommonCachingAgent.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/CommonCachingAgent.java @@ -31,35 +31,35 @@ public class CommonCachingAgent { static final String REGION = "us-west-2"; - private static final String ECU_SERVICE = "arn:aws:ecs:" + REGION + ":012345678910:"; + private static final String ECS_SERIVCE = "arn:aws:ecs:" + REGION + ":012345678910:"; static final String ACCOUNT = "test-account"; static final String APP_NAME = "testapp"; - static final String ROLE_ARN = ECU_SERVICE + "role/test-role"; + static final String ROLE_ARN = ECS_SERIVCE + "service/test-role"; static final String STATUS = "RUNNING"; static final String SERVICE_NAME_1 = APP_NAME + "-stack-detail-v1"; static final String SERVICE_NAME_2 = APP_NAME + "-stack-detail-v1"; - static final String SERVICE_ARN_1 = ECU_SERVICE + "service/" + SERVICE_NAME_1; - static final String SERVICE_ARN_2 = ECU_SERVICE + "service/" + SERVICE_NAME_2; + static final String SERVICE_ARN_1 = ECS_SERIVCE + "service/" + SERVICE_NAME_1; + static final String SERVICE_ARN_2 = ECS_SERIVCE + "service/" + SERVICE_NAME_2; static final String CLUSTER_NAME_1 = "test-cluster-1"; static final String CLUSTER_NAME_2 = "test-cluster-2"; - static final String CLUSTER_ARN_1 = ECU_SERVICE + "cluster/" + CLUSTER_NAME_1; - static final String CLUSTER_ARN_2 = ECU_SERVICE + "cluster/" + CLUSTER_NAME_2; + static final String CLUSTER_ARN_1 = ECS_SERIVCE + "cluster/" + CLUSTER_NAME_1; + static final String CLUSTER_ARN_2 = ECS_SERIVCE + "cluster/" + CLUSTER_NAME_2; static final String TASK_ID_1 = "1dc5c17a-422b-4dc4-b493-371970c6c4d6"; static final String TASK_ID_2 = "1dc5c17a-422b-4dc4-b493-371970c6c4d6"; - static final String TASK_ARN_1 = ECU_SERVICE + "task/" + TASK_ID_1; - static final String TASK_ARN_2 = ECU_SERVICE + "task/" + TASK_ID_2; + static final String TASK_ARN_1 = ECS_SERIVCE + "task/" + TASK_ID_1; + static final String TASK_ARN_2 = ECS_SERIVCE + "task/" + TASK_ID_2; - static final String CONTAINER_INSTANCE_ARN_1 = ECU_SERVICE + "container-instance/14e8cce9-0b16-4af4-bfac-a85f7587aa98"; - static final String CONTAINER_INSTANCE_ARN_2 = ECU_SERVICE + "container-instance/deadbeef-0b16-4af4-bfac-a85f7587aa98"; + static final String CONTAINER_INSTANCE_ARN_1 = ECS_SERIVCE + "container-instance/14e8cce9-0b16-4af4-bfac-a85f7587aa98"; + static final String CONTAINER_INSTANCE_ARN_2 = ECS_SERIVCE + "container-instance/deadbeef-0b16-4af4-bfac-a85f7587aa98"; static final String EC2_INSTANCE_ID_1 = "i-042f39dc"; static final String EC2_INSTANCE_ID_2 = "i-deadbeef"; - static final String TASK_DEFINITION_ARN_1 = ECU_SERVICE + "task-definition/hello_world:10"; - static final String TASK_DEFINITION_ARN_2 = ECU_SERVICE + "task-definition/hello_world:20"; + static final String TASK_DEFINITION_ARN_1 = ECS_SERIVCE + "task-definition/hello_world:10"; + static final String TASK_DEFINITION_ARN_2 = ECS_SERIVCE + "task-definition/hello_world:20"; static final AmazonECS ecs = mock(AmazonECS.class); static final AmazonClientProvider clientProvider = mock(AmazonClientProvider.class); @@ -72,5 +72,4 @@ public class CommonCachingAgent { public static void setUp() { when(clientProvider.getAmazonEcs(anyString(), any(AWSCredentialsProvider.class), anyString())).thenReturn(ecs); } - } diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCacheTest.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCacheTest.java index 7c1a84835b4..899667d8fd9 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCacheTest.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/IamRoleCacheTest.java @@ -21,6 +21,7 @@ import com.amazonaws.services.identitymanagement.model.ListRolesRequest; import com.amazonaws.services.identitymanagement.model.ListRolesResult; import com.amazonaws.services.identitymanagement.model.Role; +import com.fasterxml.jackson.databind.ObjectMapper; import com.netflix.spinnaker.cats.agent.CacheResult; import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.Keys; @@ -53,6 +54,7 @@ public class IamRoleCacheTest extends CommonCachingAgent { public void shouldRetrieveFromWrittenCache() { //Given when(clientProvider.getIam(anyString(), any(AWSCredentialsProvider.class), anyString())).thenReturn(iam); + ObjectMapper mapper = new ObjectMapper(); String name = "iam-role-name"; String roleArn = "iam-role-arn"; String key = Keys.getIamRoleKey(ACCOUNT, name); diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCacheTest.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCacheTest.java index 624ae1f6cf2..ea476938413 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCacheTest.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCacheTest.java @@ -24,6 +24,7 @@ import com.amazonaws.services.ecs.model.ListServicesRequest; import com.amazonaws.services.ecs.model.ListServicesResult; import com.amazonaws.services.ecs.model.Service; +import com.fasterxml.jackson.databind.ObjectMapper; import com.netflix.spinnaker.cats.agent.CacheResult; import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.Keys; @@ -42,6 +43,8 @@ import static org.mockito.Mockito.when; public class ServiceCacheTest extends CommonCachingAgent { + private final ObjectMapper mapper = new ObjectMapper(); + private final ServiceCachingAgent agent = new ServiceCachingAgent(ACCOUNT, REGION, clientProvider, credentialsProvider, registry); @Subject private final ServiceCacheClient client = new ServiceCacheClient(providerCache, mapper); diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCachingAgentTest.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCachingAgentTest.java index bcbd8859fbf..d1a96d27d7c 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCachingAgentTest.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/ServiceCachingAgentTest.java @@ -44,6 +44,7 @@ import static org.mockito.Matchers.any; import static org.mockito.Mockito.when; + public class ServiceCachingAgentTest extends CommonCachingAgent { @Subject private final ServiceCachingAgent agent = new ServiceCachingAgent(ACCOUNT, REGION, clientProvider, credentialsProvider, registry); diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCacheTest.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCacheTest.java index 418ebf6f4cd..1b56332549d 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCacheTest.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskCacheTest.java @@ -23,6 +23,7 @@ import com.amazonaws.services.ecs.model.ListTasksRequest; import com.amazonaws.services.ecs.model.ListTasksResult; import com.amazonaws.services.ecs.model.Task; +import com.fasterxml.jackson.databind.ObjectMapper; import com.netflix.spinnaker.cats.agent.CacheResult; import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.Keys; @@ -41,6 +42,7 @@ public class TaskCacheTest extends CommonCachingAgent { + private final ObjectMapper mapper = new ObjectMapper(); @Subject private final TaskCachingAgent agent = new TaskCachingAgent(ACCOUNT, REGION, clientProvider, credentialsProvider, registry); @Subject diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCacheTest.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCacheTest.java index e7d3ec9d5f2..9fc1aa398a2 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCacheTest.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCacheTest.java @@ -21,6 +21,7 @@ import com.amazonaws.services.ecs.model.ListTaskDefinitionsRequest; import com.amazonaws.services.ecs.model.ListTaskDefinitionsResult; import com.amazonaws.services.ecs.model.TaskDefinition; +import com.fasterxml.jackson.databind.ObjectMapper; import com.netflix.spinnaker.cats.agent.CacheResult; import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.Keys; @@ -39,11 +40,11 @@ public class TaskDefinitionCacheTest extends CommonCachingAgent { - @Subject - private final TaskDefinitionCacheClient client = new TaskDefinitionCacheClient(providerCache, mapper); + private final ObjectMapper mapper = new ObjectMapper(); @Subject private final TaskDefinitionCachingAgent agent = new TaskDefinitionCachingAgent(ACCOUNT, REGION, clientProvider, credentialsProvider, registry, mapper); - + @Subject + private final TaskDefinitionCacheClient client = new TaskDefinitionCacheClient(providerCache, mapper); @Test public void shouldRetrieveFromWrittenCache() { @@ -52,7 +53,6 @@ public void shouldRetrieveFromWrittenCache() { TaskDefinition taskDefinition = new TaskDefinition(); taskDefinition.setTaskDefinitionArn(TASK_DEFINITION_ARN_1); - taskDefinition.setTaskRoleArn(ROLE_ARN); taskDefinition.setContainerDefinitions(Collections.emptyList()); when(ecs.listTaskDefinitions(any(ListTaskDefinitionsRequest.class))).thenReturn(new ListTaskDefinitionsResult().withTaskDefinitionArns(TASK_DEFINITION_ARN_1)); diff --git a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCachingAgentTest.java b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCachingAgentTest.java index a57d270353d..a604e57b37b 100644 --- a/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCachingAgentTest.java +++ b/clouddriver-ecs/src/test/java/com/netflix/spinnaker/clouddriver/ecs/provider/agent/TaskDefinitionCachingAgentTest.java @@ -21,6 +21,7 @@ import com.amazonaws.services.ecs.model.ListTaskDefinitionsRequest; import com.amazonaws.services.ecs.model.ListTaskDefinitionsResult; import com.amazonaws.services.ecs.model.TaskDefinition; +import com.fasterxml.jackson.databind.ObjectMapper; import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.ecs.cache.Keys; import org.junit.Test; @@ -41,6 +42,8 @@ public class TaskDefinitionCachingAgentTest extends CommonCachingAgent { + ObjectMapper mapper = new ObjectMapper(); + @Subject private final TaskDefinitionCachingAgent agent = new TaskDefinitionCachingAgent(ACCOUNT, REGION, clientProvider, credentialsProvider, registry, mapper); @@ -75,7 +78,7 @@ public void shouldGenerateFreshData() { for (String taskDefArn : taskDefinitionArns) { keys.add(Keys.getTaskDefinitionKey(ACCOUNT, REGION, taskDefArn)); - tasks.add(new TaskDefinition().withTaskDefinitionArn(taskDefArn).withTaskRoleArn(ROLE_ARN) + tasks.add(new TaskDefinition().withTaskDefinitionArn(taskDefArn) .withContainerDefinitions(Collections.emptyList())); } diff --git a/clouddriver-elasticsearch-aws/clouddriver-elasticsearch-aws.gradle b/clouddriver-elasticsearch-aws/clouddriver-elasticsearch-aws.gradle new file mode 100644 index 00000000000..8ea5572c026 --- /dev/null +++ b/clouddriver-elasticsearch-aws/clouddriver-elasticsearch-aws.gradle @@ -0,0 +1,10 @@ +apply from: "$rootDir/gradle/kotlin.gradle" + +repositories { + jcenter() +} + +dependencies { + compile project(":clouddriver-aws") + compile project(":clouddriver-elasticsearch") +} diff --git a/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/ElasticSearchClient.kt b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/ElasticSearchClient.kt new file mode 100644 index 00000000000..aeb55456a42 --- /dev/null +++ b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/ElasticSearchClient.kt @@ -0,0 +1,105 @@ +/* + * Copyright 2018 Netflix, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.elasticsearch + +import com.fasterxml.jackson.databind.ObjectMapper +import com.netflix.spinnaker.clouddriver.elasticsearch.model.ElasticSearchException +import com.netflix.spinnaker.clouddriver.elasticsearch.model.Model +import com.netflix.spinnaker.clouddriver.elasticsearch.model.ModelType +import io.searchbox.client.JestClient +import io.searchbox.core.Bulk +import io.searchbox.core.BulkResult +import io.searchbox.core.Index +import io.searchbox.indices.CreateIndex +import io.searchbox.indices.DeleteIndex +import io.searchbox.indices.aliases.AddAliasMapping +import io.searchbox.indices.aliases.GetAliases +import io.searchbox.indices.aliases.ModifyAliases +import java.io.IOException + +class ElasticSearchClient(private val objectMapper : ObjectMapper, private val jestClient: JestClient) { + fun getPreviousIndexes(prefix: String): Set { + try { + val result = jestClient.execute(GetAliases.Builder().build()) + val r = objectMapper.readValue(result.jsonString, Map::class.java) as Map + return r.keys.filter { k -> k.startsWith(prefix) }.toSet() + } catch (e: IOException) { + throw ElasticSearchException("Unable to fetch previous indexes (prefix: $prefix)", e) + } + } + + fun createIndex(prefix: String): String { + val newIndexName = "${prefix}_${System.currentTimeMillis()}" + + try { + jestClient.execute(CreateIndex.Builder(newIndexName).build()) + return newIndexName + } catch (e: IOException) { + throw ElasticSearchException("Unable to create index (index: $newIndexName)", e) + } + } + + fun createAlias(index: String, alias: String) { + try { + jestClient.execute( + ModifyAliases.Builder( + AddAliasMapping.Builder(index, alias).build() + ).build() + ) + } catch (e: IOException) { + throw ElasticSearchException("Unable to create alias (index: $index, alias: $alias)", e) + } + } + + fun deleteIndex(index: String) { + try { + jestClient.execute( + DeleteIndex.Builder(index).build() + ) + } catch (e: IOException) { + throw ElasticSearchException("Unable to delete index (index: $index)", e) + } + } + + fun store(index: String, type: ModelType, partition: List) { + var builder: Bulk.Builder = Bulk.Builder().defaultIndex(index) + + for (serverGroupModel in partition) { + builder = builder.addAction( + Index.Builder(objectMapper.convertValue(serverGroupModel, Map::class.java)) + .index(index) + .type(type.toString()) + .id(serverGroupModel.id) + .build() + ) + } + + val bulk = builder.build() + try { + val jestResult = jestClient.execute(bulk) + if (!jestResult.isSucceeded) { + throw ElasticSearchException( + java.lang.String.format("Failed to index server groups, reason: '%s'", jestResult.getErrorMessage()) + ) + } + } catch (e: IOException) { + throw ElasticSearchException( + java.lang.String.format("Failed to index server groups, reason: '%s'", e.message) + ) + } + } +} diff --git a/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonCachingAgentProvider.kt b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonCachingAgentProvider.kt new file mode 100644 index 00000000000..b528ec07945 --- /dev/null +++ b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonCachingAgentProvider.kt @@ -0,0 +1,73 @@ +/* + * Copyright 2018 Netflix, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.elasticsearch.aws + +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.databind.SerializationFeature +import com.netflix.spectator.api.Registry +import com.netflix.spinnaker.cats.agent.Agent +import com.netflix.spinnaker.cats.agent.AgentProvider +import com.netflix.spinnaker.clouddriver.aws.provider.AwsProvider +import com.netflix.spinnaker.clouddriver.aws.security.AmazonClientProvider +import com.netflix.spinnaker.clouddriver.aws.security.NetflixAmazonCredentials +import com.netflix.spinnaker.clouddriver.elasticsearch.ElasticSearchClient +import com.netflix.spinnaker.clouddriver.security.AccountCredentialsProvider +import com.netflix.spinnaker.kork.core.RetrySupport +import io.searchbox.client.JestClient + +open class ElasticSearchAmazonCachingAgentProvider( + private val objectMapper: ObjectMapper, + private val jestClient: JestClient, + private val retrySupport: RetrySupport, + private val registry: Registry, + private val amazonClientProvider: AmazonClientProvider, + private val accountCredentialsProvider: AccountCredentialsProvider +) : AgentProvider { + + override fun supports(providerName: String): Boolean { + return providerName.equals(AwsProvider.PROVIDER_NAME, ignoreCase = true) + } + + override fun agents(): Collection { + val credentials = accountCredentialsProvider + .all + .filter { NetflixAmazonCredentials::class.java.isInstance(it) } + .map { c -> c as NetflixAmazonCredentials } + + val elasticSearchClient = ElasticSearchClient( + objectMapper.copy().enable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS), + jestClient + ) + + return listOf( + ElasticSearchAmazonServerGroupCachingAgent( + retrySupport, + registry, + amazonClientProvider, + credentials, + elasticSearchClient + ), + ElasticSearchAmazonInstanceCachingAgent( + retrySupport, + registry, + amazonClientProvider, + credentials, + elasticSearchClient + ) + ) + } +} diff --git a/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonInstanceCachingAgent.kt b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonInstanceCachingAgent.kt new file mode 100644 index 00000000000..5ca541b37c7 --- /dev/null +++ b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonInstanceCachingAgent.kt @@ -0,0 +1,130 @@ +/* + * Copyright 2018 Netflix, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.elasticsearch.aws + +import com.amazonaws.services.ec2.AmazonEC2 +import com.amazonaws.services.ec2.model.DescribeInstancesRequest +import com.amazonaws.services.ec2.model.Instance +import com.google.common.collect.Lists +import com.netflix.spectator.api.Registry +import com.netflix.spinnaker.cats.agent.RunnableAgent +import com.netflix.spinnaker.clouddriver.aws.provider.AwsProvider +import com.netflix.spinnaker.clouddriver.aws.security.AmazonClientProvider +import com.netflix.spinnaker.clouddriver.aws.security.NetflixAmazonCredentials +import com.netflix.spinnaker.clouddriver.cache.CustomScheduledAgent +import com.netflix.spinnaker.clouddriver.elasticsearch.ElasticSearchClient +import com.netflix.spinnaker.clouddriver.elasticsearch.model.* +import com.netflix.spinnaker.kork.core.RetrySupport +import org.slf4j.LoggerFactory +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +class ElasticSearchAmazonInstanceCachingAgent( + val retrySupport: RetrySupport, + val registry: Registry, + val amazonClientProvider: AmazonClientProvider, + val accounts: Collection, + val elasticSearchClient: ElasticSearchClient +) : RunnableAgent, CustomScheduledAgent { + + private val log = LoggerFactory.getLogger(ElasticSearchAmazonInstanceCachingAgent::class.java) + + override fun getPollIntervalMillis(): Long { + return TimeUnit.MINUTES.toMillis(10) + } + + override fun getTimeoutMillis(): Long { + return TimeUnit.MINUTES.toMillis(20) + } + + override fun getAgentType(): String { + return ElasticSearchAmazonInstanceCachingAgent::class.java.simpleName + } + + override fun getProviderName(): String { + return AwsProvider.PROVIDER_NAME + } + + override fun run() { + val prefix = "aws_instances" + val previousIndexes = elasticSearchClient.getPreviousIndexes(prefix) + val index = elasticSearchClient.createIndex(prefix) + + val instanceCount = AtomicInteger(0) + + for (credentials in accounts) { + for (region in credentials.regions) { + val instanceModels = fetchInstanceModels(credentials, region.name) + + instanceCount.addAndGet(instanceModels.size) + + for (partition in Lists.partition(instanceModels, 1000)) { + retrySupport.retry( + { elasticSearchClient.store(index, ModelType.Intance, partition) }, + 10, + 2000, + false + ) + } + } + } + + log.info("Total # of instances: $instanceCount") + + elasticSearchClient.createAlias(index, prefix) + for (previousIndex in previousIndexes) { + elasticSearchClient.deleteIndex(previousIndex) + } + } + + private fun fetchInstanceModels(credentials: NetflixAmazonCredentials, region: String): List { + val amazonEC2 = amazonClientProvider.getAmazonEC2(credentials, region) + + log.debug("Describing All Instances in ${credentials.name}:${region}") + + return fetchAllInstances(amazonEC2).map { instance -> + InstanceModel( + "${credentials.accountId}:${region}:${instance.instanceId}".toLowerCase(), + instance.instanceId, + InstanceTypeModel(instance.instanceType), + LocationModel("availabilityZone", instance.placement.availabilityZone), + AccountModel(credentials.accountId, credentials.name), + listOfNotNull(instance.publicIpAddress, instance.privateIpAddress), + instance.launchTime, + instance.tags.map { TagModel(it.key, it.value) } + ) + } + } + + private fun fetchAllInstances(amazonEC2: AmazonEC2): List { + val instances = mutableListOf() + + var request = DescribeInstancesRequest() + while (true) { + val response = amazonEC2.describeInstances(request) + + instances.addAll(response.reservations.flatMap { it.instances }) + if (response.nextToken != null) { + request = request.withNextToken(response.nextToken) + } else { + break + } + } + + return instances; + } +} diff --git a/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonServerGroupCachingAgent.kt b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonServerGroupCachingAgent.kt new file mode 100644 index 00000000000..a656a8abddb --- /dev/null +++ b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/aws/ElasticSearchAmazonServerGroupCachingAgent.kt @@ -0,0 +1,173 @@ +/* + * Copyright 2018 Netflix, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.elasticsearch.aws + + +import com.amazonaws.services.autoscaling.AmazonAutoScaling +import com.amazonaws.services.autoscaling.model.AutoScalingGroup +import com.amazonaws.services.autoscaling.model.DescribeAutoScalingGroupsRequest +import com.amazonaws.services.autoscaling.model.DescribeLaunchConfigurationsRequest +import com.amazonaws.services.autoscaling.model.LaunchConfiguration +import com.google.common.collect.Lists +import com.netflix.frigga.Names +import com.netflix.spectator.api.Registry +import com.netflix.spinnaker.cats.agent.RunnableAgent +import com.netflix.spinnaker.clouddriver.aws.provider.AwsProvider +import com.netflix.spinnaker.clouddriver.aws.security.AmazonClientProvider +import com.netflix.spinnaker.clouddriver.aws.security.NetflixAmazonCredentials +import com.netflix.spinnaker.clouddriver.cache.CustomScheduledAgent +import com.netflix.spinnaker.clouddriver.elasticsearch.ElasticSearchClient +import com.netflix.spinnaker.clouddriver.elasticsearch.model.* +import com.netflix.spinnaker.kork.core.RetrySupport +import org.slf4j.LoggerFactory + +import java.util.concurrent.TimeUnit +import java.util.concurrent.atomic.AtomicInteger + +class ElasticSearchAmazonServerGroupCachingAgent( + val retrySupport: RetrySupport, + val registry: Registry, + val amazonClientProvider: AmazonClientProvider, + val accounts: Collection, + val elasticSearchClient: ElasticSearchClient +) : RunnableAgent, CustomScheduledAgent { + + private val log = LoggerFactory.getLogger(ElasticSearchAmazonServerGroupCachingAgent::class.java) + + override fun getPollIntervalMillis(): Long { + return TimeUnit.SECONDS.toMillis(180) + } + + override fun getTimeoutMillis(): Long { + return TimeUnit.MINUTES.toMillis(5) + } + + override fun getAgentType(): String { + return ElasticSearchAmazonServerGroupCachingAgent::class.java.simpleName + } + + override fun getProviderName(): String { + return AwsProvider.PROVIDER_NAME + } + + override fun run() { + val prefix = "aws_server_groups" + val previousIndexes = elasticSearchClient.getPreviousIndexes(prefix) + val index = elasticSearchClient.createIndex(prefix) + + val serverGroupCount = AtomicInteger(0) + + for (credentials in accounts) { + for (region in credentials.regions) { + val serverGroupModels = fetchServerGroupModels(credentials, region.name) + + serverGroupCount.addAndGet(serverGroupModels.size) + + for (partition in Lists.partition(serverGroupModels, 1000)) { + retrySupport.retry( + { elasticSearchClient.store(index, ModelType.ServerGroup, partition) }, + 10, + 2000, + false + ) + } + } + } + + log.info("Total # of server groups: $serverGroupCount") + + elasticSearchClient.createAlias(index, prefix) + for (previousIndex in previousIndexes) { + elasticSearchClient.deleteIndex(previousIndex) + } + } + + private fun fetchServerGroupModels(credentials: NetflixAmazonCredentials, region: String): List { + val amazonAutoScaling = amazonClientProvider.getAutoScaling(credentials, region) + + log.debug("Describing All Autoscaling Groups in ${credentials.name}:${region}") + val autoScalingGroups = fetchAllAutoScalingGroups(amazonAutoScaling) + + log.debug("Describing All Launch Configurations in ${credentials.name}:${region}") + val launchConfigurationsByName = fetchAllLaunchConfigurations(amazonAutoScaling) + .map { it.launchConfigurationName.toLowerCase() to it } + .toMap() + + return autoScalingGroups.map { asg -> + val launchConfiguration = launchConfigurationsByName.getOrDefault( + asg.launchConfigurationName?.toLowerCase(), + LaunchConfiguration() + ) + + val blockDeviceType = when { + launchConfiguration.blockDeviceMappings.isEmpty() -> "none" + launchConfiguration.blockDeviceMappings[0].ebs != null -> "ebs" + else -> "ephemeral" + } + + val instanceTypes = listOfNotNull(launchConfiguration.instanceType).map { InstanceTypeModel(it) } + + ServerGroupModel( + "${credentials.accountId}:${region}:${asg.autoScalingGroupName}".toLowerCase(), + asg.autoScalingGroupName, + Names.parseName(asg.autoScalingGroupName).toMoniker(), + LocationModel("region", region), + AccountModel(credentials.accountId, credentials.name), + instanceTypes, + BlockDeviceModel(blockDeviceType) + ) + } + } + + private fun fetchAllAutoScalingGroups(amazonAutoScaling: AmazonAutoScaling): List { + val autoScalingGroups = mutableListOf() + + var request = DescribeAutoScalingGroupsRequest() + while (true) { + val response = amazonAutoScaling.describeAutoScalingGroups(request) + + autoScalingGroups.addAll(response.autoScalingGroups) + if (response.nextToken != null) { + request = request.withNextToken(response.nextToken) + } else { + break + } + } + + return autoScalingGroups; + } + + private fun fetchAllLaunchConfigurations(amazonAutoScaling: AmazonAutoScaling): List { + val launchConfigurations = mutableListOf() + + var request = DescribeLaunchConfigurationsRequest() + while (true) { + val response = amazonAutoScaling.describeLaunchConfigurations(request) + + launchConfigurations.addAll(response.launchConfigurations) + if (response.nextToken != null) { + request = request.withNextToken(response.nextToken) + } else { + break + } + } + + return launchConfigurations; + } + + fun Names.toMoniker() = Moniker(this.app, this.stack, this.detail, this.cluster) +} diff --git a/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/model/ElasticSearchModels.kt b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/model/ElasticSearchModels.kt new file mode 100644 index 00000000000..4d4b755751f --- /dev/null +++ b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/clouddriver/elasticsearch/model/ElasticSearchModels.kt @@ -0,0 +1,59 @@ +/* + * Copyright 2018 Netflix, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.elasticsearch.model + +interface Model { + val id: String +} + +enum class ModelType { + ServerGroup, + Intance +} + +data class LocationModel(val type: String, val value: String) + +data class AccountModel(val id: String, val name: String) + +data class InstanceTypeModel(val type: String) + +data class BlockDeviceModel(val type: String) + +data class TagModel(val key: String, val value: Any) + +data class Moniker(val application: String, + val stack: String?, + val details: String?, + val cluster: String) + + +data class ServerGroupModel(override val id: String, + val name: String, + val moniker: Moniker, + val location: LocationModel, + val account: AccountModel, + val instanceTypes: Collection, + val blockDevice: BlockDeviceModel) : Model + +data class InstanceModel(override val id: String, + val instanceId: String, + val instanceType: InstanceTypeModel, + val location: LocationModel, + val account: AccountModel, + val ipAddresses: Collection, + val launchTime: java.util.Date, + val tags: Collection) : Model diff --git a/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/config/ElasticSearchAmazonConfig.kt b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/config/ElasticSearchAmazonConfig.kt new file mode 100644 index 00000000000..4c70ee521aa --- /dev/null +++ b/clouddriver-elasticsearch-aws/src/main/kotlin/com/netflix/spinnaker/config/ElasticSearchAmazonConfig.kt @@ -0,0 +1,48 @@ +/* + * Copyright 2018 Netflix, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.config + +import com.fasterxml.jackson.databind.ObjectMapper +import com.netflix.spectator.api.Registry +import com.netflix.spinnaker.clouddriver.aws.security.AmazonClientProvider +import com.netflix.spinnaker.clouddriver.elasticsearch.aws.ElasticSearchAmazonCachingAgentProvider +import com.netflix.spinnaker.clouddriver.security.AccountCredentialsProvider +import com.netflix.spinnaker.kork.core.RetrySupport +import io.searchbox.client.JestClient +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty +import org.springframework.context.annotation.Bean +import org.springframework.context.annotation.Configuration + +@Configuration +@ConditionalOnProperty("elasticSearch.caching.enabled") +open class ElasticSearchAmazonConfig { + @Bean + open fun elasticSearchAmazonCachingAgentProvider(objectMapper: ObjectMapper, + jestClient: JestClient, + retrySupport: RetrySupport, + registry: Registry, + amazonClientProvider: AmazonClientProvider, + accountCredentialsProvider: AccountCredentialsProvider) = + ElasticSearchAmazonCachingAgentProvider( + objectMapper, + jestClient, + retrySupport, + registry, + amazonClientProvider, + accountCredentialsProvider + ) +} diff --git a/clouddriver-google/src/main/groovy/com/netflix/spinnaker/clouddriver/google/deploy/GCEUtil.groovy b/clouddriver-google/src/main/groovy/com/netflix/spinnaker/clouddriver/google/deploy/GCEUtil.groovy index 1470e9f663f..24832dc5070 100644 --- a/clouddriver-google/src/main/groovy/com/netflix/spinnaker/clouddriver/google/deploy/GCEUtil.groovy +++ b/clouddriver-google/src/main/groovy/com/netflix/spinnaker/clouddriver/google/deploy/GCEUtil.groovy @@ -1766,7 +1766,7 @@ class GCEUtil { while (!executedAtLeastOnce || nextPageToken) { HttpsHealthCheckList httpsHealthCheckList = agent.timeExecute( compute.httpsHealthChecks().list(project).setPageToken(nextPageToken), - "compute.httpsHealtchChecks.list", + "compute.httpsHealthChecks.list", agent.TAG_SCOPE, agent.SCOPE_GLOBAL) executedAtLeastOnce = true @@ -1783,7 +1783,7 @@ class GCEUtil { while (!executedAtLeastOnce || nextPageToken) { HealthCheckList healthCheckList = agent.timeExecute( compute.healthChecks().list(project).setPageToken(nextPageToken), - "compute.healtchChecks.list", + "compute.healthChecks.list", agent.TAG_SCOPE, agent.SCOPE_GLOBAL) executedAtLeastOnce = true diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/config/KubernetesConfigurationProperties.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/config/KubernetesConfigurationProperties.groovy index afcbd12f7fd..61666269f46 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/config/KubernetesConfigurationProperties.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/config/KubernetesConfigurationProperties.groovy @@ -42,7 +42,8 @@ class KubernetesConfigurationProperties { List dockerRegistries List requiredGroupMembership Permissions.Builder permissions = new Permissions.Builder() - Boolean debug = false; + String namingStrategy = "kubernetesAnnotations" + Boolean debug = false } List accounts = [] diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentials.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentials.java index a67be6472a9..d9e16768233 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentials.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentials.java @@ -29,6 +29,7 @@ import com.netflix.spinnaker.clouddriver.security.AccountCredentialsRepository; import com.netflix.spinnaker.clouddriver.security.ProviderVersion; import com.netflix.spinnaker.fiat.model.resources.Permissions; +import com.netflix.spinnaker.moniker.Namer; import groovy.util.logging.Slf4j; import org.apache.commons.lang3.StringUtils; @@ -49,6 +50,7 @@ public class KubernetesNamedAccountCredentials final private String user; final private String userAgent; final private String kubeconfigFile; + final private String kubectlExecutable; final private Boolean serviceAccount; private List namespaces; private List omitNamespaces; @@ -70,6 +72,7 @@ public class KubernetesNamedAccountCredentials String cluster, String user, String kubeconfigFile, + String kubectlExecutable, Boolean serviceAccount, List namespaces, List omitNamespaces, @@ -88,6 +91,7 @@ public class KubernetesNamedAccountCredentials this.user = user; this.userAgent = userAgent; this.kubeconfigFile = kubeconfigFile; + this.kubectlExecutable = kubectlExecutable; this.serviceAccount = serviceAccount; this.namespaces = namespaces; this.omitNamespaces = omitNamespaces; @@ -129,6 +133,10 @@ public C getCredentials() { return credentials; } + public String getKubectlExecutable() { + return kubectlExecutable; + } + @Override public String getCloudProvider() { return cloudProvider; @@ -163,6 +171,7 @@ static class Builder { String user; String userAgent; String kubeconfigFile; + String kubectlExecutable; Boolean serviceAccount; Boolean configureImagePullSecrets; List namespaces; @@ -175,6 +184,7 @@ static class Builder { Registry spectatorRegistry; AccountCredentialsRepository accountCredentialsRepository; KubectlJobExecutor jobExecutor; + Namer namer; boolean debug; Builder name(String name) { @@ -232,8 +242,13 @@ Builder kubeconfigFile(String kubeconfigFile) { return this; } + Builder kubectlExecutable(String kubectlExecutable) { + this.kubectlExecutable = kubectlExecutable; + return this; + } + Builder serviceAccount(Boolean serviceAccount) { - this.serviceAccount = serviceAccount;; + this.serviceAccount = serviceAccount; return this; } @@ -300,6 +315,11 @@ Builder debug(boolean debug) { return this; } + Builder namer(Namer namer) { + this.namer = namer; + return this; + } + private C buildCredentials() { switch (providerVersion) { case v1: @@ -322,10 +342,11 @@ private C buildCredentials() { NamerRegistry.lookup() .withProvider(KubernetesCloudProvider.getID()) .withAccount(name) - .setNamer(KubernetesManifest.class, new KubernetesManifestNamer()); + .setNamer(KubernetesManifest.class, namer); return (C) new KubernetesV2Credentials.Builder() .accountName(name) .kubeconfigFile(kubeconfigFile) + .kubectlExecutable(kubectlExecutable) .context(context) .oAuthServiceAccount(oAuthServiceAccount) .oAuthScopes(oAuthScopes) @@ -387,6 +408,7 @@ KubernetesNamedAccountCredentials build() { cluster, user, kubeconfigFile, + kubectlExecutable, serviceAccount, namespaces, omitNamespaces, diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentialsInitializer.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentialsInitializer.groovy index cf269baa8b9..b0c6192ae62 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentialsInitializer.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/security/KubernetesNamedAccountCredentialsInitializer.groovy @@ -21,6 +21,7 @@ import com.netflix.spinnaker.cats.module.CatsModule import com.netflix.spinnaker.cats.provider.ProviderSynchronizerTypeWrapper import com.netflix.spinnaker.clouddriver.kubernetes.config.KubernetesConfigurationProperties import com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor +import com.netflix.spinnaker.clouddriver.names.NamerRegistry import com.netflix.spinnaker.clouddriver.security.AccountCredentialsRepository import com.netflix.spinnaker.clouddriver.security.CredentialsInitializerSynchronizable import com.netflix.spinnaker.clouddriver.security.ProviderUtils @@ -39,6 +40,7 @@ class KubernetesNamedAccountCredentialsInitializer implements CredentialsInitial @Autowired Registry spectatorRegistry @Autowired KubectlJobExecutor jobExecutor + @Autowired NamerRegistry namerRegistry @Bean List kubernetesNamedAccountCredentials( @@ -96,6 +98,7 @@ class KubernetesNamedAccountCredentialsInitializer implements CredentialsInitial .permissions(managedAccount.permissions.build()) .spectatorRegistry(spectatorRegistry) .jobExecutor(jobExecutor) + .namer(namerRegistry.getNamingStrategy(managedAccount.namingStrategy)) .debug(managedAccount.debug) .build() diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiAdaptor.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiAdaptor.groovy index a4fedfce18a..b73941e4762 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiAdaptor.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiAdaptor.groovy @@ -498,6 +498,22 @@ class KubernetesApiAdaptor { } } + ReplicaSet annotateReplicaSet(String namespace, String name, String key, String value) { + exceptionWrapper("replicaSets.annotate", "Annotate replica set $name", namespace) { + def rs = client.extensions().replicaSets().inNamespace(namespace).withName(name).edit() + rs.buildMetadata().annotations?.put(key, value) + return rs.done() + } + } + + ReplicationController annotateReplicationController(String namespace, String name, String key, String value) { + exceptionWrapper("replicationControllers.annotate", "Annotate replication controller $name", namespace) { + def rc = client.replicationControllers().inNamespace(namespace).withName(name).edit() + rc.buildMetadata().annotations?.put(key, value) + return rc.done() + } + } + boolean deleteDeployment(String namespace, String name) { exceptionWrapper("deployments.delete", "Delete Deployment $name", namespace) { client.extensions().deployments().inNamespace(namespace).withName(name).delete() diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiConverter.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiConverter.groovy index 083cb7fa9fe..14278eb74cc 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiConverter.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesApiConverter.groovy @@ -417,6 +417,7 @@ class KubernetesApiConverter { return res.withMountPath(mount.mountPath) .withName(mount.name) .withReadOnly(mount.readOnly) + .withSubPath(mount.subPath) .build() } @@ -616,7 +617,12 @@ class KubernetesApiConverter { } - null containerDescription.volumeMounts = container?.volumeMounts?.collect { volumeMount -> - new KubernetesVolumeMount(name: volumeMount.name, readOnly: volumeMount.readOnly, mountPath: volumeMount.mountPath) + new KubernetesVolumeMount( + name: volumeMount.name, + readOnly: volumeMount.readOnly, + mountPath: volumeMount.mountPath, + subPath: volumeMount.subPath + ) } containerDescription.args = container?.args ?: [] diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesClientApiConverter.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesClientApiConverter.groovy index 4ce04e7dd10..3320573d64c 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesClientApiConverter.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/api/KubernetesClientApiConverter.groovy @@ -198,7 +198,12 @@ class KubernetesClientApiConverter { containerDescription.readinessProbe = fromV1Probe(container?.readinessProbe) containerDescription.volumeMounts = container?.volumeMounts?.collect { volumeMount -> - new KubernetesVolumeMount(name: volumeMount.name, readOnly: volumeMount.readOnly, mountPath: volumeMount.mountPath) + new KubernetesVolumeMount( + name: volumeMount.name, + readOnly: volumeMount.readOnly, + mountPath: volumeMount.mountPath, + subPath: volumeMount.subPath + ) } containerDescription.args = container?.args ?: [] @@ -746,6 +751,7 @@ class KubernetesClientApiConverter { res.name = mount.name res.mountPath = mount.mountPath res.readOnly = mount.readOnly + res.subPath = mount.subPath volumeMounts.add(res) } v1container.volumeMounts = volumeMounts diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/KubernetesUtil.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/KubernetesUtil.groovy index 24b4b139e1b..82a2b6edead 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/KubernetesUtil.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/KubernetesUtil.groovy @@ -42,6 +42,8 @@ class KubernetesUtil { private static int SECURITY_GROUP_LABEL_PREFIX_LENGTH = SECURITY_GROUP_LABEL_PREFIX.length() private static int LOAD_BALANCER_LABEL_PREFIX_LENGTH = LOAD_BALANCER_LABEL_PREFIX.length() + static String ENABLE_DISABLE_ANNOTATION = "service.spinnaker.io/enabled" + static String getNextSequence(String clusterName, String namespace, KubernetesV1Credentials credentials) { def maxSeqNumber = -1 def replicationControllers = credentials.apiAdaptor.getReplicationControllers(namespace) diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/description/servergroup/DeployKubernetesAtomicOperationDescription.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/description/servergroup/DeployKubernetesAtomicOperationDescription.groovy index 54ea4a20962..32e84bc0fed 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/description/servergroup/DeployKubernetesAtomicOperationDescription.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/description/servergroup/DeployKubernetesAtomicOperationDescription.groovy @@ -260,6 +260,7 @@ class KubernetesVolumeMount { String name Boolean readOnly String mountPath + String subPath } enum KubernetesVolumeSourceType { diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/ops/servergroup/AbstractEnableDisableKubernetesAtomicOperation.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/ops/servergroup/AbstractEnableDisableKubernetesAtomicOperation.groovy index e8554f0d835..df021f1c960 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/ops/servergroup/AbstractEnableDisableKubernetesAtomicOperation.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/deploy/ops/servergroup/AbstractEnableDisableKubernetesAtomicOperation.groovy @@ -88,6 +88,7 @@ abstract class AbstractEnableDisableKubernetesAtomicOperation implements AtomicO def getGeneration = null def getResource = null def desired = null + def disableAnnotation = null if (replicationController) { desired = credentials.apiAdaptor.toggleReplicationControllerSpecLabels(namespace, description.serverGroupName, services, action) getGeneration = { ReplicationController rc -> @@ -96,6 +97,9 @@ abstract class AbstractEnableDisableKubernetesAtomicOperation implements AtomicO getResource = { return credentials.apiAdaptor.getReplicationController(namespace, description.serverGroupName) } + disableAnnotation = { -> + return credentials.apiAdaptor.annotateReplicationController(namespace, description.serverGroupName, KubernetesUtil.ENABLE_DISABLE_ANNOTATION, action) + } } else if (replicaSet) { desired = credentials.apiAdaptor.toggleReplicaSetSpecLabels(namespace, description.serverGroupName, services, action) getGeneration = { ReplicaSet rs -> @@ -104,17 +108,25 @@ abstract class AbstractEnableDisableKubernetesAtomicOperation implements AtomicO getResource = { return credentials.apiAdaptor.getReplicaSet(namespace, description.serverGroupName) } + disableAnnotation = { -> + return credentials.apiAdaptor.annotateReplicaSet(namespace, description.serverGroupName, KubernetesUtil.ENABLE_DISABLE_ANNOTATION, action) + } } else { throw new KubernetesOperationException("No replication controller or replica set $description.serverGroupName in $namespace.") } if (!credentials.apiAdaptor.blockUntilResourceConsistent(desired, getGeneration, getResource)) { - throw new KubernetesOperationException("Server group failed to reach a consistent state. This is likely a bug with Kubernetes itself.") + throw new KubernetesOperationException("Server group failed to reach a consistent state while waiting for label to be applied. This is likely a bug with Kubernetes itself.") + } + + if (!credentials.apiAdaptor.blockUntilResourceConsistent(disableAnnotation(), getGeneration, getResource)) { + throw new KubernetesOperationException("Server group failed to reach a consistent state while waiting for annotation be applied. This is likely a bug with Kubernetes itself.") } } - if (!replicationController && !replicaSet ) + if (!replicationController && !replicaSet ) { throw new KubernetesOperationException("No replication controller or replica set $description.serverGroupName in $namespace.") + } KubernetesV1ServerGroup serverGroup = clusterProviders.getServerGroup(description.account, namespace, description.serverGroupName) serverGroup.instances.forEach( { instance -> pods.add(instance.getPod())}) diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesJobStatus.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesJobStatus.groovy index 2ada455cce4..956d9bd9486 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesJobStatus.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesJobStatus.groovy @@ -83,6 +83,11 @@ class KubernetesJobStatus implements JobStatus, Serializable { message = terminated.getMessage() reason = terminated.getReason() + // Kind of a hack, seems that jobs can have exit code 0 even when being OOMKilled + if (reason.equalsIgnoreCase("oomkilled")) { + return JobState.Failed + } + if (exitCode == 0) { return JobState.Succeeded } else { diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesV1ServerGroup.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesV1ServerGroup.groovy index 98ffdeadfd4..3ee61c93e6d 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesV1ServerGroup.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/model/KubernetesV1ServerGroup.groovy @@ -37,6 +37,8 @@ import io.fabric8.kubernetes.client.internal.SerializationUtils import io.kubernetes.client.models.V1beta1DaemonSet import io.kubernetes.client.models.V1beta1StatefulSet +import static com.netflix.spinnaker.clouddriver.kubernetes.v1.deploy.KubernetesUtil.ENABLE_DISABLE_ANNOTATION + @CompileStatic @EqualsAndHashCode(includes = ["name", "namespace", "account"]) class KubernetesV1ServerGroup implements ServerGroup, Serializable { @@ -55,6 +57,7 @@ class KubernetesV1ServerGroup implements ServerGroup, Serializable { Set securityGroups = [] as Set Map launchConfig Map labels = [:] + Map annotations = [:] DeployKubernetesAtomicOperationDescription deployDescription KubernetesAutoscalerStatus autoscalerStatus KubernetesDeploymentStatus deploymentStatus @@ -92,7 +95,7 @@ class KubernetesV1ServerGroup implements ServerGroup, Serializable { if (labels) { def lbCount = labels.count { key, value -> KubernetesUtil.isLoadBalancerLabel(key) } if (lbCount == 0) { - return false + return annotations?.get(ENABLE_DISABLE_ANNOTATION) == "false" } def enabledCount = labels.count { key, value -> KubernetesUtil.isLoadBalancerLabel(key) && value == "true" } @@ -162,6 +165,7 @@ class KubernetesV1ServerGroup implements ServerGroup, Serializable { this.deployDescription = KubernetesApiConverter.fromReplicaSet(replicaSet) this.yaml = SerializationUtils.dumpWithoutRuntimeStateAsYaml(replicaSet) this.kind = replicaSet.kind + this.annotations = replicaSet.metadata?.annotations this.events = events?.collect { new KubernetesEvent(it) } @@ -187,6 +191,7 @@ class KubernetesV1ServerGroup implements ServerGroup, Serializable { this.deployDescription = KubernetesApiConverter.fromReplicationController(replicationController) this.yaml = SerializationUtils.dumpWithoutRuntimeStateAsYaml(replicationController) this.kind = replicationController.kind + this.annotations = replicationController.metadata?.annotations this.events = events?.collect { new KubernetesEvent(it) } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesControllersCachingAgent.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesControllersCachingAgent.groovy new file mode 100644 index 00000000000..04ce778688b --- /dev/null +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesControllersCachingAgent.groovy @@ -0,0 +1,424 @@ +/* + * Copyright 2017 Cisco, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License") + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.agent + +import com.fasterxml.jackson.core.type.TypeReference +import com.fasterxml.jackson.databind.ObjectMapper +import com.netflix.frigga.Names +import com.netflix.spectator.api.Registry +import com.netflix.spinnaker.cats.agent.AgentDataType +import com.netflix.spinnaker.cats.agent.CacheResult +import com.netflix.spinnaker.cats.agent.DefaultCacheResult +import com.netflix.spinnaker.cats.cache.CacheData +import com.netflix.spinnaker.cats.cache.DefaultCacheData +import com.netflix.spinnaker.cats.provider.ProviderCache +import com.netflix.spinnaker.clouddriver.cache.OnDemandAgent +import com.netflix.spinnaker.clouddriver.cache.OnDemandMetricsSupport +import com.netflix.spinnaker.clouddriver.kubernetes.KubernetesCloudProvider +import com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesNamedAccountCredentials +import com.netflix.spinnaker.clouddriver.kubernetes.v1.caching.Keys +import com.netflix.spinnaker.clouddriver.kubernetes.v1.deploy.KubernetesUtil +import com.netflix.spinnaker.clouddriver.kubernetes.v1.model.KubernetesV1ServerGroup +import com.netflix.spinnaker.clouddriver.kubernetes.v1.provider.view.MutableCacheData +import com.netflix.spinnaker.clouddriver.kubernetes.v1.security.KubernetesV1Credentials +import groovy.util.logging.Slf4j +import io.fabric8.kubernetes.api.model.Event +import io.kubernetes.client.models.V1PodList +import io.kubernetes.client.models.V1beta1DaemonSet +import io.kubernetes.client.models.V1beta1StatefulSet + +import static com.netflix.spinnaker.cats.agent.AgentDataType.Authority.AUTHORITATIVE +import static com.netflix.spinnaker.cats.agent.AgentDataType.Authority.INFORMATIVE + +@Slf4j +class KubernetesControllersCachingAgent extends KubernetesV1CachingAgent implements OnDemandAgent { + final String category = 'serverGroup' + final OnDemandMetricsSupport metricsSupport + + static final Set types = Collections.unmodifiableSet([ + AUTHORITATIVE.forType(Keys.Namespace.APPLICATIONS.ns), + AUTHORITATIVE.forType(Keys.Namespace.CLUSTERS.ns), + INFORMATIVE.forType(Keys.Namespace.LOAD_BALANCERS.ns), + AUTHORITATIVE.forType(Keys.Namespace.SERVER_GROUPS.ns), + INFORMATIVE.forType(Keys.Namespace.INSTANCES.ns), + ] as Set) + + KubernetesControllersCachingAgent(KubernetesNamedAccountCredentials namedAccountCredentials, + ObjectMapper objectMapper, + Registry registry, + int agentIndex, + int agentCount) { + super(namedAccountCredentials, objectMapper, registry, agentIndex, agentCount) + this.metricsSupport = new OnDemandMetricsSupport(registry, this, "$KubernetesCloudProvider.ID:$OnDemandAgent.OnDemandType.ServerGroup") + } + + @Override + Collection getProvidedDataTypes() { + return types + } + + @Override + String getOnDemandAgentType() { + "${getAgentType()}-OnDemand" + } + + @Override + OnDemandMetricsSupport getMetricsSupport() { + return null + } + + @Override + boolean handles(OnDemandAgent.OnDemandType type, String cloudProvider) { + OnDemandAgent.OnDemandType.ServerGroup == type && cloudProvider == KubernetesCloudProvider.ID + } + + @Override + OnDemandAgent.OnDemandResult handle(ProviderCache providerCache, Map data) { + if (!data.containsKey("serverGroupName")) { + return null + } + + if (data.account != accountName) { + return null + } + + reloadNamespaces() + String namespace = data.region + if (!namespaces.contains(namespace)) { + return null + } + + def serverGroupName = data.serverGroupName.toString() + + V1beta1StatefulSet statefulSet = metricsSupport.readData { + loadStatefulSet(namespace, serverGroupName) + } + + V1beta1DaemonSet daemonSet = metricsSupport.readData { + loadDaemonSet(namespace, serverGroupName) + } + + CacheResult result = metricsSupport.transformData { + buildCacheResult([new KubernetesController(statefulController: statefulSet, daemonController: daemonSet)], [:], [], Long.MAX_VALUE) + } + + def jsonResult = objectMapper.writeValueAsString(result.cacheResults) + boolean isControllerSetCachingAgentType = true + if (result.cacheResults.values().flatten().isEmpty()) { + + // Determine if this is the correct agent to delete cache which can avoid double deletion + CacheData serverGroup = providerCache.get(Keys.Namespace.SERVER_GROUPS.ns, Keys.getServerGroupKey(accountName, namespace, serverGroupName)) + + if (serverGroup) { + String kind = serverGroup.attributes?.get("serverGroup")?.get("kind") + if (kind == "StatefulSet" || kind == "DaemonSet") { + // Avoid writing an empty onDemand cache record (instead delete any that may have previously existed). + providerCache.evictDeletedItems(Keys.Namespace.ON_DEMAND.ns, [Keys.getServerGroupKey(accountName, namespace, serverGroupName)]) + }else{ + isControllerSetCachingAgentType = false + } + } + } else { + metricsSupport.onDemandStore { + def cacheData = new DefaultCacheData( + Keys.getServerGroupKey(accountName, namespace, serverGroupName), + 10 * 60, // ttl is 10 minutes + [ + cacheTime: System.currentTimeMillis(), + cacheResults: jsonResult, + processedCount: 0, + processedTime: null + ], + [:] + ) + providerCache.putCacheData(Keys.Namespace.ON_DEMAND.ns, cacheData) + } + } + + // Evict this server group if it no longer exists. + Map> evictions + if (isControllerSetCachingAgentType) { + evictions = statefulSet || daemonSet ? [:] : [ + (Keys.Namespace.SERVER_GROUPS.ns): [ + Keys.getServerGroupKey(accountName, namespace, serverGroupName) + ] + ] + } + + log.info("On demand cache refresh (data: ${data}) succeeded.") + + return new OnDemandAgent.OnDemandResult( + sourceAgentType: getOnDemandAgentType(), + cacheResult: result, + evictions: evictions + ) + } + + @Override + Collection pendingOnDemandRequests(ProviderCache providerCache) { + def keys = providerCache.getIdentifiers(Keys.Namespace.ON_DEMAND.ns) + keys = keys.findResults { + def parse = Keys.parse(it) + if (parse && namespaces.contains(parse.namespace) && parse.account == accountName) { + return it + } else { + return null + } + } + + def keyCount = keys.size() + def be = keyCount == 1 ? "is" : "are" + def pluralize = keyCount == 1 ? "" : "s" + log.info("There $be $keyCount pending on demand request$pluralize") + + providerCache.getAll(Keys.Namespace.ON_DEMAND.ns, keys).collect { + def details = Keys.parse(it.id) + + return [ + details : details, + moniker : convertOnDemandDetails(details), + cacheTime : it.attributes.cacheTime, + processedCount: it.attributes.processedCount, + processedTime : it.attributes.processedTime + ] + } + } + + /** + * Triggered by an AgentScheduler to tell this Agent to load its data. + * + * @param providerCache Cache associated with this Agent's provider + * @return the complete set of data for this Agent. + */ + @Override + CacheResult loadData(ProviderCache providerCache) { + reloadNamespaces() + Long start = System.currentTimeMillis() + List statefulSet = loadStatefulSets() + List daemonSet = loadDaemonSets() + List serverGroups = (statefulSet.collect { + it ? new KubernetesController(statefulController: it) : null + }+ daemonSet.collect { + it ? new KubernetesController(daemonController: it) : null + } + ) - null + List evictFromOnDemand = [] + List keepInOnDemand = [] + providerCache.getAll(Keys.Namespace.ON_DEMAND.ns, + serverGroups.collect { serverGroup -> + Keys.getServerGroupKey(accountName, serverGroup.namespace, serverGroup.name) + }) + .each { CacheData onDemandEntry -> + // Ensure that we don't overwrite data that was inserted by the `handle` method while we retrieved the + // replication controllers. Furthermore, cache data that hasn't been processed needs to be updated in the ON_DEMAND + // cache, so don't evict data without a processedCount > 0. + if (onDemandEntry.attributes.cacheTime < start && onDemandEntry.attributes.processedCount > 0) { + evictFromOnDemand << onDemandEntry + } else { + keepInOnDemand << onDemandEntry + } + } + + def result = buildCacheResult(serverGroups, keepInOnDemand.collectEntries { CacheData onDemandEntry -> + [(onDemandEntry.id): onDemandEntry] + }, evictFromOnDemand*.id, start) + + result.cacheResults[Keys.Namespace.ON_DEMAND.ns].each { CacheData onDemandEntry -> + onDemandEntry.attributes.processedTime = System.currentTimeMillis() + onDemandEntry.attributes.processedCount = (onDemandEntry.attributes.processedCount ?: 0) + 1 + } + + return result + } + + List loadStatefulSets() { + namespaces.collect { String namespace -> + credentials.apiClientAdaptor.getStatefulSets(namespace) + }.flatten() + } + + List loadDaemonSets() { + namespaces.collect { String namespace -> + credentials.apiClientAdaptor.getDaemonSets(namespace) + }.flatten() + } + + V1PodList loadPods(KubernetesController serverGroup) { + credentials.apiClientAdaptor.getPods(serverGroup.namespace, serverGroup.selector) + } + + V1beta1StatefulSet loadStatefulSet(String namespace, String name) { + credentials.apiClientAdaptor.getStatefulSet(name, namespace) + } + + V1beta1DaemonSet loadDaemonSet(String namespace, String name) { + credentials.apiClientAdaptor.getDaemonSet(name, namespace) + } + + private CacheResult buildCacheResult(List serverGroups, Map onDemandKeep, List onDemandEvict, Long start) { + log.info("Describing items in ${agentType}") + + Map cachedApplications = MutableCacheData.mutableCacheMap() + Map cachedClusters = MutableCacheData.mutableCacheMap() + Map cachedServerGroups = MutableCacheData.mutableCacheMap() + Map cachedInstances = MutableCacheData.mutableCacheMap() + Map cachedLoadBalancers = MutableCacheData.mutableCacheMap() + + Map> stateFulsetEvents = [:].withDefault { _ -> [:] } + Map> daemonsetEvents = [:].withDefault { _ -> [:] } + + try { + namespaces.each { String namespace -> + stateFulsetEvents[namespace] = credentials.apiAdaptor.getEvents(namespace, "V1beta1StatefulSet") + daemonsetEvents[namespace] = credentials.apiAdaptor.getEvents(namespace, "V1beta1DaemonSet") + } + } catch (Exception e) { + log.warn "Failure fetching events for all server groups in $namespaces", e + } + + for (KubernetesController serverGroup: serverGroups) { + if (!serverGroup.exists()) { + continue + } + + def onDemandData = onDemandKeep ? onDemandKeep[Keys.getServerGroupKey(accountName, serverGroup.namespace, serverGroup.name)] : null + + if (onDemandData && onDemandData.attributes.cacheTime >= start) { + Map> cacheResults = objectMapper.readValue(onDemandData.attributes.cacheResults as String, + new TypeReference>>() { }) + cache(cacheResults, Keys.Namespace.APPLICATIONS.ns, cachedApplications) + cache(cacheResults, Keys.Namespace.CLUSTERS.ns, cachedClusters) + cache(cacheResults, Keys.Namespace.SERVER_GROUPS.ns, cachedServerGroups) + cache(cacheResults, Keys.Namespace.INSTANCES.ns, cachedInstances) + } else { + def serverGroupName = serverGroup.name + def pods = loadPods(serverGroup) + def names = Names.parseName(serverGroupName) + def applicationName = names.app + def clusterName = names.cluster + def serverGroupKey = Keys.getServerGroupKey(accountName, serverGroup.namespace, serverGroupName) + def applicationKey = Keys.getApplicationKey(applicationName) + def clusterKey = Keys.getClusterKey(accountName, applicationName, category, clusterName) + def instanceKeys = [] + def loadBalancerKeys = serverGroup.loadBalancers.collect({ + Keys.getLoadBalancerKey(accountName, serverGroup.namespace, it) + }) + cachedApplications[applicationKey].with { + attributes.name = applicationName + relationships[Keys.Namespace.CLUSTERS.ns].add(clusterKey) + relationships[Keys.Namespace.SERVER_GROUPS.ns].add(serverGroupKey) + relationships[Keys.Namespace.LOAD_BALANCERS.ns].addAll(loadBalancerKeys) + } + + cachedClusters[clusterKey].with { + attributes.name = clusterName + relationships[Keys.Namespace.APPLICATIONS.ns].add(applicationKey) + relationships[Keys.Namespace.SERVER_GROUPS.ns].add(serverGroupKey) + relationships[Keys.Namespace.LOAD_BALANCERS.ns].addAll(loadBalancerKeys) + } + pods?.getItems().forEach { pod -> + def key = Keys.getInstanceKey(accountName, pod.metadata.namespace, pod.metadata.name) + instanceKeys << key + cachedInstances[key].with { + relationships[Keys.Namespace.APPLICATIONS.ns].add(applicationKey) + relationships[Keys.Namespace.CLUSTERS.ns].add(clusterKey) + relationships[Keys.Namespace.SERVER_GROUPS.ns].add(serverGroupKey) + relationships[Keys.Namespace.LOAD_BALANCERS.ns].addAll(loadBalancerKeys) + } + } + boolean isDaemonset + cachedServerGroups[serverGroupKey].with { + def events = null + attributes.name = serverGroupName + + if (serverGroup.statefulController instanceof V1beta1StatefulSet) { + events = stateFulsetEvents[serverGroup.namespace][serverGroupName] + } else if (serverGroup.daemonController instanceof V1beta1DaemonSet) { + events = daemonsetEvents[serverGroup.namespace][serverGroupName] + isDaemonset = true + } + attributes.serverGroup = new KubernetesV1ServerGroup(serverGroup.statefulController ?: serverGroup.daemonController, accountName, events) + if (isDaemonset) { + attributes.serverGroup.replicas = pods?.getItems().size() + } + + relationships[Keys.Namespace.APPLICATIONS.ns].add(applicationKey) + relationships[Keys.Namespace.CLUSTERS.ns].add(clusterKey) + relationships[Keys.Namespace.INSTANCES.ns].addAll(instanceKeys) + relationships[Keys.Namespace.LOAD_BALANCERS.ns].addAll(loadBalancerKeys) + } + } + } + + log.info("Caching ${cachedApplications.size()} applications in ${agentType}") + log.info("Caching ${cachedClusters.size()} clusters in ${agentType}") + log.info("Caching ${cachedServerGroups.size()} server groups in ${agentType}") + log.info("Caching ${cachedInstances.size()} instances in ${agentType}") + + new DefaultCacheResult([ + (Keys.Namespace.APPLICATIONS.ns): cachedApplications.values(), + (Keys.Namespace.LOAD_BALANCERS.ns): cachedLoadBalancers.values(), + (Keys.Namespace.CLUSTERS.ns): cachedClusters.values(), + (Keys.Namespace.SERVER_GROUPS.ns): cachedServerGroups.values(), + (Keys.Namespace.INSTANCES.ns): cachedInstances.values(), + (Keys.Namespace.ON_DEMAND.ns): onDemandKeep.values() + ],[ + (Keys.Namespace.ON_DEMAND.ns): onDemandEvict, + ]) + + } + + private static void cache(Map> cacheResults, String cacheNamespace, Map cacheDataById) { + cacheResults[cacheNamespace].each { + def existingCacheData = cacheDataById[it.id] + if (existingCacheData) { + existingCacheData.attributes.putAll(it.attributes) + it.relationships.each { String relationshipName, Collection relationships -> + existingCacheData.relationships[relationshipName].addAll(relationships) + } + } else { + cacheDataById[it.id] = it + } + } + } + + static class KubernetesController{ + def statefulController + def daemonController + + String getName() { + statefulController ? statefulController.metadata.name : daemonController.metadata.name + } + + String getNamespace() { + statefulController ? statefulController.metadata.namespace : daemonController.metadata.namespace + } + + Map getSelector() { + statefulController ? statefulController.spec.selector.matchLabels : daemonController.spec.selector.matchLabels + } + + boolean exists() { + statefulController ?: daemonController + } + + List getLoadBalancers() { + statefulController ? KubernetesUtil.getLoadBalancers(statefulController.spec?.template?.metadata?.labels ?: [:]) : + KubernetesUtil.getLoadBalancers(daemonController.spec?.template?.metadata?.labels ?: [:]) + } + } +} diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesServerGroupCachingAgent.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesServerGroupCachingAgent.groovy index 00604d66d00..2902ea3e621 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesServerGroupCachingAgent.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesServerGroupCachingAgent.groovy @@ -110,10 +110,18 @@ class KubernetesServerGroupCachingAgent extends KubernetesV1CachingAgent impleme } def jsonResult = objectMapper.writeValueAsString(result.cacheResults) - + boolean isControllerSetCachingAgentType = false if (result.cacheResults.values().flatten().isEmpty()) { // Avoid writing an empty onDemand cache record (instead delete any that may have previously existed). providerCache.evictDeletedItems(Keys.Namespace.ON_DEMAND.ns, [Keys.getServerGroupKey(accountName, namespace, serverGroupName)]) + // Determine if this is the correct agent to delete cache which can avoid double deletion + CacheData serverGroup = providerCache.get(Keys.Namespace.SERVER_GROUPS.ns, Keys.getServerGroupKey(accountName, namespace, serverGroupName)) + if (serverGroup) { + String kind = serverGroup.attributes?.get("serverGroup")?.get("kind") + if (kind == "StatefulSet" || kind == "DaemonSet") { + isControllerSetCachingAgentType = true + } + } } else { metricsSupport.onDemandStore { def cacheData = new DefaultCacheData( @@ -133,11 +141,15 @@ class KubernetesServerGroupCachingAgent extends KubernetesV1CachingAgent impleme } // Evict this server group if it no longer exists. - Map> evictions = replicationController || replicaSet ? [:] : [ - (Keys.Namespace.SERVER_GROUPS.ns): [ - Keys.getServerGroupKey(accountName, namespace, serverGroupName) + Map> evictions + if (!isControllerSetCachingAgentType) { + evictions = replicationController || replicaSet ? [:] : [ + (Keys.Namespace.SERVER_GROUPS.ns): [ + Keys.getServerGroupKey(accountName, namespace, serverGroupName) + ] ] - ] + } + log.info("On demand cache refresh (data: ${data}) succeeded.") diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesV1CachingAgentDispatcher.groovy b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesV1CachingAgentDispatcher.groovy index cbff5f764cc..d6530cb191c 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesV1CachingAgentDispatcher.groovy +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v1/provider/agent/KubernetesV1CachingAgentDispatcher.groovy @@ -45,6 +45,7 @@ class KubernetesV1CachingAgentDispatcher implements KubernetesCachingAgentDispat agents << new KubernetesServiceAccountCachingAgent(credentials, objectMapper, registry, index, credentials.cacheThreads) agents << new KubernetesConfigMapCachingAgent(credentials, objectMapper, registry, index, credentials.cacheThreads) agents << new KubernetesSecretCachingAgent(credentials, objectMapper, registry, index, credentials.cacheThreads) + agents << new KubernetesControllersCachingAgent(credentials, objectMapper, registry, index, credentials.cacheThreads) } return agents diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacer.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacer.java index 13ac26ac792..ea351303da6 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacer.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacer.java @@ -42,6 +42,8 @@ import java.util.HashSet; import java.util.List; import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -104,6 +106,7 @@ public Set findAll(KubernetesManifest input) { .map(s -> Artifact.builder() .type(r.getType().toString()) .reference(s) + .name(r.getNameFromReference(s)) .build() ); } catch (Exception e) { @@ -122,6 +125,8 @@ public Set findAll(KubernetesManifest input) { public static class Replacer { private final String replacePath; private final String findPath; + private final Pattern namePattern; // the first group should be the artifact name + @Getter private final ArtifactTypes type; @@ -142,6 +147,19 @@ ArrayNode findAll(DocumentContext obj) { return obj.read(findPath); } + String getNameFromReference(String reference) { + if (namePattern == null) { + return null; + } + + Matcher m = namePattern.matcher(reference); + if (m.find() && m.groupCount() > 0 && StringUtils.isNotEmpty(m.group(1))) { + return m.group(1); + } else { + return null; + } + } + boolean replaceIfPossible(DocumentContext obj, Artifact artifact) { if (artifact == null || StringUtils.isEmpty(artifact.getType())) { throw new IllegalArgumentException("Artifact and artifact type must be set."); diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacerFactory.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacerFactory.java new file mode 100644 index 00000000000..61492f20841 --- /dev/null +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactReplacerFactory.java @@ -0,0 +1,49 @@ +/* + * Copyright 2018 Google, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact; + +import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactReplacer.Replacer; + +import java.util.regex.Pattern; + +public class ArtifactReplacerFactory { + public static Replacer dockerImageReplacer() { + return Replacer.builder() + .replacePath("$.spec.template.spec.containers.[?( @.image == \"{%name%}\" )].image") + .findPath("$.spec.template.spec.containers.*.image") + .namePattern(Pattern.compile("([0-9A-Za-z./]+).*")) + .type(ArtifactTypes.DOCKER_IMAGE) + .build(); + } + + public static Replacer configMapVolumeReplacer() { + return Replacer.builder() + .replacePath("$.spec.template.spec.volumes.[?( @.configMap.name == \"{%name%}\" )].configMap.name") + .findPath("$.spec.template.spec.volumes.*.configMap.name") + .type(ArtifactTypes.KUBERNETES_CONFIG_MAP) + .build(); + } + + public static Replacer secretVolumeReplacer() { + return Replacer.builder() + .replacePath("$.spec.template.spec.volumes.[?( @.secret.name == \"{%name%}\" )].secret.name") + .findPath("$.spec.template.spec.volumes.*.secret.name") + .type(ArtifactTypes.KUBERNETES_SECRET) + .build(); + } +} diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactTypes.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactTypes.java index 3efbd1d1fc3..ac5855e5e4e 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactTypes.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/artifact/ArtifactTypes.java @@ -18,13 +18,14 @@ package com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact; // TODO(lwander): move to clouddriver-artifacts when ready -public enum ArtifactTypes { - DOCKER_IMAGE("docker/image"), - KUBERNETES_CONFIG_MAP("kubernetes/configMap"); +public class ArtifactTypes { + public static final ArtifactTypes DOCKER_IMAGE = new ArtifactTypes("docker/image"); + public static final ArtifactTypes KUBERNETES_CONFIG_MAP = new ArtifactTypes("kubernetes/configMap"); + public static final ArtifactTypes KUBERNETES_SECRET = new ArtifactTypes("kubernetes/secret"); final private String id; - ArtifactTypes(String id) { + public ArtifactTypes(String id) { this.id = id; } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConverter.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConverter.java index e167b422a0c..23482bb5d27 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConverter.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConverter.java @@ -21,6 +21,7 @@ import com.google.common.collect.ImmutableMap; import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.cats.cache.DefaultCacheData; +import com.netflix.spinnaker.clouddriver.kubernetes.KubernetesCloudProvider; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesCachingProperties; @@ -29,8 +30,11 @@ import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestAnnotater; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestMetadata; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestSpinnakerRelationships; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.names.KubernetesManifestNamer; +import com.netflix.spinnaker.clouddriver.names.NamerRegistry; import com.netflix.spinnaker.kork.artifacts.model.Artifact; import com.netflix.spinnaker.moniker.Moniker; +import com.netflix.spinnaker.moniker.Namer; import io.kubernetes.client.JSON; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.StringUtils; @@ -90,7 +94,7 @@ public static CacheData convertAsArtifact(String account, KubernetesManifest man Map attributes = new ImmutableMap.Builder() .put("artifact", artifact) - .put("creationTimestamp", manifest.getCreationTimestamp()) + .put("creationTimestamp", Optional.ofNullable(manifest.getCreationTimestamp()).orElse("")) .build(); Map> cacheRelationships = new HashMap<>(); @@ -140,7 +144,10 @@ public static CacheData mergeCacheData(CacheData current, CacheData added) { return new DefaultCacheData(id, ttl, attributes, relationships); } - public static CacheData convertAsResource(String account, KubernetesManifest manifest, List resourceRelationships) { + public static CacheData convertAsResource(String account, + KubernetesManifest manifest, + List resourceRelationships, + boolean hasClusterRelationship) { KubernetesCachingProperties cachingProperties = KubernetesManifestAnnotater.getCachingProperties(manifest); if (cachingProperties.isIgnore()) { return null; @@ -152,7 +159,13 @@ public static CacheData convertAsResource(String account, KubernetesManifest man KubernetesApiVersion apiVersion = manifest.getApiVersion(); String name = manifest.getName(); String namespace = manifest.getNamespace(); - Moniker moniker = KubernetesManifestAnnotater.getMoniker(manifest); + Namer namer = account == null + ? new KubernetesManifestNamer() + : NamerRegistry.lookup() + .withProvider(KubernetesCloudProvider.getID()) + .withAccount(account) + .withResource(KubernetesManifest.class); + Moniker moniker = namer.deriveMoniker(manifest); Map attributes = new ImmutableMap.Builder() .put("kind", kind) @@ -178,7 +191,7 @@ public static CacheData convertAsResource(String account, KubernetesManifest man if (StringUtils.isEmpty(application)) { log.debug("Encountered not-spinnaker-owned resource " + namespace + ":" + manifest.getFullResourceName()); } else { - cacheRelationships.putAll(annotatedRelationships(account, namespace, metadata)); + cacheRelationships.putAll(annotatedRelationships(account, metadata, hasClusterRelationship)); } // TODO(lwander) avoid overwriting keys here @@ -211,33 +224,22 @@ public static T getResource(KubernetesManifest manifest, Class clazz) { return json.deserialize(json.serialize(manifest), clazz); } - static Map> annotatedRelationships(String account, String namespace, KubernetesManifestMetadata metadata) { - KubernetesManifestSpinnakerRelationships relationships = metadata.getRelationships(); + static Map> annotatedRelationships(String account, + KubernetesManifestMetadata metadata, + boolean hasClusterRelationship) { Moniker moniker = metadata.getMoniker(); + String application = moniker.getApp(); Artifact artifact = metadata.getArtifact(); Map> cacheRelationships = new HashMap<>(); - String application = moniker.getApp(); cacheRelationships.put(ARTIFACT.toString(), Collections.singletonList(Keys.artifact(artifact.getType(), artifact.getName(), artifact.getLocation(), artifact.getVersion()))); cacheRelationships.put(APPLICATIONS.toString(), Collections.singletonList(Keys.application(application))); String cluster = moniker.getCluster(); - if (!StringUtils.isEmpty(cluster)) { + if (StringUtils.isNotEmpty(cluster) && hasClusterRelationship) { cacheRelationships.put(CLUSTERS.toString(), Collections.singletonList(Keys.cluster(account, application, cluster))); } - if (relationships.getLoadBalancers() != null) { - for (String loadBalancer : relationships.getLoadBalancers()) { - addSingleRelationship(cacheRelationships, account, namespace, loadBalancer); - } - } - - if (relationships.getSecurityGroups() != null) { - for (String securityGroup : relationships.getSecurityGroups()) { - addSingleRelationship(cacheRelationships, account, namespace, securityGroup); - } - } - return cacheRelationships; } @@ -355,7 +357,7 @@ static void logMalformedManifest(Supplier contextMessage, KubernetesMani log.warn("{}: manifest name may not be null, {}", contextMessage.get(), manifest); } - if (StringUtils.isEmpty(manifest.getNamespace())) { + if (StringUtils.isEmpty(manifest.getNamespace()) && manifest.getKind() != KubernetesKind.NAMESPACE) { log.warn("{}: manifest namespace may not be null, {}", contextMessage.get(), manifest); } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDaemonSetCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDaemonSetCachingAgent.java index 6e360e4ddc3..f70c48acc4f 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDaemonSetCachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDaemonSetCachingAgent.java @@ -48,13 +48,18 @@ public class KubernetesDaemonSetCachingAgent extends KubernetesV2OnDemandCaching @Getter final private Collection providedDataTypes = Collections.unmodifiableSet( new HashSet<>(Arrays.asList( - INFORMATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), + AUTHORITATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), INFORMATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), INFORMATIVE.forType(KubernetesKind.DEPLOYMENT.toString()), AUTHORITATIVE.forType(KubernetesKind.DAEMON_SET.toString()) )) ); + @Override + protected boolean hasClusterRelationship() { + return true; + } + @Override protected KubernetesKind primaryKind() { return KubernetesKind.DAEMON_SET; diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDeploymentCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDeploymentCachingAgent.java index 6cc3c1778f5..bed7400e4fe 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDeploymentCachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesDeploymentCachingAgent.java @@ -48,12 +48,17 @@ public class KubernetesDeploymentCachingAgent extends KubernetesV2OnDemandCachin @Getter final private Collection providedDataTypes = Collections.unmodifiableSet( new HashSet<>(Arrays.asList( - INFORMATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), + AUTHORITATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), INFORMATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), AUTHORITATIVE.forType(KubernetesKind.DEPLOYMENT.toString()) )) ); + @Override + protected boolean hasClusterRelationship() { + return true; + } + @Override protected KubernetesKind primaryKind() { return KubernetesKind.DEPLOYMENT; diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesHorizontalPodAutoscalerCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesHorizontalPodAutoscalerCachingAgent.java new file mode 100644 index 00000000000..d7eac8b6423 --- /dev/null +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesHorizontalPodAutoscalerCachingAgent.java @@ -0,0 +1,59 @@ +/* + * Copyright 2018 Google, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.netflix.spectator.api.Registry; +import com.netflix.spinnaker.cats.agent.AgentDataType; +import com.netflix.spinnaker.clouddriver.kubernetes.security.KubernetesNamedAccountCredentials; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesKind; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials; +import lombok.Getter; + +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashSet; + +import static com.netflix.spinnaker.cats.agent.AgentDataType.Authority.AUTHORITATIVE; +import static com.netflix.spinnaker.cats.agent.AgentDataType.Authority.INFORMATIVE; + +public class KubernetesHorizontalPodAutoscalerCachingAgent extends KubernetesV2CachingAgent { + KubernetesHorizontalPodAutoscalerCachingAgent(KubernetesNamedAccountCredentials namedAccountCredentials, + ObjectMapper objectMapper, + Registry registry, + int agentIndex, + int agentCount) { + super(namedAccountCredentials, objectMapper, registry, agentIndex, agentCount); + } + + @Override + protected KubernetesKind primaryKind() { + return KubernetesKind.HORIZONTAL_POD_AUTOSCALER; + } + + @Getter + final private Collection providedDataTypes = Collections.unmodifiableSet( + new HashSet<>(Arrays.asList( + INFORMATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), + INFORMATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), + AUTHORITATIVE.forType(KubernetesKind.HORIZONTAL_POD_AUTOSCALER.toString()) + )) + ); +} diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesPodCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesPodCachingAgent.java index 6be6bf1b871..7aa6d3ab950 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesPodCachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesPodCachingAgent.java @@ -54,10 +54,15 @@ protected KubernetesKind primaryKind() { final private Collection providedDataTypes = Collections.unmodifiableSet( new HashSet<>(Arrays.asList( INFORMATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), - INFORMATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), + AUTHORITATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), INFORMATIVE.forType(KubernetesKind.DEPLOYMENT.toString()), INFORMATIVE.forType(KubernetesKind.REPLICA_SET.toString()), AUTHORITATIVE.forType(KubernetesKind.POD.toString()) )) ); + + @Override + protected boolean hasClusterRelationship() { + return true; + } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesReplicaSetCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesReplicaSetCachingAgent.java index 5145ceb52a8..ca0e57a0c22 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesReplicaSetCachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesReplicaSetCachingAgent.java @@ -48,7 +48,7 @@ public class KubernetesReplicaSetCachingAgent extends KubernetesV2OnDemandCachin @Getter final private Collection providedDataTypes = Collections.unmodifiableSet( new HashSet<>(Arrays.asList( - INFORMATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), + AUTHORITATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), INFORMATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), INFORMATIVE.forType(KubernetesKind.DEPLOYMENT.toString()), AUTHORITATIVE.forType(KubernetesKind.REPLICA_SET.toString()) @@ -59,4 +59,9 @@ public class KubernetesReplicaSetCachingAgent extends KubernetesV2OnDemandCachin protected KubernetesKind primaryKind() { return KubernetesKind.REPLICA_SET; } + + @Override + protected boolean hasClusterRelationship() { + return true; + } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesStatefulSetCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesStatefulSetCachingAgent.java index 7cc0e96426d..c21706d59c0 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesStatefulSetCachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesStatefulSetCachingAgent.java @@ -55,7 +55,7 @@ public class KubernetesStatefulSetCachingAgent extends KubernetesV2OnDemandCachi @Getter final private Collection providedDataTypes = Collections.unmodifiableSet( new HashSet<>(Arrays.asList( - INFORMATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), + AUTHORITATIVE.forType(Keys.LogicalKind.APPLICATIONS.toString()), INFORMATIVE.forType(Keys.LogicalKind.CLUSTERS.toString()), INFORMATIVE.forType(KubernetesKind.SERVICE.toString()), AUTHORITATIVE.forType(KubernetesKind.STATEFUL_SET.toString()) @@ -88,4 +88,9 @@ protected Map> loadSecondaryResourc return result; } + + @Override + protected boolean hasClusterRelationship() { + return true; + } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2CachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2CachingAgent.java index e2ccc0c8d07..81d0454c502 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2CachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2CachingAgent.java @@ -58,6 +58,11 @@ protected KubernetesV2CachingAgent(KubernetesNamedAccountCredentials loadPrimaryResourceList() { return namespaces.stream() .map(n -> credentials.list(primaryKind(), n)) @@ -73,7 +78,13 @@ protected KubernetesManifest loadPrimaryResource(String namespace, String name) public CacheResult loadData(ProviderCache providerCache) { log.info(getAgentType() + " is starting"); reloadNamespaces(); - return buildCacheResult(loadPrimaryResourceList()); + + try { + return buildCacheResult(loadPrimaryResourceList()); + } catch (KubectlJobExecutor.NoResourceTypeException e) { + log.warn(getAgentType() + ": resource for this caching agent is not supported for this cluster"); + return new DefaultCacheResult(new HashMap<>()); + } } protected CacheResult buildCacheResult(KubernetesManifest resource) { @@ -84,7 +95,7 @@ protected CacheResult buildCacheResult(List resources) { Map> relationships = loadSecondaryResourceRelationships(resources); List resourceData = resources.stream() - .map(rs -> KubernetesCacheDataConverter.convertAsResource(accountName, rs, relationships.get(rs))) + .map(rs -> KubernetesCacheDataConverter.convertAsResource(accountName, rs, relationships.get(rs), hasClusterRelationship())) .filter(Objects::nonNull) .collect(Collectors.toList()); diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2OnDemandCachingAgent.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2OnDemandCachingAgent.java index 1934c42cb4e..914f6463213 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2OnDemandCachingAgent.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesV2OnDemandCachingAgent.java @@ -34,6 +34,7 @@ import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesKind; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifest; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor; import com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials; import com.netflix.spinnaker.clouddriver.names.NamerRegistry; import com.netflix.spinnaker.moniker.Namer; @@ -90,7 +91,13 @@ public CacheResult loadData(ProviderCache providerCache) { reloadNamespaces(); Long start = System.currentTimeMillis(); - List primaryResource = loadPrimaryResourceList(); + List primaryResource; + try { + primaryResource = loadPrimaryResourceList(); + } catch (KubectlJobExecutor.NoResourceTypeException e) { + log.warn(getAgentType() + ": resource for this caching agent is not supported for this cluster"); + return new DefaultCacheResult(new HashMap<>()); + } List primaryKeys = primaryResource.stream() .map(rs -> objectMapper.convertValue(rs, KubernetesManifest.class)) diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/KubernetesV2ServerGroup.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/KubernetesV2ServerGroup.java index e9d638cbd0d..2dc1b856e47 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/KubernetesV2ServerGroup.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/KubernetesV2ServerGroup.java @@ -21,14 +21,12 @@ import com.netflix.spinnaker.cats.cache.CacheData; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesCacheDataConverter; -import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesKind; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifest; import com.netflix.spinnaker.clouddriver.model.HealthState; import com.netflix.spinnaker.clouddriver.model.Instance; import com.netflix.spinnaker.clouddriver.model.LoadBalancerServerGroup; import com.netflix.spinnaker.clouddriver.model.ServerGroup; import com.netflix.spinnaker.clouddriver.model.ServerGroupSummary; -import io.kubernetes.client.models.V1beta1ReplicaSet; import lombok.Data; import lombok.EqualsAndHashCode; import lombok.extern.slf4j.Slf4j; @@ -82,9 +80,19 @@ private KubernetesV2ServerGroup(KubernetesManifest manifest, String key, List(instances); this.loadBalancers = loadBalancers; - V1beta1ReplicaSet replicaSet = KubernetesCacheDataConverter.getResource(manifest, V1beta1ReplicaSet.class); + Object odesired = ((Map) manifest + .getOrDefault("spec", new HashMap())) + .getOrDefault("replicas", 0); + Integer desired = 0; + + if (odesired instanceof Number) { + desired = ((Number) odesired).intValue(); + } else { + log.warn("Unable to cast replica count from unexpected type: {}", odesired.getClass()); + } + this.capacity = Capacity.builder() - .desired(replicaSet.getSpec().getReplicas()) + .desired(desired) .build(); } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/ManifestBasedModel.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/ManifestBasedModel.java index cd410ba4111..1148a17b2f7 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/ManifestBasedModel.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/model/ManifestBasedModel.java @@ -19,6 +19,7 @@ import com.netflix.spinnaker.clouddriver.kubernetes.KubernetesCloudProvider; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesKind; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifest; import com.netflix.spinnaker.clouddriver.names.NamerRegistry; import com.netflix.spinnaker.moniker.Moniker; @@ -88,6 +89,10 @@ public Long getCreatedTime() { return null; } + public KubernetesKind getKind() { + return getManifest().getKind(); + } + abstract protected KubernetesManifest getManifest(); abstract protected Keys.InfrastructureCacheKey getKey(); } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ApplicationProvider.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ApplicationProvider.java index 3fdc81678e4..cd2247adce4 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ApplicationProvider.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ApplicationProvider.java @@ -92,6 +92,10 @@ public Application getApplication(String name) { .map(k -> (ClusterCacheKey) k) .collect(Collectors.toList()); + if (keys.isEmpty()) { + return null; + } + return KubernetesV2Application.builder() .name(name) .clusterNames(groupClustersByAccount(keys)) diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ManifestProvider.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ManifestProvider.java index 7df1c0ac1c2..ab87967476f 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ManifestProvider.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/view/provider/KubernetesV2ManifestProvider.java @@ -67,7 +67,7 @@ public KubernetesV2Manifest getManifest(String account, String location, String } CacheData data = dataOptional.get(); - KubernetesHandler deployer = registry.get(kind).getHandler(); + KubernetesHandler handler = registry.get(kind).getHandler(); KubernetesManifest manifest = KubernetesCacheDataConverter.getManifest(data); Moniker moniker = KubernetesCacheDataConverter.getMoniker(data); @@ -77,8 +77,8 @@ public KubernetesV2Manifest getManifest(String account, String location, String .location(location) .manifest(manifest) .moniker(moniker) - .status(deployer.status(manifest)) - .artifacts(deployer.listArtifacts(manifest)) + .status(handler.status(manifest)) + .artifacts(handler.listArtifacts(manifest)) .build(); } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesApiVersion.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesApiVersion.java index aa37c6898ab..ea6a590574a 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesApiVersion.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesApiVersion.java @@ -22,7 +22,9 @@ import lombok.EqualsAndHashCode; import java.util.ArrayList; +import java.util.Collections; import java.util.List; +import java.util.Optional; @EqualsAndHashCode public class KubernetesApiVersion { @@ -38,7 +40,7 @@ public class KubernetesApiVersion { protected KubernetesApiVersion(String name) { if (values == null) { - values = new ArrayList<>(); + values = Collections.synchronizedList(new ArrayList<>()); } this.name = name; @@ -53,9 +55,11 @@ public String toString() { @JsonCreator public static KubernetesApiVersion fromString(String name) { - return values.stream() + Optional versionOptional = values.stream() .filter(v -> v.name.equalsIgnoreCase(name)) - .findAny() - .orElseThrow(() -> new IllegalArgumentException("API version " + name + " is not yet supported.")); + .findAny(); + + // separate from the above chain to avoid concurrent modification of the values list + return versionOptional.orElseGet(() -> new KubernetesApiVersion(name)); } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesKind.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesKind.java index b8e779c354f..26bfd3403a8 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesKind.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesKind.java @@ -21,13 +21,16 @@ import com.fasterxml.jackson.annotation.JsonValue; import java.util.ArrayList; +import java.util.Collections; import java.util.List; +import java.util.Optional; public class KubernetesKind { public static KubernetesKind CONFIG_MAP = new KubernetesKind("configMap", "cm"); public static KubernetesKind CONTROLLER_REVISION = new KubernetesKind("controllerRevision"); public static KubernetesKind DAEMON_SET = new KubernetesKind("daemonSet", "ds"); public static KubernetesKind DEPLOYMENT = new KubernetesKind("deployment", "deploy"); + public static KubernetesKind HORIZONTAL_POD_AUTOSCALER = new KubernetesKind("horizontalpodautoscaler", "hpa"); public static KubernetesKind INGRESS = new KubernetesKind("ingress", "ing"); public static KubernetesKind POD = new KubernetesKind("pod", "po"); public static KubernetesKind REPLICA_SET = new KubernetesKind("replicaSet", "rs"); @@ -44,7 +47,7 @@ public class KubernetesKind { protected KubernetesKind(String name, String alias) { if (values == null) { - values = new ArrayList<>(); + values = Collections.synchronizedList(new ArrayList<>()); } this.name = name; @@ -64,9 +67,11 @@ public String toString() { @JsonCreator public static KubernetesKind fromString(String name) { - return values.stream() + Optional kindOptional = values.stream() .filter(v -> v.name.equalsIgnoreCase(name) || (v.alias != null && v.alias.equalsIgnoreCase(name))) - .findAny() - .orElseThrow(() -> new IllegalArgumentException("Kubernetes kind '" + name + "' is not supported.")); + .findAny(); + + // separate from the above chain to avoid concurrent modification of the values list + return kindOptional.orElseGet(() -> new KubernetesKind(name)); } } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifest.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifest.java index 045ca028109..1456220bf53 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifest.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifest.java @@ -96,7 +96,9 @@ public void setNamespace(String namespace) { @JsonIgnore public String getCreationTimestamp() { - return getMetadata().get("creationTimestamp").toString(); + return getMetadata().containsKey("creationTimestamp") + ? getMetadata().get("creationTimestamp").toString() + : null; } @JsonIgnore diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifestAnnotater.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifestAnnotater.java index 1d01647ee0c..b1e9282e60d 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifestAnnotater.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesManifestAnnotater.java @@ -81,8 +81,9 @@ private static T getAnnotation(Map annotations, String key, try { return objectMapper.readValue(value, typeReference); - } catch (IOException e) { - throw new IllegalArgumentException("Illegally annotated resource for '" + key + "': " + e); + } catch (Exception e) { + log.warn("Illegally annotated resource for '" + key + "': " + e); + return null; } } @@ -174,9 +175,9 @@ public static Moniker getMoniker(KubernetesManifest manifest) { return Moniker.builder() .cluster(getAnnotation(annotations, CLUSTER, new TypeReference() {}, parsed.getCluster())) .app(getAnnotation(annotations, APPLICATION, new TypeReference() {}, parsed.getApp())) - .stack(getAnnotation(annotations, STACK, new TypeReference() {}, parsed.getStack())) - .detail(getAnnotation(annotations, DETAIL, new TypeReference() {}, parsed.getDetail())) - .sequence(getAnnotation(annotations, DEPLOYMENT_REVISION, new TypeReference() {}, parsed.getSequence())) + .stack(getAnnotation(annotations, STACK, new TypeReference() {}, null)) + .detail(getAnnotation(annotations, DETAIL, new TypeReference() {}, null)) + .sequence(getAnnotation(annotations, DEPLOYMENT_REVISION, new TypeReference() {}, null)) .build(); } diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/names/KubernetesManifestNamer.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/names/KubernetesManifestNamer.java index a16763c6d05..6c795ecbc2d 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/names/KubernetesManifestNamer.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/names/KubernetesManifestNamer.java @@ -19,10 +19,17 @@ import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifest; import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestAnnotater; +import com.netflix.spinnaker.clouddriver.names.NamingStrategy; import com.netflix.spinnaker.moniker.Moniker; -import com.netflix.spinnaker.moniker.Namer; +import org.springframework.stereotype.Component; + +@Component +public class KubernetesManifestNamer implements NamingStrategy { + @Override + public String getName() { + return "kubernetesAnnotations"; + } -public class KubernetesManifestNamer implements Namer { @Override public void applyMoniker(KubernetesManifest obj, Moniker moniker) { KubernetesManifestAnnotater.annotateManifest(obj, moniker); diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesDeploymentHandler.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesDeploymentHandler.java index 1d7e5a380bd..5c7c1399dcc 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesDeploymentHandler.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesDeploymentHandler.java @@ -22,6 +22,7 @@ import static com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion.EXTENSIONS_V1BETA1; import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactReplacer; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactReplacerFactory; import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactTypes; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesCacheDataConverter; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesDeploymentCachingAgent; @@ -45,21 +46,9 @@ public class KubernetesDeploymentHandler extends KubernetesHandler implements CanUndoRollout { public KubernetesDeploymentHandler() { - registerReplacer( - ArtifactReplacer.Replacer.builder() - .replacePath("$.spec.template.spec.containers.[?( @.image == \"{%name%}\" )].image") - .findPath("$.spec.template.spec.containers.*.image") - .type(ArtifactTypes.DOCKER_IMAGE) - .build() - ); - - registerReplacer( - ArtifactReplacer.Replacer.builder() - .replacePath("$.spec.template.spec.volumes.[?( @.configMap.name == \"{%name%}\" )].configMap.name") - .findPath("$.spec.template.spec.volumes.*.configMap.name") - .type(ArtifactTypes.KUBERNETES_CONFIG_MAP) - .build() - ); + registerReplacer(ArtifactReplacerFactory.dockerImageReplacer()); + registerReplacer(ArtifactReplacerFactory.configMapVolumeReplacer()); + registerReplacer(ArtifactReplacerFactory.secretVolumeReplacer()); } @Override diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesHorizontalPodAutoscalerHandler.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesHorizontalPodAutoscalerHandler.java new file mode 100644 index 00000000000..6f39d46ce89 --- /dev/null +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesHorizontalPodAutoscalerHandler.java @@ -0,0 +1,54 @@ +/* + * Copyright 2018 Google, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package com.netflix.spinnaker.clouddriver.kubernetes.v2.op.deployer; + +import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesHorizontalPodAutoscalerCachingAgent; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesV2CachingAgent; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.KubernetesSpinnakerKindMap.SpinnakerKind; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesKind; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifest; +import com.netflix.spinnaker.clouddriver.model.Manifest.Status; +import org.springframework.stereotype.Component; + +@Component +public class KubernetesHorizontalPodAutoscalerHandler extends KubernetesHandler implements CanDelete { + @Override + public KubernetesKind kind() { + return KubernetesKind.HORIZONTAL_POD_AUTOSCALER; + } + + @Override + public boolean versioned() { + return true; + } + + @Override + public SpinnakerKind spinnakerKind() { + return SpinnakerKind.UNCLASSIFIED; + } + + @Override + public Status status(KubernetesManifest manifest) { + return new Status(); + } + + @Override + public Class cachingAgentClass() { + return KubernetesHorizontalPodAutoscalerCachingAgent.class; + } +} diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesReplicaSetHandler.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesReplicaSetHandler.java index dd0cabe9240..add12c665bd 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesReplicaSetHandler.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/deployer/KubernetesReplicaSetHandler.java @@ -17,11 +17,7 @@ package com.netflix.spinnaker.clouddriver.kubernetes.v2.op.deployer; -import static com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion.APPS_V1BETA2; -import static com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion.EXTENSIONS_V1BETA1; - -import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactReplacer.Replacer; -import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactTypes; +import com.netflix.spinnaker.clouddriver.kubernetes.v2.artifact.ArtifactReplacerFactory; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesCacheDataConverter; import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent.KubernetesReplicaSetCachingAgent; @@ -38,6 +34,9 @@ import java.util.Map; +import static com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion.APPS_V1BETA2; +import static com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion.EXTENSIONS_V1BETA1; + @Component public class KubernetesReplicaSetHandler extends KubernetesHandler implements CanResize, @@ -45,13 +44,9 @@ public class KubernetesReplicaSetHandler extends KubernetesHandler implements CanScale { public KubernetesReplicaSetHandler() { - registerReplacer( - Replacer.builder() - .replacePath("$.spec.template.spec.containers.[?( @.image == \"{%name%}\" )].image") - .findPath("$.spec.template.spec.containers.*.image") - .type(ArtifactTypes.DOCKER_IMAGE) - .build() - ); + registerReplacer(ArtifactReplacerFactory.dockerImageReplacer()); + registerReplacer(ArtifactReplacerFactory.configMapVolumeReplacer()); + registerReplacer(ArtifactReplacerFactory.secretVolumeReplacer()); } @Override diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/job/KubectlJobExecutor.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/job/KubectlJobExecutor.java index 16195b7551a..b7fb2613f32 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/job/KubectlJobExecutor.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/job/KubectlJobExecutor.java @@ -61,6 +61,8 @@ public class KubectlJobExecutor { @Value("${kubernetes.oAuth.executable:oauth2l}") String oAuthExecutable; + private final static String NO_RESOURCE_TYPE_ERROR = "doesn't have a resource type"; + private final JobExecutor jobExecutor; private final Gson gson = new Gson(); @@ -313,6 +315,8 @@ public KubernetesManifest get(KubernetesV2Credentials credentials, KubernetesKin if (status.getResult() != JobStatus.Result.SUCCESS) { if (status.getStdErr().contains("(NotFound)")) { return null; + } else if (status.getStdErr().contains(NO_RESOURCE_TYPE_ERROR)) { + throw new NoResourceTypeException(status.getStdErr()); } throw new KubectlException("Failed to read " + kind + " from " + namespace + ": " + status.getStdErr()); @@ -333,7 +337,11 @@ public List list(KubernetesV2Credentials credentials, Kubern JobStatus status = backoffWait(jobId, credentials.isDebug()); if (status.getResult() != JobStatus.Result.SUCCESS) { - throw new KubectlException("Failed to read " + kind + " from " + namespace + ": " + status.getStdErr()); + if (status.getStdErr().contains(NO_RESOURCE_TYPE_ERROR)) { + throw new NoResourceTypeException(status.getStdErr()); + } else { + throw new KubectlException("Failed to read " + kind + " from " + namespace + ": " + status.getStdErr()); + } } if (status.getStdErr().contains("No resources found")) { @@ -416,7 +424,11 @@ private void logDebugMessages(String jobId, JobStatus jobStatus) { private List kubectlAuthPrefix(KubernetesV2Credentials credentials) { List command = new ArrayList<>(); - command.add(executable); + if (StringUtils.isNotEmpty(credentials.getKubectlExecutable())) { + command.add(credentials.getKubectlExecutable()); + } else { + command.add(executable); + } if (credentials.isDebug()) { command.add("-v"); @@ -498,6 +510,12 @@ private String getOAuthToken(KubernetesV2Credentials credentials) { return status.getStdOut(); } + public static class NoResourceTypeException extends RuntimeException { + public NoResourceTypeException(String message) { + super(message); + } + } + public static class KubectlException extends RuntimeException { public KubectlException(String message) { super(message); diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/manifest/KubernetesDeployManifestOperation.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/manifest/KubernetesDeployManifestOperation.java index a90c3f1bf49..e7279ce5d5c 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/manifest/KubernetesDeployManifestOperation.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/op/manifest/KubernetesDeployManifestOperation.java @@ -98,11 +98,9 @@ public OperationResult operate(List _unused) { Artifact artifact = converter.toArtifact(provider, manifest); Moniker moniker = description.getMoniker(); - KubernetesManifestSpinnakerRelationships relationships = description.getRelationships(); getTask().updateStatus(OP_NAME, "Annotating manifest with artifact, relationships & moniker..."); KubernetesManifestAnnotater.annotateManifest(manifest, artifact); - KubernetesManifestAnnotater.annotateManifest(manifest, relationships); namer.applyMoniker(manifest, moniker); getTask().updateStatus(OP_NAME, "Setting a resource name..."); diff --git a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/security/KubernetesV2Credentials.java b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/security/KubernetesV2Credentials.java index 6c2d5e5fe90..7c41356c498 100644 --- a/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/security/KubernetesV2Credentials.java +++ b/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/security/KubernetesV2Credentials.java @@ -53,6 +53,10 @@ public class KubernetesV2Credentials implements KubernetesCredentials { private final List namespaces; private final List omitNamespaces; + // remove when kubectl is no longer a dependency + @Getter + private final String kubectlExecutable; + // remove when kubectl is no longer a dependency @Getter private final String kubeconfigFile; @@ -95,6 +99,7 @@ public static class Builder { String accountName; String kubeconfigFile; String context; + String kubectlExecutable; String oAuthServiceAccount; List oAuthScopes; String userAgent; @@ -114,6 +119,11 @@ public Builder kubeconfigFile(String kubeconfigFile) { return this; } + public Builder kubectlExecutable(String kubectlExecutable) { + this.kubectlExecutable = kubectlExecutable; + return this; + } + public Builder context(String context) { this.context = context; return this; @@ -185,6 +195,7 @@ public KubernetesV2Credentials build() { omitNamespaces, registry, kubeconfigFile, + kubectlExecutable, context, oAuthServiceAccount, oAuthScopes, @@ -199,6 +210,7 @@ private KubernetesV2Credentials(@NotNull String accountName, @NotNull List omitNamespaces, @NotNull Registry registry, String kubeconfigFile, + String kubectlExecutable, String context, String oAuthServiceAccount, List oAuthScopes, @@ -210,7 +222,7 @@ private KubernetesV2Credentials(@NotNull String accountName, this.omitNamespaces = omitNamespaces; this.jobExecutor = jobExecutor; this.debug = debug; - + this.kubectlExecutable = kubectlExecutable; this.kubeconfigFile = kubeconfigFile; this.context = context; this.oAuthServiceAccount = oAuthServiceAccount; diff --git a/clouddriver-kubernetes/src/test/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConvertSpec.groovy b/clouddriver-kubernetes/src/test/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConvertSpec.groovy index aa2b98db1b8..1131fc1108c 100644 --- a/clouddriver-kubernetes/src/test/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConvertSpec.groovy +++ b/clouddriver-kubernetes/src/test/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/caching/agent/KubernetesCacheDataConvertSpec.groovy @@ -19,6 +19,7 @@ package com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.agent import com.fasterxml.jackson.databind.ObjectMapper import com.netflix.spinnaker.cats.cache.DefaultCacheData +import com.netflix.spinnaker.clouddriver.kubernetes.KubernetesCloudProvider import com.netflix.spinnaker.clouddriver.kubernetes.v2.caching.Keys import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesApiVersion import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesKind @@ -26,6 +27,8 @@ import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.Kube import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestAnnotater import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestMetadata import com.netflix.spinnaker.clouddriver.kubernetes.v2.description.manifest.KubernetesManifestSpinnakerRelationships +import com.netflix.spinnaker.clouddriver.kubernetes.v2.names.KubernetesManifestNamer +import com.netflix.spinnaker.clouddriver.names.NamerRegistry import com.netflix.spinnaker.kork.artifacts.model.Artifact import com.netflix.spinnaker.moniker.Moniker import org.apache.commons.lang3.tuple.Pair @@ -58,11 +61,18 @@ metadata: .cluster(cluster) .build() + if (account != null) { + NamerRegistry.lookup() + .withProvider(KubernetesCloudProvider.ID) + .withAccount(account) + .setNamer(KubernetesManifest, new KubernetesManifestNamer()) + } + def manifest = stringToManifest(rawManifest) KubernetesManifestAnnotater.annotateManifest(manifest, moniker) when: - def cacheData = KubernetesCacheDataConverter.convertAsResource(account, manifest, []) + def cacheData = KubernetesCacheDataConverter.convertAsResource(account, manifest, [], true) then: if (application == null) { @@ -143,47 +153,4 @@ metadata: } != null } } - - @Unroll - def "correctly derive annotated spinnaker relationships"() { - setup: - def spinnakerRelationships = new KubernetesManifestSpinnakerRelationships() - .setLoadBalancers(loadBalancers) - - def moniker = Moniker.builder() - .cluster(cluster) - .app(application) - .build() - - def artifact = new Artifact() - - def metadata = KubernetesManifestMetadata.builder() - .relationships(spinnakerRelationships) - .moniker(moniker) - .artifact(artifact) - .build() - - when: - def relationships = KubernetesCacheDataConverter.annotatedRelationships(ACCOUNT, NAMESPACE, metadata) - def parsedLbs = loadBalancers.collect { lb -> KubernetesManifest.fromFullResourceName(lb) } - - then: - relationships.get(Keys.LogicalKind.CLUSTERS.toString()) == [Keys.cluster(ACCOUNT, application, cluster)] - relationships.get(Keys.LogicalKind.APPLICATIONS.toString()) == [Keys.application(application)] - - def services = filterRelationships(relationships.get(KubernetesKind.SERVICE.toString()), parsedLbs) - def ingresses = filterRelationships(relationships.get(KubernetesKind.INGRESS.toString()), parsedLbs) - - ingresses.size() + services.size() == loadBalancers.size() - - where: - cluster | application | loadBalancers - "a" | "b" | ["service hi"] - "a" | "b" | ["service hi", "service bye"] - "a" | "b" | [] - "a" | "b" | ["service hi", "service bye", "ingress into"] - "a" | "b" | ["ingress into"] - "a" | "b" | ["ingress into", "ingress outof"] - "a" | "b" | ["service hi", "service bye", "ingress into", "ingress outof"] - } } diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeProvider.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeProvider.groovy index 51b1f8e7e3a..57f1763839e 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeProvider.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeProvider.groovy @@ -23,12 +23,20 @@ import org.openstack4j.model.compute.IPProtocol import org.openstack4j.model.compute.RebootType import org.openstack4j.model.compute.SecGroupExtension import org.openstack4j.model.compute.Server +import org.openstack4j.model.compute.ext.AvailabilityZone /** * Methods for interacting with the current compute api. */ interface OpenstackComputeProvider { + /** + * Requests a list of the availability zones in a given region. + * @param region + * @return + */ + List getZones(String region) + /** * Returns a list of instances in a given region. * @param region diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeV2Provider.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeV2Provider.groovy index f80a6b4884e..a18adb6c321 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeV2Provider.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackComputeV2Provider.groovy @@ -27,8 +27,9 @@ import org.openstack4j.model.compute.IPProtocol import org.openstack4j.model.compute.RebootType import org.openstack4j.model.compute.SecGroupExtension import org.openstack4j.model.compute.Server +import org.openstack4j.model.compute.ext.AvailabilityZone -public class OpenstackComputeV2Provider implements OpenstackComputeProvider, OpenstackRequestHandler, OpenstackIdentityAware { +class OpenstackComputeV2Provider implements OpenstackComputeProvider, OpenstackRequestHandler, OpenstackIdentityAware { OpenstackIdentityProvider identityProvider @@ -36,6 +37,13 @@ public class OpenstackComputeV2Provider implements OpenstackComputeProvider, Ope this.identityProvider = identityProvider } + @Override + List getZones(String region) { + handleRequest { + getRegionClient(region).compute().zones().list() + } + } + @Override List getInstances(String region) { handleRequest { diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParameters.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParameters.groovy index 65922dc0363..8160b6499d5 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParameters.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParameters.groovy @@ -50,8 +50,9 @@ class ServerGroupParameters { String sourceUserDataType String sourceUserData Map tags - String resourceFilename String floatingNetworkId + List zones + Map schedulerHints static final ObjectMapper objectMapper = new ObjectMapper() @@ -79,7 +80,8 @@ class ServerGroupParameters { source_user_data : sourceUserData ?: null, tags : objectMapper.writeValueAsString(tags ?: [:]) ?: null, user_data : rawUserData ?: null, - resource_filename : resourceFilename ?: ServerGroupConstants.SUBTEMPLATE_FILE + zones : zones?.join(',') ?: null, + scheduler_hints : objectMapper.writeValueAsString(schedulerHints ?: [:]) ?: null, ] if (floatingNetworkId) { params << [floating_network_id: floatingNetworkId] @@ -116,7 +118,8 @@ class ServerGroupParameters { tags: unescapePythonUnicodeJsonMap(params.get('tags') ?: '{}'), sourceUserDataType: params.get('source_user_data_type'), sourceUserData: params.get('source_user_data'), - resourceFilename: params.get('resource_filename') + zones: unescapePythonUnicodeJsonList(params.get('zones')), + schedulerHints: unescapePythonUnicodeJsonMap(params.get('scheduler_hints') ?: '{}'), ) } diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/StackPoolMemberAware.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/StackPoolMemberAware.groovy index c69daf6f942..d7898cf266e 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/StackPoolMemberAware.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/StackPoolMemberAware.groovy @@ -16,8 +16,6 @@ package com.netflix.spinnaker.clouddriver.openstack.deploy.ops -import com.fasterxml.jackson.databind.ObjectMapper -import com.fasterxml.jackson.dataformat.yaml.YAMLFactory import com.netflix.spinnaker.clouddriver.openstack.deploy.description.servergroup.MemberData import com.netflix.spinnaker.clouddriver.openstack.security.OpenstackCredentials import org.openstack4j.model.network.ext.ListenerV2 @@ -69,8 +67,7 @@ trait StackPoolMemberAware { * @param memberData * @return */ - String buildPoolMemberTemplate(List memberData) { - ObjectMapper mapper = new ObjectMapper(new YAMLFactory()) + Map buildPoolMemberTemplate(List memberData) { Map parameters = [address: [type: "string", description: "Server address for autoscaling group resource"]] Map resources = memberData.collectEntries { [ @@ -90,6 +87,6 @@ trait StackPoolMemberAware { description : "Pool members for autoscaling group resource", parameters : parameters, resources : resources] - mapper.writeValueAsString(memberTemplate) + return memberTemplate } } diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/AbstractStackUpdateOpenstackAtomicOperation.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/AbstractStackUpdateOpenstackAtomicOperation.groovy index 9be9eb4b076..114d2c8927c 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/AbstractStackUpdateOpenstackAtomicOperation.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/AbstractStackUpdateOpenstackAtomicOperation.groovy @@ -96,35 +96,28 @@ abstract class AbstractStackUpdateOpenstackAtomicOperation implements AtomicOper //pre update ops preUpdate(stack) - //we need to store subtemplate in server group output from create, as it is required to do an update and there is no native way of - //obtaining it from a stack - String resourceFileName = stack.parameters?.get(ServerGroupConstants.SUBTEMPLATE_FILENAME) - - if (resourceFileName) { - List> outputs = stack.outputs - String resourceSubtemplate = outputs.find { m -> m.get("output_key") == ServerGroupConstants.SUBTEMPLATE_OUTPUT }.get("output_value") - String memberTemplate = outputs.find { m -> m.get("output_key") == ServerGroupConstants.MEMBERTEMPLATE_OUTPUT }.get("output_value") - task.updateStatus phaseName, "Successfully fetched server group $foundServerGroupName" - - //get the current template from the stack - task.updateStatus phaseName, "Fetching current template for server group $foundServerGroupName" - String template = provider.getHeatTemplate(description.region, stack.name, stack.id) - task.updateStatus phaseName, "Successfully fetched current template for server group $foundServerGroupName" - - Map templateMap = [(resourceFileName): resourceSubtemplate] - if (memberTemplate) { - templateMap << [(ServerGroupConstants.MEMBERTEMPLATE_FILE): memberTemplate] - } - - //update stack - task.updateStatus phaseName, "Updating server group $stack.name" - provider.updateStack(description.region, stack.name, stack.id, template, templateMap, buildServerGroupParameters(stack), stack.tags) - task.updateStatus phaseName, "Successfully updated server group $stack.name" - } else { - task.updateStatus phaseName, "Missing resource filename in parameters list- ${stack.parameters}" - throw new OpenstackOperationException("Missing resource filename in HEAT template") + String resourceFileName = ServerGroupConstants.SUBTEMPLATE_FILE + + List> outputs = stack.outputs + String resourceSubtemplate = outputs.find { m -> m.get("output_key") == ServerGroupConstants.SUBTEMPLATE_OUTPUT }.get("output_value") + String memberTemplate = outputs.find { m -> m.get("output_key") == ServerGroupConstants.MEMBERTEMPLATE_OUTPUT }.get("output_value") + task.updateStatus phaseName, "Successfully fetched server group $foundServerGroupName" + + //get the current template from the stack + task.updateStatus phaseName, "Fetching current template for server group $foundServerGroupName" + String template = provider.getHeatTemplate(description.region, stack.name, stack.id) + task.updateStatus phaseName, "Successfully fetched current template for server group $foundServerGroupName" + + Map templateMap = [(resourceFileName): resourceSubtemplate] + if (memberTemplate) { + templateMap << [(ServerGroupConstants.MEMBERTEMPLATE_FILE): memberTemplate] } + //update stack + task.updateStatus phaseName, "Updating server group $stack.name" + provider.updateStack(description.region, stack.name, stack.id, template, templateMap, buildServerGroupParameters(stack), stack.tags) + task.updateStatus phaseName, "Successfully updated server group $stack.name" + //post update ops postUpdate(stack) diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperation.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperation.groovy index bb231c4bece..a1a22ab94e0 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperation.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperation.groovy @@ -96,7 +96,6 @@ class CloneOpenstackAtomicOperation implements AtomicOperation scaleup = description.serverGroupParameters?.scaleup ?: ancestorParams.scaleup scaledown = description.serverGroupParameters?.scaledown ?: ancestorParams.scaledown tags = description.serverGroupParameters?.tags ?: ancestorParams.tags - resourceFilename = description.serverGroupParameters?.resourceFilename ?: ancestorParams.resourceFilename // Lack of floatingNetworkId means to not set one, so can't pull this value from the ancestorParams floatingNetworkId = description.serverGroupParameters?.floatingNetworkId diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperation.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperation.groovy index 04de68792a4..8de7ece3f31 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperation.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperation.groovy @@ -16,6 +16,8 @@ package com.netflix.spinnaker.clouddriver.openstack.deploy.ops.servergroup +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory import com.netflix.spinnaker.clouddriver.deploy.DeploymentResult import com.netflix.spinnaker.clouddriver.openstack.client.OpenstackClientProvider import com.netflix.spinnaker.clouddriver.openstack.deploy.OpenstackServerGroupNameResolver @@ -56,6 +58,8 @@ class DeployOpenstackAtomicOperation implements TaskStatusAware, AtomicOperation static final Map templateMap = new ConcurrentHashMap<>() + private ObjectMapper objectMapper = new ObjectMapper(new YAMLFactory()) + DeployOpenstackAtomicOperation(DeployOpenstackAtomicOperationDescription description) { this.description = description } @@ -115,12 +119,14 @@ class DeployOpenstackAtomicOperation implements TaskStatusAware, AtomicOperation def stackName = serverGroupNameResolver.resolveNextServerGroupName(description.application, description.stack, description.freeFormDetails, false) task.updateStatus BASE_PHASE, "Heat stack name chosen to be ${stackName}." - Map subtemplates = [:] - String template = getTemplateFile(ServerGroupConstants.TEMPLATE_FILE) - String resourceFilename = ServerGroupConstants.SUBTEMPLATE_FILE + Map templates = [ + main: objectMapper.readValue(getTemplateFile(ServerGroupConstants.TEMPLATE_FILE), Map) + ] if (description.serverGroupParameters.floatingNetworkId) { - template = getTemplateFile(ServerGroupConstants.TEMPLATE_FILE_FLOAT) + templates.main.parameters.floating_network_id = [type: "string", description: "Network used to allocate a floating IP for each server."] + templates.main.resources.servergroup.properties.resource.properties.floating_network_id = [get_param: "floating_network_id"] + } if (description.serverGroupParameters.loadBalancers && !description.serverGroupParameters.loadBalancers.isEmpty()) { //look up all load balancer listeners -> pool ids and internal ports @@ -128,33 +134,38 @@ class DeployOpenstackAtomicOperation implements TaskStatusAware, AtomicOperation List memberDataList = buildMemberData(description.credentials, description.region, description.serverGroupParameters.subnetId, description.serverGroupParameters.loadBalancers, this.&parseListenerKey) task.updateStatus BASE_PHASE, "Finished getting load balancer details for load balancers $description.serverGroupParameters.loadBalancers." + templates[ServerGroupConstants.SUBTEMPLATE_FILE] = objectMapper.readValue(getTemplateFile(ServerGroupConstants.SUBTEMPLATE_FILE), Map) //check for floating ip if (description.serverGroupParameters.floatingNetworkId) { - resourceFilename = ServerGroupConstants.SUBTEMPLATE_FILE_FLOAT + templates[ServerGroupConstants.SUBTEMPLATE_FILE].parameters.floating_network_id = [type: "string", description: "Network used to allocate a floating IP for each server."] + templates[ServerGroupConstants.SUBTEMPLATE_FILE].resources.server_floating_ip = [ + type: "OS::Neutron::FloatingIP", + properties: [ + floating_network_id: [get_param: "floating_network_id"], + port_id: [get_attr: ["server", "addresses", [get_param: "network_id"], 0, "port"]] + ] + ] } task.updateStatus BASE_PHASE, "Loading lbaas subtemplates..." - String subtemplate = getTemplateFile(resourceFilename) - if (subtemplate) { - subtemplates << [(resourceFilename): subtemplate] - if (subtemplate.contains(ServerGroupConstants.MEMBERTEMPLATE_FILE)) { - subtemplates << [(ServerGroupConstants.MEMBERTEMPLATE_FILE): buildPoolMemberTemplate(memberDataList)] - } + if (objectMapper.writeValueAsString(templates[ServerGroupConstants.SUBTEMPLATE_FILE]).contains(ServerGroupConstants.MEMBERTEMPLATE_FILE)) { + templates[ServerGroupConstants.MEMBERTEMPLATE_FILE] = buildPoolMemberTemplate(memberDataList) } task.updateStatus BASE_PHASE, "Finished loading lbaas templates." } else { task.updateStatus BASE_PHASE, "Loading subtemplates..." //check for floating ip + templates[ServerGroupConstants.SUBTEMPLATE_FILE] = objectMapper.readValue(getTemplateFile(ServerGroupConstants.SUBTEMPLATE_SERVER_FILE), Map) if (description.serverGroupParameters.floatingNetworkId) { - resourceFilename = ServerGroupConstants.SUBTEMPLATE_SERVER_FILE_FLOAT - } else { - resourceFilename = ServerGroupConstants.SUBTEMPLATE_SERVER_FILE - } - - String subtemplate = getTemplateFile(resourceFilename) - if (subtemplate) { - subtemplates << [(resourceFilename): subtemplate] + templates[ServerGroupConstants.SUBTEMPLATE_FILE].parameters.floating_network_id = [type: "string", description: "Network used to allocate a floating IP for each server."] + templates[ServerGroupConstants.SUBTEMPLATE_FILE].resources.server_floating_ip = [ + type: "OS::Neutron::FloatingIP", + properties: [ + floating_network_id: [get_param: "floating_network_id"], + port_id: [get_attr: ["server", "addresses", [get_param: "network_id"], 0, "port"]] + ] + ] } task.updateStatus BASE_PHASE, "Finished loading templates." } @@ -166,15 +177,24 @@ class DeployOpenstackAtomicOperation implements TaskStatusAware, AtomicOperation String userData = getUserData(provider, stackName) + if (description.serverGroupParameters.zones) { + task.updateStatus BASE_PHASE, "Creating zone policy for ${description.serverGroupParameters.zones.size()} zones" + addZonePlacementPolicy(description.serverGroupParameters.zones, templates.main, templates[ServerGroupConstants.SUBTEMPLATE_FILE]) + } + task.updateStatus BASE_PHASE, "Creating heat stack $stackName..." ServerGroupParameters params = description.serverGroupParameters.identity { it.networkId = subnet.networkId it.rawUserData = userData it.sourceUserDataType = description.userDataType it.sourceUserData = description.userData - it.resourceFilename = resourceFilename it } + + def template = objectMapper.writeValueAsString(templates.main) + //drop the primary template and convert everything to string + def subtemplates = (Map) templates.findAll { it.key != "main"}.collectEntries {k, v -> [(k): objectMapper.writeValueAsString(v)]} + provider.deploy(description.region, stackName, template, subtemplates, params, description.disableRollback, description.timeoutMins, description.serverGroupParameters.loadBalancers) task.updateStatus BASE_PHASE, "Finished creating heat stack $stackName." @@ -228,4 +248,38 @@ class DeployOpenstackAtomicOperation implements TaskStatusAware, AtomicOperation template ?: "" } } + + private static void addZonePlacementPolicy(List zones, Map mainTemplate, Map resourceTemplate) { + def placementList = zones.collect { zone -> + [ + name: zone, + weight: 100 + ] + } + mainTemplate.resources.zone_policy = [ + type: "OS::Senlin::Policy", + properties: [ + type: "senlin.policy.zone_placement", + version: "1.0", + properties: [ + regions: placementList + ] + ] + ] + mainTemplate.resources.zone_policy_group = [ + type: "OS::Nova::ServerGroup", + properties: [ + policies: [ + [ + get_resource: "zone_policy" + ] + ] + ] + ] + resourceTemplate.resources.server.properties.scheduler_hints = [ + group: [ + get_resource: "zone_policy_group" + ] + ] + } } diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/ServerGroupConstants.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/ServerGroupConstants.groovy index 72a3b4bb09d..45405bb369c 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/ServerGroupConstants.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/ServerGroupConstants.groovy @@ -41,11 +41,11 @@ class ServerGroupConstants { //this is the name of the subtemplate referenced by the template, //and needs to be loaded into memory as a String - final static String SUBTEMPLATE_FILE = "${SUBTEMPLATE_OUTPUT}.yaml" + final static String SUBTEMPLATE_FILE = "${SUBTEMPLATE_OUTPUT}.yaml".toString() //with floating ip for each instance - final static String SUBTEMPLATE_FILE_FLOAT = "${SUBTEMPLATE_OUTPUT_FLOAT}.yaml" + final static String SUBTEMPLATE_FILE_FLOAT = "${SUBTEMPLATE_OUTPUT_FLOAT}.yaml".toString() //this is the name of the member template referenced by the subtemplate, //and is contructed on the fly - final static String MEMBERTEMPLATE_FILE = "${MEMBERTEMPLATE_OUTPUT}.yaml" + final static String MEMBERTEMPLATE_FILE = "${MEMBERTEMPLATE_OUTPUT}.yaml".toString() } diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/provider/agent/OpenstackServerGroupCachingAgent.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/provider/agent/OpenstackServerGroupCachingAgent.groovy index 147a787b460..67326bd2401 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/provider/agent/OpenstackServerGroupCachingAgent.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/provider/agent/OpenstackServerGroupCachingAgent.groovy @@ -94,72 +94,76 @@ class OpenstackServerGroupCachingAgent extends AbstractOpenstackCachingAgent imp Map> instancesByStackId = getInstanceIdsByStack(region, stacks) stacks?.each { Stack stack -> - String serverGroupName = stack.name - Names names = Names.parseName(serverGroupName) - if (!names && !names.app && !names.cluster) { - log.info("Skipping server group ${serverGroupName}") - } else { - String applicationName = names.app - String clusterName = names.cluster - - String serverGroupKey = Keys.getServerGroupKey(serverGroupName, accountName, region) - String clusterKey = Keys.getClusterKey(accountName, applicationName, clusterName) - String appKey = Keys.getApplicationKey(applicationName) - - cacheResultBuilder.namespace(APPLICATIONS.ns).keep(appKey).with { - attributes.name = applicationName - relationships[CLUSTERS.ns].add(clusterKey) - } + try { + String serverGroupName = stack.name + Names names = Names.parseName(serverGroupName) + if (!names && !names.app && !names.cluster) { + log.info("Skipping server group ${serverGroupName}") + } else { + String applicationName = names.app + String clusterName = names.cluster - cacheResultBuilder.namespace(CLUSTERS.ns).keep(clusterKey).with { - attributes.name = clusterName - attributes.accountName = accountName - relationships[APPLICATIONS.ns].add(appKey) - relationships[SERVER_GROUPS.ns].add(serverGroupKey) - } + String serverGroupKey = Keys.getServerGroupKey(serverGroupName, accountName, region) + String clusterKey = Keys.getClusterKey(accountName, applicationName, clusterName) + String appKey = Keys.getApplicationKey(applicationName) + + cacheResultBuilder.namespace(APPLICATIONS.ns).keep(appKey).with { + attributes.name = applicationName + relationships[CLUSTERS.ns].add(clusterKey) + } - Stack detail = clientProvider.getStack(region, stack.name) - Set loadBalancerKeys = [].toSet() - Set statuses = [].toSet() - if (detail && detail.parameters) { - statuses = ServerGroupParameters.fromParamsMap(detail.parameters).loadBalancers?.collect { loadBalancerId -> - LoadBalancerV2Status status = null - try { - status = clientProvider.getLoadBalancerStatusTree(region, loadBalancerId)?.loadBalancerV2Status - if (status) { - String loadBalancerKey = Keys.getLoadBalancerKey(status.name, status.id, accountName, region) - cacheResultBuilder.namespace(LOAD_BALANCERS.ns).keep(loadBalancerKey).with { - relationships[SERVER_GROUPS.ns].add(serverGroupKey) + cacheResultBuilder.namespace(CLUSTERS.ns).keep(clusterKey).with { + attributes.name = clusterName + attributes.accountName = accountName + relationships[APPLICATIONS.ns].add(appKey) + relationships[SERVER_GROUPS.ns].add(serverGroupKey) + } + + Stack detail = clientProvider.getStack(region, stack.name) + Set loadBalancerKeys = [].toSet() + Set statuses = [].toSet() + if (detail && detail.parameters) { + statuses = ServerGroupParameters.fromParamsMap(detail.parameters).loadBalancers?.collect { loadBalancerId -> + LoadBalancerV2Status status = null + try { + status = clientProvider.getLoadBalancerStatusTree(region, loadBalancerId)?.loadBalancerV2Status + if (status) { + String loadBalancerKey = Keys.getLoadBalancerKey(status.name, status.id, accountName, region) + cacheResultBuilder.namespace(LOAD_BALANCERS.ns).keep(loadBalancerKey).with { + relationships[SERVER_GROUPS.ns].add(serverGroupKey) + } + loadBalancerKeys << loadBalancerKey } - loadBalancerKeys << loadBalancerKey + } catch (OpenstackProviderException e) { + //Do nothing ... Load balancer not found. } - } catch (OpenstackProviderException e) { - //Do nothing ... Load balancer not found. - } - status - }?.findAll()?.toSet() - } - - List instanceKeys = [] - instancesByStackId[stack.id]?.each { String id -> - String instanceKey = Keys.getInstanceKey(id, accountName, region) - cacheResultBuilder.namespace(INSTANCES.ns).keep(instanceKey).relationships[SERVER_GROUPS.ns].add(serverGroupKey) - instanceKeys.add(instanceKey) - } + status + }?.findAll()?.toSet() + } - OpenstackServerGroup openstackServerGroup = buildServerGroup(providerCache, detail, statuses, instanceKeys) + List instanceKeys = [] + instancesByStackId[stack.id]?.each { String id -> + String instanceKey = Keys.getInstanceKey(id, accountName, region) + cacheResultBuilder.namespace(INSTANCES.ns).keep(instanceKey).relationships[SERVER_GROUPS.ns].add(serverGroupKey) + instanceKeys.add(instanceKey) + } - if (shouldUseOnDemandData(cacheResultBuilder, serverGroupKey)) { - moveOnDemandDataToNamespace(objectMapper, typeReference, cacheResultBuilder, serverGroupKey) - } else { - cacheResultBuilder.namespace(SERVER_GROUPS.ns).keep(serverGroupKey).with { - attributes = objectMapper.convertValue(openstackServerGroup, ATTRIBUTES) - relationships[APPLICATIONS.ns].add(appKey) - relationships[CLUSTERS.ns].add(clusterKey) - relationships[LOAD_BALANCERS.ns].addAll(loadBalancerKeys) - relationships[INSTANCES.ns].addAll(instanceKeys) + OpenstackServerGroup openstackServerGroup = buildServerGroup(providerCache, detail, statuses, instanceKeys) + + if (shouldUseOnDemandData(cacheResultBuilder, serverGroupKey)) { + moveOnDemandDataToNamespace(objectMapper, typeReference, cacheResultBuilder, serverGroupKey) + } else { + cacheResultBuilder.namespace(SERVER_GROUPS.ns).keep(serverGroupKey).with { + attributes = objectMapper.convertValue(openstackServerGroup, ATTRIBUTES) + relationships[APPLICATIONS.ns].add(appKey) + relationships[CLUSTERS.ns].add(clusterKey) + relationships[LOAD_BALANCERS.ns].addAll(loadBalancerKeys) + relationships[INSTANCES.ns].addAll(instanceKeys) + } } } + } catch (Exception e) { + log.error("Error building cache for stack ${stack}", e) } } @@ -236,14 +240,16 @@ class OpenstackServerGroupCachingAgent extends AbstractOpenstackCachingAgent imp if (appVersionKey) { AppVersion appVersion = AppVersion.parseName(appVersionKey) - result.packageName = appVersion.packageName - result.version = appVersion.version - result.commit = appVersion.commit + if (appVersion) { + result.packageName = appVersion.packageName + result.version = appVersion.version + result.commit = appVersion.commit + } String buildHost = properties.get('build_host') String buildInfoUrl = properties.get('build_info_url') - if (appVersion.buildJobName) { + if (appVersion && appVersion.buildJobName) { Map jenkinsMap = [name: appVersion.buildJobName, number: appVersion.buildNumber] if (buildHost) { jenkinsMap.put('host', buildHost) diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackCredentialsInitializer.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackCredentialsInitializer.groovy index 93e8c624251..dea06f0bc41 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackCredentialsInitializer.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackCredentialsInitializer.groovy @@ -63,21 +63,22 @@ class OpenstackCredentialsInitializer implements CredentialsInitializerSynchroni accountsToAdd.each { OpenstackConfigurationProperties.ManagedAccount managedAccount -> LOG.info("Found openstack managed account $managedAccount") try { - def openstackAccount = new OpenstackNamedAccountCredentials(managedAccount.name, - managedAccount.environment ?: managedAccount.name, - managedAccount.accountType ?: managedAccount.name, - managedAccount.username, - managedAccount.password, - managedAccount.projectName, - managedAccount.domainName, - managedAccount.authUrl, - managedAccount.regions, - managedAccount.insecure, - managedAccount.heatTemplatePath, - managedAccount.lbaas, - managedAccount.consul, - managedAccount.userDataFile - ) + def openstackAccount = new OpenstackNamedAccountCredentials.Builder() + .name(managedAccount.name) + .environment(managedAccount.environment ?: managedAccount.name) + .accountType(managedAccount.accountType ?: managedAccount.name) + .username(managedAccount.username) + .password(managedAccount.password) + .projectName(managedAccount.projectName) + .domainName(managedAccount.domainName) + .authUrl(managedAccount.authUrl) + .regions(managedAccount.regions) + .insecure(managedAccount.insecure) + .heatTemplateLocation(managedAccount.heatTemplatePath) + .lbaasConfig(managedAccount.lbaas) + .consulConfig(managedAccount.consul) + .userDataFile(managedAccount.userDataFile) + .build() LOG.info("Saving openstack account $openstackAccount") accountCredentialsRepository.save(managedAccount.name, openstackAccount) } catch (e) { diff --git a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentials.groovy b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentials.groovy index cfdb1f0f234..5c96793f228 100644 --- a/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentials.groovy +++ b/clouddriver-openstack/src/main/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentials.groovy @@ -21,6 +21,7 @@ import com.netflix.spinnaker.clouddriver.consul.config.ConsulConfig import com.netflix.spinnaker.clouddriver.openstack.config.OpenstackConfigurationProperties.LbaasConfig import com.netflix.spinnaker.clouddriver.security.AccountCredentials import groovy.transform.ToString +import org.openstack4j.model.compute.ext.AvailabilityZone @ToString(includeNames = true, excludes = "password") class OpenstackNamedAccountCredentials implements AccountCredentials { @@ -42,6 +43,8 @@ class OpenstackNamedAccountCredentials implements AccountCredentials> regionToZones + OpenstackNamedAccountCredentials(String accountName, @@ -202,7 +205,7 @@ class OpenstackNamedAccountCredentials implements AccountCredentials + [(region): provider.getZones(region).findAll { zone -> zone.zoneState.available }.collect { zone -> zone.zoneName}] + } + account.regionToZones = regionToZoneMap + return account } } diff --git a/clouddriver-openstack/src/main/resources/servergroup.yaml b/clouddriver-openstack/src/main/resources/servergroup.yaml index 81551050100..5e4b042bc76 100644 --- a/clouddriver-openstack/src/main/resources/servergroup.yaml +++ b/clouddriver-openstack/src/main/resources/servergroup.yaml @@ -23,6 +23,10 @@ parameters: type: comma_delimited_list description: Comma-separated string of load balancers to associate to the stack. This is not used in the stack and is defined for auditing purposes. default: [] + zones: + type: comma_delimited_list + description: Comma-separated string of availability zones + default: [] security_groups: type: comma_delimited_list description: Comma-separated string of security groups to use @@ -81,9 +85,10 @@ parameters: type: string description: Raw base64-encoded string that will execute upon server boot, if cloud-init is installed default: "" - resource_filename: - type: string - description: Member resource file name (i.e. servergroup_resource.yaml) + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} resources: servergroup: type: OS::Heat::AutoScalingGroup @@ -92,7 +97,7 @@ resources: max_size: {get_param: max_size} desired_capacity: {get_param: desired_size} resource: - type: {get_param: resource_filename} + type: servergroup_resource.yaml properties: flavor: {get_param: flavor} image: {get_param: image} @@ -107,6 +112,7 @@ resources: security_groups: {get_param: security_groups} subnet_id: {get_param: subnet_id} user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} web_server_scaleup_policy: type: OS::Heat::ScalingPolicy properties: @@ -153,8 +159,8 @@ outputs: # we need to store subtemplate in servergroup output from create, as it is required to do an update and there is no native way # of obtaining it from a stack servergroup_resource: - description: resource_filename template value - value: {get_file: {get_param: resource_filename} } + description: servergroup_resource.yaml template value + value: {get_file: servergroup_resource.yaml } # we need to store subtemplate in servergroup output from create, as it is required to do an update and there is no native way # of obtaining it from a stack servergroup_resource_member: diff --git a/clouddriver-openstack/src/main/resources/servergroup_resource.yaml b/clouddriver-openstack/src/main/resources/servergroup_resource.yaml index d465413e9a4..cb914ee1996 100644 --- a/clouddriver-openstack/src/main/resources/servergroup_resource.yaml +++ b/clouddriver-openstack/src/main/resources/servergroup_resource.yaml @@ -22,6 +22,10 @@ parameters: user_data: type: string description: String that will execute upon server boot, if cloud-init is installed. + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} resources: server: type: OS::Nova::Server @@ -33,6 +37,7 @@ resources: - subnet: {get_param: subnet_id} security_groups: {get_param: security_groups} user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} user_data_format: RAW member: type: OS::Heat::ResourceGroup diff --git a/clouddriver-openstack/src/main/resources/servergroup_server.yaml b/clouddriver-openstack/src/main/resources/servergroup_server.yaml index 2683620c637..970ca798fb4 100644 --- a/clouddriver-openstack/src/main/resources/servergroup_server.yaml +++ b/clouddriver-openstack/src/main/resources/servergroup_server.yaml @@ -22,6 +22,10 @@ parameters: user_data: type: string description: Raw base64-encoded string that will execute upon server boot, if cloud-init is installed. + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} resources: server: type: OS::Nova::Server @@ -33,6 +37,7 @@ resources: - subnet: {get_param: subnet_id} security_groups: {get_param: security_groups} user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} user_data_format: RAW outputs: diff --git a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackOrchestrationV1ClientProviderSpec.groovy b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackOrchestrationV1ClientProviderSpec.groovy index 34f777f0c67..409f1015ca2 100644 --- a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackOrchestrationV1ClientProviderSpec.groovy +++ b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/client/OpenstackOrchestrationV1ClientProviderSpec.groovy @@ -55,7 +55,6 @@ class OpenstackOrchestrationV1ClientProviderSpec extends OpenstackClientProvider String networkId = '1234' List loadBalancerIds = ['5678'] List securityGroups = ['sg1'] - String resourceFileName = 'servergroup_resource' ServerGroupParameters parameters = new ServerGroupParameters(instanceType: instanceType, image: image, maxSize: maxSize, minSize: minSize, desiredSize: desiredSize, subnetId: subnetId, networkId: networkId, loadBalancers: loadBalancerIds, securityGroups: securityGroups, @@ -66,7 +65,8 @@ class OpenstackOrchestrationV1ClientProviderSpec extends OpenstackClientProvider tags: ['foo': 'bar'], sourceUserDataType: 'Text', sourceUserData: 'echo foobar', - resourceFilename: resourceFileName + zones: ["az1","az2"], + schedulerHints: ["key": "value"] ) Map params = [ flavor : parameters.instanceType, @@ -91,7 +91,8 @@ class OpenstackOrchestrationV1ClientProviderSpec extends OpenstackClientProvider source_user_data : 'echo foobar', tags : '{"foo":"bar"}', user_data : parameters.rawUserData, - resource_filename : resourceFileName + zones : 'az1,az2', + scheduler_hints : '{"key":"value"}' ] List tags = loadBalancerIds.collect { "lb-${it}" } StackCreate stackCreate = Builders.stack().disableRollback(disableRollback).files(subtmpl).name(stackName).parameters(params).template(tmpl).timeoutMins(timeoutMins).tags(tags.join(',')).build() @@ -345,7 +346,7 @@ class OpenstackOrchestrationV1ClientProviderSpec extends OpenstackClientProvider ServerGroupParameters parameters = new ServerGroupParameters(instanceType: instanceType, image: image, maxSize: maxSize, minSize: minSize, desiredSize: desiredSize, networkId: networkId, subnetId: subnetId, loadBalancers: loadBalancerIds, securityGroups: securityGroups, rawUserData: 'echo foobar', tags: ['foo': 'bar'], - sourceUserDataType: 'Text', sourceUserData: 'echo foobar', resourceFilename: resourceFileName) + sourceUserDataType: 'Text', sourceUserData: 'echo foobar', zones: ["az1","az2"], schedulerHints: ["key": "value"]) Map params = [ flavor : parameters.instanceType, image : parameters.image, @@ -369,7 +370,8 @@ class OpenstackOrchestrationV1ClientProviderSpec extends OpenstackClientProvider source_user_data : 'echo foobar', tags : '{"foo":"bar"}', user_data : parameters.rawUserData, - resource_filename : resourceFileName + zones : 'az1,az2', + scheduler_hints : '{"key":"value"}' ] String template = "foo: bar" Map subtmpl = [sub: "foo: bar"] diff --git a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParametersSpec.groovy b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParametersSpec.groovy index 542098b105d..336e1e9fb98 100644 --- a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParametersSpec.groovy +++ b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/description/servergroup/ServerGroupParametersSpec.groovy @@ -81,22 +81,51 @@ class ServerGroupParametersSpec extends Specification { def createServerGroupParams() { ServerGroupParameters.Scaler scaleup = new ServerGroupParameters.Scaler(cooldown: 60, period: 60, adjustment: 1, threshold: 50) ServerGroupParameters.Scaler scaledown = new ServerGroupParameters.Scaler(cooldown: 60, period: 600, adjustment: -1, threshold: 15) - new ServerGroupParameters(instanceType: "m1.medium", image: "image", - maxSize: 5, minSize: 3, desiredSize: 4, - networkId: "net", subnetId: "sub", loadBalancers: ["poop"], - securityGroups: ["sg1"], - autoscalingType: ServerGroupParameters.AutoscalingType.CPU, - scaleup: scaleup, scaledown: scaledown, rawUserData: "echo foobar", tags: ["foo": "bar"], - sourceUserDataType: 'Text', sourceUserData: 'echo foobar', resourceFilename: 'servergroup_resource') + new ServerGroupParameters(instanceType: "m1.medium", + image: "image", + maxSize: 5, minSize: 3, desiredSize: 4, + networkId: "net", + subnetId: "sub", + loadBalancers: ["poop"], + securityGroups: ["sg1"], + autoscalingType: ServerGroupParameters.AutoscalingType.CPU, + scaleup: scaleup, + scaledown: scaledown, + rawUserData: "echo foobar", + tags: ["foo": "bar"], + sourceUserDataType: 'Text', + sourceUserData: 'echo foobar', + zones: ["az1", "az2"], + schedulerHints: ["key": "value"]) } @Ignore def getMap() { - [flavor : 'm1.medium', image: 'image', max_size: 5, min_size: 3, desired_size: 4, - network_id : 'net', subnet_id: 'sub', load_balancers: 'poop', security_groups: 'sg1', autoscaling_type: 'cpu_util', - scaleup_cooldown : 60, scaleup_adjustment: 1, scaleup_period: 60, scaleup_threshold: 50, - scaledown_cooldown : 60, scaledown_adjustment: -1, scaledown_period: 600, scaledown_threshold: 15, - source_user_data_type: 'Text', source_user_data: 'echo foobar', tags: '{"foo":"bar"}', user_data: "echo foobar", resource_filename: 'servergroup_resource'] + [flavor: 'm1.medium', + image: 'image', + max_size: 5, + min_size: 3, + desired_size: 4, + network_id: 'net', + subnet_id: 'sub', + load_balancers: 'poop', + security_groups: 'sg1', + autoscaling_type: 'cpu_util', + scaleup_cooldown: 60, + scaleup_adjustment: 1, + scaleup_period: 60, + scaleup_threshold: 50, + scaledown_cooldown: 60, + scaledown_adjustment: -1, + scaledown_period: 600, + scaledown_threshold: 15, + source_user_data_type: 'Text', + source_user_data: 'echo foobar', + tags: '{"foo":"bar"}', + user_data: "echo foobar", + zones: "az1,az2", + scheduler_hints: '{"key":"value"}' + ] } } diff --git a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperationSpec.groovy b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperationSpec.groovy index 88d6dc7533b..0ae597072fe 100644 --- a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperationSpec.groovy +++ b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/CloneOpenstackAtomicOperationSpec.groovy @@ -87,7 +87,6 @@ class CloneOpenstackAtomicOperationSpec extends Specification { scaleup: scaleup, scaledown: scaledown, tags: ['foo':'bar'], - resourceFilename: 'servergroup_resource', ) new DeployOpenstackAtomicOperationDescription( stack: STACK, @@ -118,7 +117,6 @@ class CloneOpenstackAtomicOperationSpec extends Specification { scaleup: scaleup, scaledown: scaledown, tags: ["foo":"barbar"], - resourceFilename: 'servergroup_resource' ) new DeployOpenstackAtomicOperationDescription( stack: STACK_N, @@ -255,7 +253,6 @@ class CloneOpenstackAtomicOperationSpec extends Specification { scaledown: scaledown, tags: ['foo':'bar'], floatingNetworkId: UUID.toString(), - resourceFilename: 'servergroup_resource', ) def ancestor = new DeployOpenstackAtomicOperationDescription( stack: STACK, diff --git a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperationSpec.groovy b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperationSpec.groovy index d1bfbb63b0d..4b867b5f200 100644 --- a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperationSpec.groovy +++ b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/DeployOpenstackAtomicOperationSpec.groovy @@ -16,6 +16,8 @@ package com.netflix.spinnaker.clouddriver.openstack.deploy.ops.servergroup +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory import com.netflix.spinnaker.clouddriver.data.task.Task import com.netflix.spinnaker.clouddriver.data.task.TaskRepository import com.netflix.spinnaker.clouddriver.openstack.client.OpenstackClientProvider @@ -25,7 +27,6 @@ import com.netflix.spinnaker.clouddriver.openstack.deploy.description.servergrou import com.netflix.spinnaker.clouddriver.openstack.deploy.description.servergroup.UserDataType import com.netflix.spinnaker.clouddriver.openstack.deploy.exception.OpenstackOperationException import com.netflix.spinnaker.clouddriver.openstack.deploy.exception.OpenstackProviderException -import com.netflix.spinnaker.clouddriver.openstack.deploy.ops.OpenstackUserDataProvider import com.netflix.spinnaker.clouddriver.openstack.security.OpenstackCredentials import com.netflix.spinnaker.clouddriver.openstack.security.OpenstackNamedAccountCredentials import org.openstack4j.model.heat.Stack @@ -35,6 +36,7 @@ import org.openstack4j.model.network.ext.LoadBalancerV2 import org.openstack4j.openstack.networking.domain.ext.ListItem import spock.lang.Specification import spock.lang.Subject +import spock.lang.Unroll class DeployOpenstackAtomicOperationSpec extends Specification { String accountName = 'myaccount' @@ -83,7 +85,6 @@ class DeployOpenstackAtomicOperationSpec extends Specification { // Add the computed parts to the server group params expectedServerGroupParams = serverGroupParams.clone() expectedServerGroupParams.with { - it.resourceFilename = 'servergroup_resource.yaml' it.networkId = '1234' it.rawUserData = '' } @@ -224,4 +225,75 @@ class DeployOpenstackAtomicOperationSpec extends Specification { actual.cause == throwable } + @Unroll + def "creates HEAT template: #type"() { + given: + def mapper = new ObjectMapper(new YAMLFactory()) + @Subject def operation = new DeployOpenstackAtomicOperation(description) + String createdStackName = 'app-stack-details-v000' + if (fip) { + description.serverGroupParameters.floatingNetworkId = "net-9876" + } + + if (!loadBalancers) { + description.serverGroupParameters.loadBalancers = [] + tags = [] + } + + when: + operation.operate([]) + + then: + 1 * provider.listStacks(region) >> [] + if (loadBalancers) { + 1 * provider.getLoadBalancer(region, lbId) >> mockLb + 1 * provider.getListener(region, listenerId) >> mockListener + } + 1 * provider.getSubnet(region, subnetId) >> mockSubnet + 1 * provider.deploy(region, createdStackName, { assertTemplate(it, mainTemplate) }, { assertTemplates(it, subtemplates)}, { params(it) }, _ as Boolean, _ as Long, tags) + noExceptionThrown() + + where: + type | fip | loadBalancers || mainTemplate | subtemplates | params + "no fip, no load balancers" | false | false || exampleTemplate("servergroup.yaml") | ["servergroup_resource.yaml": exampleTemplate("servergroup_server.yaml")] | { ServerGroupParameters params -> true } + "fip, no load balancers" | true | false || exampleTemplate("servergroup_float.yaml") | ["servergroup_resource.yaml": exampleTemplate("servergroup_server_float.yaml")] | { ServerGroupParameters params -> true } + "no fip, load balancers" | false | true || exampleTemplate("servergroup.yaml") | ["servergroup_resource.yaml": exampleTemplate("servergroup_resource.yaml"), "servergroup_resource_member.yaml": memberDataTemplate()] | { ServerGroupParameters params -> true } + "fip, load balancers" | true | true || exampleTemplate("servergroup_float.yaml") | ["servergroup_resource.yaml": exampleTemplate("servergroup_resource_float.yaml"), "servergroup_resource_member.yaml": memberDataTemplate()] | { ServerGroupParameters params -> true } + } + + private boolean assertTemplate(String actual, String expected) { + def mapper = new ObjectMapper(new YAMLFactory()) + return mapper.readValue(actual, Map) == mapper.readValue(expected, Map) + } + + private boolean assertTemplates(Map actual, Map expected) { + def mapper = new ObjectMapper(new YAMLFactory()) + return actual.collectEntries {k, v -> [(k): mapper.readValue(v, Map)]} == expected.collectEntries { k, v -> [(k): mapper.readValue(v, Map)] } + } + + private String exampleTemplate(String name) { + DeployOpenstackAtomicOperationSpec.class.getResource(name).getText("utf-8") + } + + private String memberDataTemplate() { + return """\ +--- +heat_template_version: "2016-04-08" +description: "Pool members for autoscaling group resource" +parameters: + address: + type: "string" + description: "Server address for autoscaling group resource" +resources: + member-mockpool-99-null-null: + type: "OS::Neutron::LBaaS::PoolMember" + properties: + address: + get_param: "address" + pool: "8888" + protocol_port: null + subnet: "1234" +""" + } + } diff --git a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentialsSpec.groovy b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentialsSpec.groovy index 4e3dc776ee8..982c07f6147 100644 --- a/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentialsSpec.groovy +++ b/clouddriver-openstack/src/test/groovy/com/netflix/spinnaker/clouddriver/openstack/security/OpenstackNamedAccountCredentialsSpec.groovy @@ -17,12 +17,15 @@ package com.netflix.spinnaker.clouddriver.openstack.security import com.netflix.spinnaker.clouddriver.consul.config.ConsulConfig +import com.netflix.spinnaker.clouddriver.openstack.client.OpenstackClientProvider import com.netflix.spinnaker.clouddriver.openstack.client.OpenstackIdentityV3Provider import com.netflix.spinnaker.clouddriver.openstack.config.OpenstackConfigurationProperties.LbaasConfig import org.openstack4j.api.OSClient import org.openstack4j.api.client.IOSClientBuilder import org.openstack4j.model.identity.v3.Token +import org.openstack4j.openstack.compute.domain.ext.ExtAvailabilityZone import spock.lang.Specification +import spock.lang.Unroll class OpenstackNamedAccountCredentialsSpec extends Specification { @@ -49,4 +52,32 @@ class OpenstackNamedAccountCredentialsSpec extends Specification { client instanceof OSClient.OSClientV3 } + static def azA = new ExtAvailabilityZone(zoneName: "azA", zoneState: new ExtAvailabilityZone.ExtZoneState(available: true)) + static def azB = new ExtAvailabilityZone(zoneName: "azB", zoneState: new ExtAvailabilityZone.ExtZoneState(available: true)) + static def azUnavailable = new ExtAvailabilityZone(zoneName: "azC", zoneState: new ExtAvailabilityZone.ExtZoneState(available: false)) + + @Unroll() + def "Builder populates region-to-zone map: #description"() { + setup: + OpenstackClientProvider mockProvider = Mock(OpenstackClientProvider) + OpenstackCredentials.metaClass.getProvider = { mockProvider } + + when: + def builder = new OpenstackNamedAccountCredentials.Builder() + builder.regions = regions + def account = builder.build() + + then: + 1 * mockProvider.getZones("r1") >> r1_zones + _ * mockProvider.getZones("r2") >> r2_zones + account.regionToZones == expected + + where: + description | regions | r1_zones | r2_zones | expected + "simple case" | ["r1"] | [azA] | null | ["r1": ["azA"]] + "multiple regions" | ["r1", "r2"] | [azA] | [azB] | ["r1": ["azA"], "r2": ["azB"]] + "multiple zones" | ["r1"] | [azA, azB] | null | ["r1": ["azA", "azB"]] + "skips unavailable zones" | ["r1"] | [azA, azUnavailable, azB] | null | ["r1": ["azA", "azB"]] + "empty region" | ["r1", "r2"] | null | [azA, azB] | ["r1": [], "r2": ["azA", "azB"]] + } } diff --git a/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup.yaml b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup.yaml new file mode 100644 index 00000000000..5e4b042bc76 --- /dev/null +++ b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup.yaml @@ -0,0 +1,184 @@ +heat_template_version: 2016-04-08 +description: Auto scaling group for Spinnaker +parameters: + flavor: + type: string + description: Flavor used by the web servers + image: + type: string + description: Image used for servers + max_size: + type: number + description: Maximum cluster size + min_size: + type: number + description: Minimum cluster size + desired_size: + type: number + description: Desired cluster size + network_id: + type: string + description: Network used by the servers. Retained for auditing purposes. + load_balancers: + type: comma_delimited_list + description: Comma-separated string of load balancers to associate to the stack. This is not used in the stack and is defined for auditing purposes. + default: [] + zones: + type: comma_delimited_list + description: Comma-separated string of availability zones + default: [] + security_groups: + type: comma_delimited_list + description: Comma-separated string of security groups to use + subnet_id: + type: string + description: Subnet used to allocate a fixed IP for each server + autoscaling_type: + type: string + description: Type of autoscaling to perform. can be cpu_util, network.incoming.bytes.rate, or network.outgoing.bytes.rate + default: cpu_util + scaleup_cooldown: + type: number + description: Minimum amount of time (in seconds) between scaleup operations + default: 60 + scaleup_adjustment: + type: number + description: Amount by which to change the instance count. Must be positive + default: 1 + scaleup_period: + type: number + description: Amount of time (in seconds) before the scaleup action is taken + default: 60 + scaleup_threshold: + type: number + description: Threshold that causes the scaleup action to occur, if held for scaleup_period seconds + default: 50 + scaledown_cooldown: + type: number + description: Minimum amount of time (in seconds) between scaledown operations + default: 60 + scaledown_adjustment: + type: number + description: Amount by which to change the instance count. Must be negative + default: -1 + scaledown_period: + type: number + description: Amount of time (in seconds) before the scaledown action is taken + default: 600 + scaledown_threshold: + type: number + description: Threshold that causes the scaledown action to occur, if held for scaledown_period seconds + default: 15 + source_user_data_type: + type: string + description: The source user data type (Swift, URL, Text), retained for auditing purposes + default: "" + source_user_data: + type: string + description: The unencoded source user data, retained for auditing purposes + default: "" + tags: + type: json + description: Map of key-value pairs to store in instance metadata + default: {} + user_data: + type: string + description: Raw base64-encoded string that will execute upon server boot, if cloud-init is installed + default: "" + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} +resources: + servergroup: + type: OS::Heat::AutoScalingGroup + properties: + min_size: {get_param: min_size} + max_size: {get_param: max_size} + desired_capacity: {get_param: desired_size} + resource: + type: servergroup_resource.yaml + properties: + flavor: {get_param: flavor} + image: {get_param: image} + # metering.stack is used by ceilometer to autoscale against instances that are part of this stack + # the others are user-specified + metadata: + map_merge: + - {"metering.stack": {get_param: "OS::stack_id"}} + - {"metering.stack.name": {get_param: "OS::stack_name"}} + - {get_param: tags} + network_id: {get_param: network_id} + security_groups: {get_param: security_groups} + subnet_id: {get_param: subnet_id} + user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} + web_server_scaleup_policy: + type: OS::Heat::ScalingPolicy + properties: + adjustment_type: change_in_capacity + auto_scaling_group_id: {get_resource: servergroup} + cooldown: {get_param: scaleup_cooldown} + scaling_adjustment: {get_param: scaleup_adjustment} + web_server_scaledown_policy: + type: OS::Heat::ScalingPolicy + properties: + adjustment_type: change_in_capacity + auto_scaling_group_id: {get_resource: servergroup} + cooldown: {get_param: scaledown_cooldown} + scaling_adjustment: {get_param: scaledown_adjustment} + meter_alarm_high: + type: OS::Ceilometer::Alarm + properties: + description: Scale up if the average meter_name > scaleup_threshold for scaleup_period seconds + meter_name: {get_param: autoscaling_type} + statistic: avg + period: {get_param: scaleup_period} + evaluation_periods: 1 + threshold: {get_param: scaleup_threshold} + alarm_actions: + - {get_attr: [web_server_scaleup_policy, alarm_url]} + matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}} + comparison_operator: gt + meter_alarm_low: + type: OS::Ceilometer::Alarm + properties: + description: Scale up if the average meter_name < scaledown_threshold for scaledown_period seconds + meter_name: {get_param: autoscaling_type} + statistic: avg + period: {get_param: scaledown_period} + evaluation_periods: 1 + threshold: {get_param: scaledown_threshold} + alarm_actions: + - {get_attr: [web_server_scaledown_policy, alarm_url]} + matching_metadata: {'metadata.user_metadata.stack': {get_param: "OS::stack_id"}} + comparison_operator: lt +outputs: + OS::stack_id: + value: {get_resource: servergroup} + # we need to store subtemplate in servergroup output from create, as it is required to do an update and there is no native way + # of obtaining it from a stack + servergroup_resource: + description: servergroup_resource.yaml template value + value: {get_file: servergroup_resource.yaml } + # we need to store subtemplate in servergroup output from create, as it is required to do an update and there is no native way + # of obtaining it from a stack + servergroup_resource_member: + description: servergroup_resource_member.yaml template value + value: {get_file: servergroup_resource_member.yaml} + scale_up_url: + description: > + This URL is the webhook to scale up the autoscaling group. You + can invoke the scale-up operation by doing an HTTP POST to this + URL; no body nor extra headers are needed. + value: {get_attr: [web_server_scaleup_policy, alarm_url]} + scale_dn_url: + description: > + This URL is the webhook to scale down the autoscaling group. + You can invoke the scale-down operation by doing an HTTP POST to + this URL; no body nor extra headers are needed. + value: {get_attr: [web_server_scaledown_policy, alarm_url]} + servergroup_size: + description: > + This is the current size of the auto scaling group. + value: {get_attr: [servergroup, current_size]} diff --git a/clouddriver-openstack/src/main/resources/servergroup_float.yaml b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_float.yaml similarity index 93% rename from clouddriver-openstack/src/main/resources/servergroup_float.yaml rename to clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_float.yaml index d7896b0f4b2..6b6ede124a2 100644 --- a/clouddriver-openstack/src/main/resources/servergroup_float.yaml +++ b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_float.yaml @@ -26,6 +26,10 @@ parameters: type: comma_delimited_list description: Comma-separated string of load balancers to associate to the stack. This is not used in the stack and is defined for auditing purposes. default: [] + zones: + type: comma_delimited_list + description: Comma-separated string of availability zones + default: [] security_groups: type: comma_delimited_list description: Comma-separated string of security groups to use @@ -84,9 +88,10 @@ parameters: type: string description: Raw base64-encoded string that will execute upon server boot, if cloud-init is installed default: "" - resource_filename: - type: string - description: Member resource file name (i.e. servergroup_resource.yaml) + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} resources: servergroup: type: OS::Heat::AutoScalingGroup @@ -95,7 +100,7 @@ resources: max_size: {get_param: max_size} desired_capacity: {get_param: desired_size} resource: - type: {get_param: resource_filename} + type: servergroup_resource.yaml properties: flavor: {get_param: flavor} floating_network_id: {get_param: floating_network_id} @@ -111,6 +116,7 @@ resources: security_groups: {get_param: security_groups} subnet_id: {get_param: subnet_id} user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} web_server_scaleup_policy: type: OS::Heat::ScalingPolicy properties: @@ -157,8 +163,8 @@ outputs: # we need to store subtemplate in servergroup output from create, as it is required to do an update and there is no native way # of obtaining it from a stack servergroup_resource: - description: resource_filename template value - value: {get_file: {get_param: resource_filename} } + description: servergroup_resource.yaml template value + value: {get_file: "servergroup_resource.yaml" } # we need to store subtemplate in servergroup output from create, as it is required to do an update and there is no native way # of obtaining it from a stack servergroup_resource_member: diff --git a/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource.yaml b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource.yaml new file mode 100644 index 00000000000..cb914ee1996 --- /dev/null +++ b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource.yaml @@ -0,0 +1,56 @@ +heat_template_version: 2016-04-08 +description: A load balanced server for Spinnaker. +parameters: + flavor: + type: string + description: Flavor used by the servers. + image: + type: string + description: Image used for servers. + metadata: + type: json + description: Server instance metadata. + network_id: + type: string + description: Network used by each server. Retained for auditing purposes. + security_groups: + type: comma_delimited_list + description: Security groups associate to each server. + subnet_id: + type: string + description: Subnet used to allocate a fixed IP for each server. + user_data: + type: string + description: String that will execute upon server boot, if cloud-init is installed. + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} +resources: + server: + type: OS::Nova::Server + properties: + flavor: {get_param: flavor} + image: {get_param: image} + metadata: {get_param: metadata} + networks: + - subnet: {get_param: subnet_id} + security_groups: {get_param: security_groups} + user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} + user_data_format: RAW + member: + type: OS::Heat::ResourceGroup + properties: + resource_def: + # this is dynamically generated to associate a load balancer pool member from each listener to each server + type: servergroup_resource_member.yaml + properties: + address: {get_attr: [server, first_address]} +outputs: + server_ip: + description: IP Address of the load-balanced server + value: { get_attr: [server, first_address] } + lb_member: + description: LB member details + value: { get_attr: [member, show] } diff --git a/clouddriver-openstack/src/main/resources/servergroup_resource_float.yaml b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource_float.yaml similarity index 91% rename from clouddriver-openstack/src/main/resources/servergroup_resource_float.yaml rename to clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource_float.yaml index 0ff22cd3688..bb451fecbbf 100644 --- a/clouddriver-openstack/src/main/resources/servergroup_resource_float.yaml +++ b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_resource_float.yaml @@ -25,6 +25,10 @@ parameters: user_data: type: string description: String that will execute upon server boot, if cloud-init is installed. + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} resources: server: type: OS::Nova::Server @@ -36,6 +40,7 @@ resources: - subnet: {get_param: subnet_id} security_groups: {get_param: security_groups} user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} user_data_format: RAW server_floating_ip: type: OS::Neutron::FloatingIP diff --git a/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server.yaml b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server.yaml new file mode 100644 index 00000000000..970ca798fb4 --- /dev/null +++ b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server.yaml @@ -0,0 +1,46 @@ +heat_template_version: 2016-04-08 +description: An auto-scaled server for Spinnaker without any load balancer association. +parameters: + flavor: + type: string + description: Flavor used by the servers. + image: + type: string + description: Image used for servers. + metadata: + type: json + description: Server instance metadata. + network_id: + type: string + description: Network used by each server. Retained for auditing purposes. + security_groups: + type: comma_delimited_list + description: Security groups associate to each server. + subnet_id: + type: string + description: Subnet used to allocate a fixed IP for each server. + user_data: + type: string + description: Raw base64-encoded string that will execute upon server boot, if cloud-init is installed. + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} +resources: + server: + type: OS::Nova::Server + properties: + flavor: {get_param: flavor} + image: {get_param: image} + metadata: {get_param: metadata} + networks: + - subnet: {get_param: subnet_id} + security_groups: {get_param: security_groups} + user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} + user_data_format: RAW + +outputs: + server_ip: + description: IP Address of the load-balanced server + value: { get_attr: [server, first_address] } diff --git a/clouddriver-openstack/src/main/resources/servergroup_server_float.yaml b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server_float.yaml similarity index 90% rename from clouddriver-openstack/src/main/resources/servergroup_server_float.yaml rename to clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server_float.yaml index 9ab7d9e7201..785beedfce1 100644 --- a/clouddriver-openstack/src/main/resources/servergroup_server_float.yaml +++ b/clouddriver-openstack/src/test/resources/com/netflix/spinnaker/clouddriver/openstack/deploy/ops/servergroup/servergroup_server_float.yaml @@ -25,6 +25,10 @@ parameters: user_data: type: string description: Raw base64-encoded string that will execute upon server boot, if cloud-init is installed. + scheduler_hints: + type: json + description: Key/Value pairs in json format for scheduler_hints + default: {} resources: server: type: OS::Nova::Server @@ -36,6 +40,7 @@ resources: - subnet: {get_param: subnet_id} security_groups: {get_param: security_groups} user_data: {get_param: user_data} + scheduler_hints: {get_param: scheduler_hints} user_data_format: RAW server_floating_ip: type: OS::Neutron::FloatingIP diff --git a/clouddriver-web/clouddriver-web.gradle b/clouddriver-web/clouddriver-web.gradle index ba23a98cceb..0bb995162f9 100644 --- a/clouddriver-web/clouddriver-web.gradle +++ b/clouddriver-web/clouddriver-web.gradle @@ -37,6 +37,7 @@ dependencies { compile project(':clouddriver-docker') compile project(':clouddriver-eureka') compile project(':clouddriver-elasticsearch') + compile project(':clouddriver-elasticsearch-aws') compile project(':clouddriver-oracle-bmcs') compile project(':clouddriver-dcos') diff --git a/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/CredentialsController.groovy b/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/CredentialsController.groovy index 64b872e3b19..6150694c575 100644 --- a/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/CredentialsController.groovy +++ b/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/CredentialsController.groovy @@ -26,6 +26,7 @@ import org.springframework.context.MessageSource import org.springframework.web.bind.annotation.PathVariable import org.springframework.web.bind.annotation.RequestMapping import org.springframework.web.bind.annotation.RequestMethod +import org.springframework.web.bind.annotation.RequestParam import org.springframework.web.bind.annotation.RestController @RestController @@ -45,50 +46,20 @@ class CredentialsController { MessageSource messageSource @RequestMapping(method = RequestMethod.GET) - List list() { - def listOfMaps = accountCredentialsProvider.all.collect(this.&renderSummary) - - //temporaryWorkaroundForPOC(listOfMaps) - return listOfMaps - } - - private void temporaryWorkaroundForPOC(List listOfMaps) { - LinkedHashMap ecsAccount = new LinkedHashMap<>(); - - for (Map.Entry entry : listOfMaps.get(0).entrySet()) { - - if (entry.getKey().equals("cloudProvider")) { - ecsAccount.put(entry.getKey(), "ecs") - } else if (entry.getKey().equals("name")) { - ecsAccount.put(entry.getKey(), "ecs-acct") - } else if (entry.getKey().equals("type")) { - ecsAccount.put(entry.getKey(), "ecs") - } else { - ecsAccount.put(entry.getKey(), entry.getValue()) - } - } - - listOfMaps.add(ecsAccount) + List list(@RequestParam(value = "expand", required = false) boolean expand) { + accountCredentialsProvider.all.collect { render(expand, it) } } @RequestMapping(value = "/{name:.+}", method = RequestMethod.GET) Map getAccount(@PathVariable("name") String name) { - def accountDetail = renderDetail(accountCredentialsProvider.getCredentials(name)) + def accountDetail = render(true, accountCredentialsProvider.getCredentials(name)) if (!accountDetail) { - return renderDetail(accountCredentialsProvider.getAll().iterator().next()) // TODO - implement the ECS accounts properly, so we don't need to do this shenanigan + throw new NotFoundException("Account does not exist (name: ${name})") } return accountDetail } - Map renderSummary(AccountCredentials accountCredentials) { - render(false, accountCredentials) - } - - Map renderDetail(AccountCredentials accountCredentials) { - render(true, accountCredentials) - } - Map render(boolean includeDetail, AccountCredentials accountCredentials) { if (accountCredentials == null) { return null diff --git a/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/EntityTagsController.java b/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/EntityTagsController.java index c7a7324750a..b55e00a11ee 100644 --- a/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/EntityTagsController.java +++ b/clouddriver-web/src/main/groovy/com/netflix/spinnaker/clouddriver/controllers/EntityTagsController.java @@ -58,7 +58,7 @@ public Collection list(@RequestParam(value = "cloudProvider", requir @RequestParam(value = "account", required = false) String account, @RequestParam(value = "region", required = false) String region, @RequestParam(value = "namespace", required = false) String namespace, - @RequestParam(value = "maxResults", required = false, defaultValue = "100") int maxResults, + @RequestParam(value = "maxResults", required = false, defaultValue = "2000") int maxResults, @RequestParam Map allParameters) { Map tags = allParameters.entrySet().stream() diff --git a/gradle.properties b/gradle.properties deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~HEAD b/gradle/kotlin.gradle similarity index 58% rename from clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~HEAD rename to gradle/kotlin.gradle index fc41d3d8539..f76c5a63800 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~HEAD +++ b/gradle/kotlin.gradle @@ -1,5 +1,5 @@ /* - * Copyright 2016 Google, Inc. + * Copyright 2018 Netflix, Inc. * * Licensed under the Apache License, Version 2.0 (the "License") * you may not use this file except in compliance with the License. @@ -14,21 +14,23 @@ * limitations under the License. */ -package com.netflix.spinnaker.clouddriver.model; +apply plugin: "nebula.kotlin" -import lombok.AllArgsConstructor; -import lombok.Builder; -import lombok.Data; -import lombok.NoArgsConstructor; +def kotlinVersion = "${spinnaker.version('kotlin')}" -import java.util.Map; +configurations.all { + resolutionStrategy { + eachDependency { details -> + if (details.requested.group == "org.jetbrains.kotlin") { + details.useVersion kotlinVersion + } + } + } +} -@Data -@Builder -@AllArgsConstructor -@NoArgsConstructor -public class LoadBalancerInstance { - String id; - String zone; - Map health; +compileKotlin { + kotlinOptions { + languageVersion = "1.2" + jvmTarget = "1.8" + } } diff --git a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~4441780d8c55c8e7e5dca9a1a26378fc72e0c299 b/gradle/spek.gradle similarity index 61% rename from clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~4441780d8c55c8e7e5dca9a1a26378fc72e0c299 rename to gradle/spek.gradle index fc41d3d8539..75eefbd8f1d 100644 --- a/clouddriver-core/src/main/groovy/com/netflix/spinnaker/clouddriver/model/LoadBalancerInstance.java~4441780d8c55c8e7e5dca9a1a26378fc72e0c299 +++ b/gradle/spek.gradle @@ -1,5 +1,5 @@ /* - * Copyright 2016 Google, Inc. + * Copyright 2018 Netflix, Inc. * * Licensed under the Apache License, Version 2.0 (the "License") * you may not use this file except in compliance with the License. @@ -14,21 +14,22 @@ * limitations under the License. */ -package com.netflix.spinnaker.clouddriver.model; +apply plugin: "org.junit.platform.gradle.plugin" -import lombok.AllArgsConstructor; -import lombok.Builder; -import lombok.Data; -import lombok.NoArgsConstructor; +repositories { + jcenter() + maven { url "http://dl.bintray.com/jetbrains/spek" } +} -import java.util.Map; +dependencies { + spinnaker.group('spek') +} -@Data -@Builder -@AllArgsConstructor -@NoArgsConstructor -public class LoadBalancerInstance { - String id; - String zone; - Map health; +junitPlatform { + platformVersion junitVersion + filters { + engines { + include "spek", "junit-vintage", "junit-jupiter" + } + } } diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index 2090927b21c..352686e051e 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -1,6 +1,6 @@ -#Wed Jan 10 14:20:17 EST 2018 +#Thu Nov 26 11:25:11 PST 2015 distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-3.5-all.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-3.5-bin.zip diff --git a/settings.gradle b/settings.gradle index 552f0d4dcbb..69a75b40bdb 100644 --- a/settings.gradle +++ b/settings.gradle @@ -30,6 +30,7 @@ include 'clouddriver-core', 'clouddriver-eureka', 'clouddriver-consul', 'clouddriver-elasticsearch', + 'clouddriver-elasticsearch-aws', 'clouddriver-openstack', 'clouddriver-appengine', 'clouddriver-artifacts',