Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(provider/kubernetes): v2 generic on demand caching #1884

Merged
merged 1 commit into from
Sep 16, 2017

Conversation

lwander
Copy link
Member

@lwander lwander commented Sep 7, 2017

@lwander lwander requested a review from duftler September 7, 2017 21:49
@lwander lwander force-pushed the kubernetes-v2-on-demand branch from d93330f to f5a97ed Compare September 7, 2017 21:49
@@ -0,0 +1,70 @@
/*
Copy link
Member Author

@lwander lwander Sep 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I turned this to Java because for some reason static classes nested in interfaces in groovy aren't accessible in java

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even with public keywords? Java port is fine with me just curious.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup tried all combinations, neither intellij nor gradle would pick itup

@lwander lwander force-pushed the kubernetes-v2-on-demand branch 2 times, most recently from 427d691 to 91976de Compare September 7, 2017 21:54
@lwander lwander requested a review from jtk54 September 7, 2017 21:57
@lwander
Copy link
Member Author

lwander commented Sep 15, 2017

ping

Copy link
Contributor

@jtk54 jtk54 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment, LGTM otherwise.

@@ -0,0 +1,70 @@
/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even with public keywords? Java port is fine with me just curious.

List<CacheData> evictFromOnDemand = new ArrayList<>();

providerCache.getAll(ON_DEMAND_TYPE, primaryKeys).forEach(cd -> {
// can't be a ternary op due to restrictions on non-statement expressions in lambdas
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bummer.

String name = getResourceNameFromOnDemandRequest(data).orElse("");
String namespace = (String) data.get("namespace");
if (StringUtils.isEmpty(namespace)) {
namespace = (String) data.get("region"); // sigh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe provide details for the sigh or leave it out. Seems like there is a helper function that could help self document why you have to (String) data.get() everything.

@jtk54
Copy link
Contributor

jtk54 commented Sep 15, 2017

Also needs a little bit of rebase.

@lwander lwander force-pushed the kubernetes-v2-on-demand branch from 91976de to e98abb4 Compare September 16, 2017 15:03
@lwander lwander merged commit 537efa7 into spinnaker:master Sep 16, 2017
@lwander lwander deleted the kubernetes-v2-on-demand branch September 16, 2017 15:17
BrunoCarrier added a commit to lookout/clouddriver that referenced this pull request Oct 26, 2017
* feat(provider/kubernetes): Add spectator to v2 client (spinnaker#1866)

* refactor(provider/kubernetes): Share caching agent abstraction (spinnaker#1865)

* feat(provider/kubernetes): Fix on demand caching (spinnaker#1870)

* chore(provider/kubernetes): On demand caching tests (spinnaker#1871)

* feat(provider/kuberenetes): V2 deployments (spinnaker#1868)

* feat(provider/kubernetes): Cache arbitrary resource (spinnaker#1873)

* feat(search): update application search (spinnaker#1875)

* add ability to pass search filters down to application searching in
front50.

* Additional fix for issue 1632, not able to find AMI with encrypted snapshots in target/managed account (spinnaker#1846)

* docs(intellij): Describe how to enable lombok in intellij (spinnaker#1869)

I figured we'd add a quick one-line how-to for getting gradle to build your intellij project too.

* feat(provider/kubernetes): Infer relationships from ownerReference (spinnaker#1876)

* feat(provider/kubernetes): Infer relationships from ownerReference

* feat(provider/kubernetes): Cache ancillary resources

* feat(search): add filters to project search (spinnaker#1878)

* feat(provider/ecs): Made EcsCloudProvider public (spinnaker#1848)

* feat(provider/kubernetes): Cache pods (spinnaker#1879)

* feat(provider/kubernetes): Annotate templates as well (spinnaker#1880)

A number of k8s resources use a spec.template to deploy a managed
resource. These need to be annotated by Spinnaker as well.

* fix(aws/tags): ensure tag sync is always performed during allow launch

* fix(search): allow SearchProvider filter exclusion (spinnaker#1883)

* allows search providers to specify an exclusion list for filter keys
to optionally be removed before executing a search.

* feat(provider/aws): Remove ability to automatically remove dependencies when deleting a security group (spinnaker#1894)

* fix(provider/aws): trim whitespace when newlines are detected (spinnaker#1844)

* fix(google): Add statusCode tag to google API metrics (spinnaker#1886)

This also refactors all the tests involving google API metrics.

* fix(provider/kubernetes): Add missing credential props (spinnaker#1897)

* feat(provider/kubernetes): add daemon & stateful set support

*  fix(provider/aws): ensure STSAssumeRoleSessionCredentialsProvider get the right endpoint (spinnaker#1888)

* feat(provider/kubernetes): v2 generic on demand caching (spinnaker#1884)

* refactor(provider/kubernetes): v2 include api version in resource id (spinnaker#1901)

* feat(provider/docker): add catalogFile option (spinnaker#1890)

adds the `catalogFile` option to the Docker Registry provider. This is
an attempt to make it easier to use registries that don't support
`v2/_catalog` (which happens to be a lot). This file can be managed by
an external process. Each caching cycle, Clouddriver reads this file to
determine the list of repositories it should cache tags for.

the format of the `configFile` is:
```
{
  "repositories":[
    "library/nginx",
    "library/busybox"
  ]
}
```

* fix(provider/docker): fix client constructor (spinnaker#1903)

Bug introduced in `f70d6ea`. I updated the constructor to include
`catalogFile` but didn't update the constructor for those using
`password` and `passwordFile` configurations. This fixes that potential
issue.

* feat(provider/kubernetes): v2 Cache lb & scg relationships (spinnaker#1902)

* Revert " fix(provider/aws): ensure STSAssumeRoleSessionCredentialsProvider get the right endpoint (spinnaker#1888)" (spinnaker#1904)

This reverts commit 828fbb7.

* feat(provider/kubernetes): v2 version deployed manifests as resources (spinnaker#1905)

* refactor(provider/kubernetes): V2 swap kind & version in cache key (spinnaker#1907)

For consistency with naming & other resources.

* fix(provider/aws): Clone spot price if not explicitly specified. (spinnaker#1908)

* refactor(provider/kubernetes): Segregate k8s by provider version (spinnaker#1909)

* feat(provider/kubernetes): Add proper support for source capacity. (spinnaker#1906)

Fully addresses: spinnaker/spinnaker#1657

  Also accounts for the possibility that the user might want to set
  their own targetSize while ignoring the source capacity.

  Deck work is still needed to expose "useSourceCapacity" in the UI.

  Fix cross namespace server groups

* feat(provider/kubernetes): Register k8s moniker for manifests (spinnaker#1910)

* feat(provider/kubernetes): V2 cluster provider (spinnaker#1911)

* refactor(core): s/version/providerVersion (spinnaker#1914)

Having the account property named 'version' was confusing the concept of
the version the provider was at, with the version the cloud environment
was at.

* refactor(provider/kubernetes): Rename duplicate classes (spinnaker#1913)

* feat(moniker): Adds moniker to all clusters & server groups (spinnaker#1915)

* chore(javadocs): Fix lt usage (spinnaker#1916)

* feat(provider/kubernetes): Attach k8s annotation monikers to v2 (spinnaker#1917)

resources

* fix(aws): CopyLastAsgAtomicOperation instance monitoring.

When not explicitly specified, instance monitoring would default to true due to groovy truth casting a non
null object to true instead of traversing into its enabled attribute

* fix(web): Guard against not finding a ClusterProvider (spinnaker#1921)

* feat(search): add fallback query param to search (spinnaker#1912)

* in an effort to speed up searching, this adds logic that attempts to
pick a "good" value to use as a search query string rather than just
using the empty string and pulling all the keys for a given type.

* feat(provider/kubernetes): V2 Cluster details (spinnaker#1918)

* chore(provider/kubernetes): Use version 0.2.0 of client-java (spinnaker#1923)

* fix(provider/kubernetes): Advanced targetSize use case. (spinnaker#1920)

The elvis operator before was not evaluating truthiness correctly for a value
of 0. A User could and may want to deploy an update ReplicaSet of 0 and Scale
it later. Or perhaps it's bundled in a Spinnaker Pipeline with multiple Server
Groups and one of the Server Groups is scaled to 0. If the User is implementing
'useSourceCapacity' we would want that 0 value respected and not overwritten
with a value of 1.

The proposal is that the desired capacity trumps targetSize.  It's
reasonable to assume that someone who wants a targetSize of 0 would also
have a desired capacity of 0.  However, we shouldn't use the targetSize
if the desired capacity is greater than its value.

* refactor(provider/kubernetes): v2 cache view lives in cache package (spinnaker#1926)

* feat(provider/kubernetes): v2 cache network policies (spinnaker#1927)

* feat(provider/kubernetes): Build v2 creds based on context & kubeconfigFile (spinnaker#1925)

* feat(provider/kubernetes): Integrate moniker into v2 (spinnaker#1929)

* fix(provider/kubernetes): v2 guard against empty cache relationships (spinnaker#1932)

* fix(provider/kuberentes): v2 guard against non-existent clusters (spinnaker#1934)

* feat(moniker): adds moniker to server group view model (spinnaker#1936)

* feat(provider/kubernetes): Cache v2 resources as artifacts (spinnaker#1931)

* fix(all): Broken k8s server group lookup breaking all components (spinnaker#1937)

@lwander I'm merging for short term fix since everyone is affected. However see my question about returning null and whether there are other implications.

* feat(provider/google): Adds support for multiple named ports in load balancers. (spinnaker#1930)

* refactor(provider/kubernetes): refactor artifact gen (spinnaker#1938)

* feat(moniker): Adds moniker to pendingOnDemandResults (spinnaker#1940)

* fix(provider/google): Fix named port defaults in server group deploy. (spinnaker#1939)

* feat(moniker): Add monikers to security groups & load balancers (spinnaker#1941)

* feat(provider/google): Support connection draining for LBs (spinnaker#1943)

* fix(provider/gce): s/it/cacheData/ in zonal svg caching agent. (spinnaker#1944)

* feat(provider/docker): Added insecure registry support (spinnaker#1887)

* feat(provider/docker): Added insecure registry support

Added an insecureRegistry flag to clouddriver-docker registry
support that will skip SSL certificate validation.

This will allow for addressing spinnaker/halyard#680.

* Update DockerRegistryNamedAccountCredentials.groovy

* Update DockerRegistryClientSpec.groovy

* feat(appengine): Deploy from Google Cloud Storage [accounts]. (spinnaker#1935)

* fix(provider/docker): Assign proper Spinnaker/<version> user agent (spinnaker#1946)

* feat(provider/kubernetes): Properly version all versioned resources (spinnaker#1945)

* fix(rollbacks): support for tolerating some instance failures (spinnaker#1922)

* fix(provider/gce): Paginate server groups when calculating next name. (spinnaker#1948)

* fix(appengine): Dont assume storage.gce.enabled=true (spinnaker#1953)

* fix(provider/kubernetes): fix k8s client configuration and image id p… (spinnaker#1933)

* fix(provider/kubernetes): fix k8s client configuration and image id problem.

* (feat/cats) Add support for an error interval for clustered agent schedulers (spinnaker#1895)

* feat(search): cull missing ASGs before returning (spinnaker#1951)

* removes ASGs that show up in the search results from the redis cache
but don't actually exist.

* fix(provider/kubernetes): Fixed an issue with jobProvider failing out other providers. (spinnaker#1690)

* fix(provider/dcos): fix cluster provider to guard against errors from invalid ids (spinnaker#1928)

* feat(core): Add provider version to creds controller (spinnaker#1958)

* feat(appengine): Add AppengineStorageController to list accounts (spinnaker#1957)

* chore(provider/kubernetes): Bump client to 1.0.0-beta1 (spinnaker#1961)

* fix(provider/kubernetes): v1 client call to list pods (spinnaker#1963)

* fix(provider/aws): Return load balancers for all target groups associated with an application (spinnaker#1955)

* refactor(provider/kubernetes): V2 deploy 1 manifest at a time (spinnaker#1965)

* feat(provider/gae): Modify deploy description and validators for artifacts. (spinnaker#1966)

* feat(provider/kubernetes): Feature-flag debug mode (spinnaker#1968)

* fix(provider/kubernetes): Don't store-unowned artifacts (spinnaker#1969)

* feat(provider/kuberentes): Application provider (spinnaker#1971)

* fix(provider/aws): fix issue with VPCZoneIdentifier containing deleted subnets (spinnaker#1972)

* chore(provider/docker): Only log trailing whitespace warning during failure (spinnaker#1973)

* feat(provider/kubernetes): V2 cluster provider finishing touches (spinnaker#1974)

* fix(provider/aws): Allow updating healthy threshold count on target groups (spinnaker#1977)

* feat(moniker): Add getServerGroup endpoint that does not require application name. (spinnaker#1967)

* feat(provider/kubernetes): v2 instance counts (spinnaker#1975)

* feat(provider/kuberentes): Register kind mapping using autowired deploy (spinnaker#1978)

* refactor(provider/kubernetes): Move manifest desc. into package (spinnaker#1979)

* refactor(provider/kubernetes): Move ops w/ kind into v1 only (spinnaker#1980)

* feat(provider/kubernetes): Support resize statefulset operation. (spinnaker#1976)

* feat(dynomite): hashtags & dynomite pipelines (spinnaker#1832)

* refactor(provider/google): Rely on cached instance details instead of calling platform MIG.listInstances() api. (spinnaker#1983)

* feat(provider/amazon): Create app elb specific security group for new ELBs (spinnaker#1982)

- Autocreate elb security group and ingress on application security group

* feat(core/search): Improve search speed: faster validation that search keys exists (spinnaker#1984)

* feat(provider/kubernetes): Support resize operations (spinnaker#1981)

* feat(provider/kubernetes): V2 instance state (spinnaker#1986)

* fix(provider/kubernetes): Fix v2 operation converter (spinnaker#1987)

* feat(provider/gce): Replace instanceTemplate get with list call. (spinnaker#1990)

* refactor(provider/kubernetes): use deployed names in result (spinnaker#1989)

* fix(core): Support the eviction of stale cache key identifiers (spinnaker#1985)

This PR attempts to clean up situations where the `:members` set for a
given cached type contains keys that no longer exist.

In particular, a `CachingAgent` can optionally return a pattern glob for
each authoritative type.

Any `:members` key matching the pattern glob BUT not included in the
most recent caching cycle will be evicted.

This has been enabled on a select few `aws` caching agents.

* fix(aws): Do not do stale key evictions of amazon load balancers (spinnaker#1993)

Seeing some mixed results when application/network and classic load
balancers are being used.

* feat(provider/kubernetes): v2 on-demand manifest caching (spinnaker#1988)

* feat(provider/aws): add executionId to user-agent for cloudtrail (spinnaker#1992)

* fix(aws): Enable stale key evictions for amazon load balancers (spinnaker#1994)

Key structures vary between classic and application/networking load
balancers.

* feat(provider/kubernetes): Re-encode api types (spinnaker#1991)

* feat(logging) compile logstashEncoder for json logging (spinnaker#1995)

* fix(provider/gae): Set path correct when application root non-empty. (spinnaker#1970)

* chore(dependencies): Bump spinnaker-dependencies to 0.115.0. (spinnaker#1996)

* fix hidden exception while pulling docker tags (spinnaker#1708)

* fix(provider/kubernetes): reads correct fields from cache refresh (spinnaker#1998)

* feat(provider/kubernetes): v2 delete operation (spinnaker#1999)

* feat(provider/dcos): Adding proxy for marathon-client to make use of spectator. (spinnaker#1960)

* fix(appengine): fix deploy failures (spinnaker#2001)

* feat(provider/kubernetes): Supply app name in attributes (spinnaker#2003)

* feat(provider/kuberentes): implement capacity (spinnaker#2004)

* feat(cats): Adding cache gzip compression (spinnaker#2000)

* feat(provider/kubernetes): Update deployable's patch behavior (spinnaker#2006)

* feat(provider/kubernetes): Show more in the error message (spinnaker#2007)

* feat(provider/kubernetes): V2 Cache services (spinnaker#2009)

* fix(web): Fixing missing property exception (spinnaker#2008)

* fix(cats): Get bytes in consistent chatset during decompress (spinnaker#2010)

* fix(cats): Remove use of unmodifiable collections (spinnaker#2011)

* feat(provider/kubernetes): cache non-obvious relationship (spinnaker#2012)

* feat(provider/kubernetes): load balancer -> sg relationship (spinnaker#2013)

* refactor(aws): Simplified ELB security group autocreation (spinnaker#1997)

- updated code that allows ingress from elb group to app security group

* fix(provider/gae): Delete gcs target directory in deploy. (spinnaker#2015)

* fix(provider/gae): Delete gcs target directory in deploy.

* fix(storage): Change the storage account endpoint to '/storage'.

* feat(provider/kubernetes): Enable setting of labels on k8s Service objects spinnaker/spinnaker#2035 (spinnaker#2014)

* fix(provider/kubernetes): allows dots in secretNames (spinnaker#2017)

The kubernetes api allows dots in secret names:
https://github.com/kubernetes/kubernetes/blob/e5ac41331119bb8c67004b8efaa0c3d942b31d3a/pkg/api/validation/validation.go#L160-L162
https://github.com/kubernetes/kubernetes/blob/8d7d7a5e0d4d7e75f5a860574346944b8cc0fc43/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go#L126-L142
https://github.com/kubernetes/kubernetes/blob/e5ac41331119bb8c67004b8efaa0c3d942b31d3a/pkg/api/validation/validation.go#L1488
This allows dots in the secretName field of volume source definitions.

* perf(cluster): Avoid prematurely fetching full server groups (spinnaker#2019)

This is an optimization for fetching target server groups by `CURRENT`,
`PREVIOUS`, `OLDEST` and `FAIL`.

For these strategies, the unexpanded server group is sufficient to
narrow down to a match.

Once a match is found, _only_ that server group will be expanded.

This only works for cluster providers that support loading both
expanded and unexpanded server groups (ie. aws). The behavior for other
providers should be unchanged.

* fix(aws): Fix NPE in elb group autocreation (spinnaker#2020)

- fixed NPE due to missing credentials in security group description request

* feat(provider/kubernetes): Enable annotations, labels and secrets for security groups (aka ingress resources) (spinnaker#2000) (spinnaker#2005)

* feat(google): Instrument individual google API calls. (spinnaker#2016)

* fix(cats): Remove unmodifiable collections from modifiable codepaths (spinnaker#2022)

* fix(provider/gce): Fix broken deploys with namedPorts. (spinnaker#2023)

* fix(aws): Only autocreate app elb security group on Create (spinnaker#2025)

- Updated to only auto create <appname>-elb group for new ELBs

* feat(provider/aws): Support specifying explicit subnet ids for deploy (spinnaker#2026)

Expectation is that a `subnetType` will still be provided and that
`subnetIds` will be a subnet of those valid for `subnetType`.

When explicit subnet ids are provided (or inherited), the newly created
server group will be tagged with:

`SPINNAKER_SUBNET_ID_OVERRIDE`: `"comma-separated list of subnet ids"`

* fix(provider/gce): Paginate instance template list in svg caching. (spinnaker#2027)

* fix(eureka): fix instanceId on eureka caching

The instanceId was intended to be the value in dataCenterInfo.metadata but due to some
Jackson behaviour the top level instanceId from the discovery registration was written
over the dataCenterInfo.metadata during deserialization.

This changes to explicitly handle the top level instanceId value in the JsonCreator factory
method to prevent the unhandled property behaviour from kicking in.

* Merged upstream, updated AbstractEcsCachingAgent and Keys.java
BrunoCarrier added a commit to lookout/clouddriver that referenced this pull request Oct 26, 2017
* feat(provider/kubernetes): Add spectator to v2 client (spinnaker#1866)

* refactor(provider/kubernetes): Share caching agent abstraction (spinnaker#1865)

* feat(provider/kubernetes): Fix on demand caching (spinnaker#1870)

* chore(provider/kubernetes): On demand caching tests (spinnaker#1871)

* feat(provider/kuberenetes): V2 deployments (spinnaker#1868)

* feat(provider/kubernetes): Cache arbitrary resource (spinnaker#1873)

* feat(search): update application search (spinnaker#1875)

* add ability to pass search filters down to application searching in
front50.

* Additional fix for issue 1632, not able to find AMI with encrypted snapshots in target/managed account (spinnaker#1846)

* docs(intellij): Describe how to enable lombok in intellij (spinnaker#1869)

I figured we'd add a quick one-line how-to for getting gradle to build your intellij project too.

* feat(provider/kubernetes): Infer relationships from ownerReference (spinnaker#1876)

* feat(provider/kubernetes): Infer relationships from ownerReference

* feat(provider/kubernetes): Cache ancillary resources

* feat(search): add filters to project search (spinnaker#1878)

* feat(provider/ecs): Made EcsCloudProvider public (spinnaker#1848)

* feat(provider/kubernetes): Cache pods (spinnaker#1879)

* feat(provider/kubernetes): Annotate templates as well (spinnaker#1880)

A number of k8s resources use a spec.template to deploy a managed
resource. These need to be annotated by Spinnaker as well.

* fix(aws/tags): ensure tag sync is always performed during allow launch

* fix(search): allow SearchProvider filter exclusion (spinnaker#1883)

* allows search providers to specify an exclusion list for filter keys
to optionally be removed before executing a search.

* feat(provider/aws): Remove ability to automatically remove dependencies when deleting a security group (spinnaker#1894)

* fix(provider/aws): trim whitespace when newlines are detected (spinnaker#1844)

* fix(google): Add statusCode tag to google API metrics (spinnaker#1886)

This also refactors all the tests involving google API metrics.

* fix(provider/kubernetes): Add missing credential props (spinnaker#1897)

* feat(provider/kubernetes): add daemon & stateful set support

*  fix(provider/aws): ensure STSAssumeRoleSessionCredentialsProvider get the right endpoint (spinnaker#1888)

* feat(provider/kubernetes): v2 generic on demand caching (spinnaker#1884)

* refactor(provider/kubernetes): v2 include api version in resource id (spinnaker#1901)

* feat(provider/docker): add catalogFile option (spinnaker#1890)

adds the `catalogFile` option to the Docker Registry provider. This is
an attempt to make it easier to use registries that don't support
`v2/_catalog` (which happens to be a lot). This file can be managed by
an external process. Each caching cycle, Clouddriver reads this file to
determine the list of repositories it should cache tags for.

the format of the `configFile` is:
```
{
  "repositories":[
    "library/nginx",
    "library/busybox"
  ]
}
```

* fix(provider/docker): fix client constructor (spinnaker#1903)

Bug introduced in `f70d6ea`. I updated the constructor to include
`catalogFile` but didn't update the constructor for those using
`password` and `passwordFile` configurations. This fixes that potential
issue.

* feat(provider/kubernetes): v2 Cache lb & scg relationships (spinnaker#1902)

* Revert " fix(provider/aws): ensure STSAssumeRoleSessionCredentialsProvider get the right endpoint (spinnaker#1888)" (spinnaker#1904)

This reverts commit 828fbb7.

* feat(provider/kubernetes): v2 version deployed manifests as resources (spinnaker#1905)

* refactor(provider/kubernetes): V2 swap kind & version in cache key (spinnaker#1907)

For consistency with naming & other resources.

* fix(provider/aws): Clone spot price if not explicitly specified. (spinnaker#1908)

* refactor(provider/kubernetes): Segregate k8s by provider version (spinnaker#1909)

* feat(provider/kubernetes): Add proper support for source capacity. (spinnaker#1906)

Fully addresses: spinnaker/spinnaker#1657

  Also accounts for the possibility that the user might want to set
  their own targetSize while ignoring the source capacity.

  Deck work is still needed to expose "useSourceCapacity" in the UI.

  Fix cross namespace server groups

* feat(provider/kubernetes): Register k8s moniker for manifests (spinnaker#1910)

* feat(provider/kubernetes): V2 cluster provider (spinnaker#1911)

* refactor(core): s/version/providerVersion (spinnaker#1914)

Having the account property named 'version' was confusing the concept of
the version the provider was at, with the version the cloud environment
was at.

* refactor(provider/kubernetes): Rename duplicate classes (spinnaker#1913)

* feat(moniker): Adds moniker to all clusters & server groups (spinnaker#1915)

* chore(javadocs): Fix lt usage (spinnaker#1916)

* feat(provider/kubernetes): Attach k8s annotation monikers to v2 (spinnaker#1917)

resources

* fix(aws): CopyLastAsgAtomicOperation instance monitoring.

When not explicitly specified, instance monitoring would default to true due to groovy truth casting a non
null object to true instead of traversing into its enabled attribute

* fix(web): Guard against not finding a ClusterProvider (spinnaker#1921)

* feat(search): add fallback query param to search (spinnaker#1912)

* in an effort to speed up searching, this adds logic that attempts to
pick a "good" value to use as a search query string rather than just
using the empty string and pulling all the keys for a given type.

* feat(provider/kubernetes): V2 Cluster details (spinnaker#1918)

* chore(provider/kubernetes): Use version 0.2.0 of client-java (spinnaker#1923)

* fix(provider/kubernetes): Advanced targetSize use case. (spinnaker#1920)

The elvis operator before was not evaluating truthiness correctly for a value
of 0. A User could and may want to deploy an update ReplicaSet of 0 and Scale
it later. Or perhaps it's bundled in a Spinnaker Pipeline with multiple Server
Groups and one of the Server Groups is scaled to 0. If the User is implementing
'useSourceCapacity' we would want that 0 value respected and not overwritten
with a value of 1.

The proposal is that the desired capacity trumps targetSize.  It's
reasonable to assume that someone who wants a targetSize of 0 would also
have a desired capacity of 0.  However, we shouldn't use the targetSize
if the desired capacity is greater than its value.

* refactor(provider/kubernetes): v2 cache view lives in cache package (spinnaker#1926)

* feat(provider/kubernetes): v2 cache network policies (spinnaker#1927)

* feat(provider/kubernetes): Build v2 creds based on context & kubeconfigFile (spinnaker#1925)

* feat(provider/kubernetes): Integrate moniker into v2 (spinnaker#1929)

* fix(provider/kubernetes): v2 guard against empty cache relationships (spinnaker#1932)

* fix(provider/kuberentes): v2 guard against non-existent clusters (spinnaker#1934)

* feat(moniker): adds moniker to server group view model (spinnaker#1936)

* feat(provider/kubernetes): Cache v2 resources as artifacts (spinnaker#1931)

* fix(all): Broken k8s server group lookup breaking all components (spinnaker#1937)

@lwander I'm merging for short term fix since everyone is affected. However see my question about returning null and whether there are other implications.

* feat(provider/google): Adds support for multiple named ports in load balancers. (spinnaker#1930)

* refactor(provider/kubernetes): refactor artifact gen (spinnaker#1938)

* feat(moniker): Adds moniker to pendingOnDemandResults (spinnaker#1940)

* fix(provider/google): Fix named port defaults in server group deploy. (spinnaker#1939)

* feat(moniker): Add monikers to security groups & load balancers (spinnaker#1941)

* feat(provider/google): Support connection draining for LBs (spinnaker#1943)

* fix(provider/gce): s/it/cacheData/ in zonal svg caching agent. (spinnaker#1944)

* feat(provider/docker): Added insecure registry support (spinnaker#1887)

* feat(provider/docker): Added insecure registry support

Added an insecureRegistry flag to clouddriver-docker registry
support that will skip SSL certificate validation.

This will allow for addressing spinnaker/halyard#680.

* Update DockerRegistryNamedAccountCredentials.groovy

* Update DockerRegistryClientSpec.groovy

* feat(appengine): Deploy from Google Cloud Storage [accounts]. (spinnaker#1935)

* fix(provider/docker): Assign proper Spinnaker/<version> user agent (spinnaker#1946)

* feat(provider/kubernetes): Properly version all versioned resources (spinnaker#1945)

* fix(rollbacks): support for tolerating some instance failures (spinnaker#1922)

* fix(provider/gce): Paginate server groups when calculating next name. (spinnaker#1948)

* fix(appengine): Dont assume storage.gce.enabled=true (spinnaker#1953)

* fix(provider/kubernetes): fix k8s client configuration and image id p… (spinnaker#1933)

* fix(provider/kubernetes): fix k8s client configuration and image id problem.

* (feat/cats) Add support for an error interval for clustered agent schedulers (spinnaker#1895)

* feat(search): cull missing ASGs before returning (spinnaker#1951)

* removes ASGs that show up in the search results from the redis cache
but don't actually exist.

* fix(provider/kubernetes): Fixed an issue with jobProvider failing out other providers. (spinnaker#1690)

* fix(provider/dcos): fix cluster provider to guard against errors from invalid ids (spinnaker#1928)

* feat(core): Add provider version to creds controller (spinnaker#1958)

* feat(appengine): Add AppengineStorageController to list accounts (spinnaker#1957)

* chore(provider/kubernetes): Bump client to 1.0.0-beta1 (spinnaker#1961)

* fix(provider/kubernetes): v1 client call to list pods (spinnaker#1963)

* fix(provider/aws): Return load balancers for all target groups associated with an application (spinnaker#1955)

* refactor(provider/kubernetes): V2 deploy 1 manifest at a time (spinnaker#1965)

* feat(provider/gae): Modify deploy description and validators for artifacts. (spinnaker#1966)

* feat(provider/kubernetes): Feature-flag debug mode (spinnaker#1968)

* fix(provider/kubernetes): Don't store-unowned artifacts (spinnaker#1969)

* feat(provider/kuberentes): Application provider (spinnaker#1971)

* fix(provider/aws): fix issue with VPCZoneIdentifier containing deleted subnets (spinnaker#1972)

* chore(provider/docker): Only log trailing whitespace warning during failure (spinnaker#1973)

* feat(provider/kubernetes): V2 cluster provider finishing touches (spinnaker#1974)

* fix(provider/aws): Allow updating healthy threshold count on target groups (spinnaker#1977)

* feat(moniker): Add getServerGroup endpoint that does not require application name. (spinnaker#1967)

* feat(provider/kubernetes): v2 instance counts (spinnaker#1975)

* feat(provider/kuberentes): Register kind mapping using autowired deploy (spinnaker#1978)

* refactor(provider/kubernetes): Move manifest desc. into package (spinnaker#1979)

* refactor(provider/kubernetes): Move ops w/ kind into v1 only (spinnaker#1980)

* feat(provider/kubernetes): Support resize statefulset operation. (spinnaker#1976)

* feat(dynomite): hashtags & dynomite pipelines (spinnaker#1832)

* refactor(provider/google): Rely on cached instance details instead of calling platform MIG.listInstances() api. (spinnaker#1983)

* feat(provider/amazon): Create app elb specific security group for new ELBs (spinnaker#1982)

- Autocreate elb security group and ingress on application security group

* feat(core/search): Improve search speed: faster validation that search keys exists (spinnaker#1984)

* feat(provider/kubernetes): Support resize operations (spinnaker#1981)

* feat(provider/kubernetes): V2 instance state (spinnaker#1986)

* fix(provider/kubernetes): Fix v2 operation converter (spinnaker#1987)

* feat(provider/gce): Replace instanceTemplate get with list call. (spinnaker#1990)

* refactor(provider/kubernetes): use deployed names in result (spinnaker#1989)

* fix(core): Support the eviction of stale cache key identifiers (spinnaker#1985)

This PR attempts to clean up situations where the `:members` set for a
given cached type contains keys that no longer exist.

In particular, a `CachingAgent` can optionally return a pattern glob for
each authoritative type.

Any `:members` key matching the pattern glob BUT not included in the
most recent caching cycle will be evicted.

This has been enabled on a select few `aws` caching agents.

* fix(aws): Do not do stale key evictions of amazon load balancers (spinnaker#1993)

Seeing some mixed results when application/network and classic load
balancers are being used.

* feat(provider/kubernetes): v2 on-demand manifest caching (spinnaker#1988)

* feat(provider/aws): add executionId to user-agent for cloudtrail (spinnaker#1992)

* fix(aws): Enable stale key evictions for amazon load balancers (spinnaker#1994)

Key structures vary between classic and application/networking load
balancers.

* feat(provider/kubernetes): Re-encode api types (spinnaker#1991)

* feat(logging) compile logstashEncoder for json logging (spinnaker#1995)

* fix(provider/gae): Set path correct when application root non-empty. (spinnaker#1970)

* chore(dependencies): Bump spinnaker-dependencies to 0.115.0. (spinnaker#1996)

* fix hidden exception while pulling docker tags (spinnaker#1708)

* fix(provider/kubernetes): reads correct fields from cache refresh (spinnaker#1998)

* feat(provider/kubernetes): v2 delete operation (spinnaker#1999)

* feat(provider/dcos): Adding proxy for marathon-client to make use of spectator. (spinnaker#1960)

* fix(appengine): fix deploy failures (spinnaker#2001)

* feat(provider/kubernetes): Supply app name in attributes (spinnaker#2003)

* feat(provider/kuberentes): implement capacity (spinnaker#2004)

* feat(cats): Adding cache gzip compression (spinnaker#2000)

* feat(provider/kubernetes): Update deployable's patch behavior (spinnaker#2006)

* feat(provider/kubernetes): Show more in the error message (spinnaker#2007)

* feat(provider/kubernetes): V2 Cache services (spinnaker#2009)

* fix(web): Fixing missing property exception (spinnaker#2008)

* fix(cats): Get bytes in consistent chatset during decompress (spinnaker#2010)

* fix(cats): Remove use of unmodifiable collections (spinnaker#2011)

* feat(provider/kubernetes): cache non-obvious relationship (spinnaker#2012)

* feat(provider/kubernetes): load balancer -> sg relationship (spinnaker#2013)

* refactor(aws): Simplified ELB security group autocreation (spinnaker#1997)

- updated code that allows ingress from elb group to app security group

* fix(provider/gae): Delete gcs target directory in deploy. (spinnaker#2015)

* fix(provider/gae): Delete gcs target directory in deploy.

* fix(storage): Change the storage account endpoint to '/storage'.

* feat(provider/kubernetes): Enable setting of labels on k8s Service objects spinnaker/spinnaker#2035 (spinnaker#2014)

* fix(provider/kubernetes): allows dots in secretNames (spinnaker#2017)

The kubernetes api allows dots in secret names:
https://github.com/kubernetes/kubernetes/blob/e5ac41331119bb8c67004b8efaa0c3d942b31d3a/pkg/api/validation/validation.go#L160-L162
https://github.com/kubernetes/kubernetes/blob/8d7d7a5e0d4d7e75f5a860574346944b8cc0fc43/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go#L126-L142
https://github.com/kubernetes/kubernetes/blob/e5ac41331119bb8c67004b8efaa0c3d942b31d3a/pkg/api/validation/validation.go#L1488
This allows dots in the secretName field of volume source definitions.

* perf(cluster): Avoid prematurely fetching full server groups (spinnaker#2019)

This is an optimization for fetching target server groups by `CURRENT`,
`PREVIOUS`, `OLDEST` and `FAIL`.

For these strategies, the unexpanded server group is sufficient to
narrow down to a match.

Once a match is found, _only_ that server group will be expanded.

This only works for cluster providers that support loading both
expanded and unexpanded server groups (ie. aws). The behavior for other
providers should be unchanged.

* fix(aws): Fix NPE in elb group autocreation (spinnaker#2020)

- fixed NPE due to missing credentials in security group description request

* feat(provider/kubernetes): Enable annotations, labels and secrets for security groups (aka ingress resources) (spinnaker#2000) (spinnaker#2005)

* feat(google): Instrument individual google API calls. (spinnaker#2016)

* fix(cats): Remove unmodifiable collections from modifiable codepaths (spinnaker#2022)

* fix(provider/gce): Fix broken deploys with namedPorts. (spinnaker#2023)

* fix(aws): Only autocreate app elb security group on Create (spinnaker#2025)

- Updated to only auto create <appname>-elb group for new ELBs

* feat(provider/aws): Support specifying explicit subnet ids for deploy (spinnaker#2026)

Expectation is that a `subnetType` will still be provided and that
`subnetIds` will be a subnet of those valid for `subnetType`.

When explicit subnet ids are provided (or inherited), the newly created
server group will be tagged with:

`SPINNAKER_SUBNET_ID_OVERRIDE`: `"comma-separated list of subnet ids"`

* fix(provider/gce): Paginate instance template list in svg caching. (spinnaker#2027)

* fix(eureka): fix instanceId on eureka caching

The instanceId was intended to be the value in dataCenterInfo.metadata but due to some
Jackson behaviour the top level instanceId from the discovery registration was written
over the dataCenterInfo.metadata during deserialization.

This changes to explicitly handle the top level instanceId value in the JsonCreator factory
method to prevent the unhandled property behaviour from kicking in.

* target group is now a string, instead of an array of strigns
BrunoCarrier added a commit to lookout/clouddriver that referenced this pull request Nov 6, 2017
* feat(provider/kubernetes): Add spectator to v2 client (spinnaker#1866)

* refactor(provider/kubernetes): Share caching agent abstraction (spinnaker#1865)

* feat(provider/kubernetes): Fix on demand caching (spinnaker#1870)

* chore(provider/kubernetes): On demand caching tests (spinnaker#1871)

* feat(provider/kuberenetes): V2 deployments (spinnaker#1868)

* feat(provider/kubernetes): Cache arbitrary resource (spinnaker#1873)

* feat(search): update application search (spinnaker#1875)

* add ability to pass search filters down to application searching in
front50.

* Additional fix for issue 1632, not able to find AMI with encrypted snapshots in target/managed account (spinnaker#1846)

* docs(intellij): Describe how to enable lombok in intellij (spinnaker#1869)

I figured we'd add a quick one-line how-to for getting gradle to build your intellij project too.

* feat(provider/kubernetes): Infer relationships from ownerReference (spinnaker#1876)

* feat(provider/kubernetes): Infer relationships from ownerReference

* feat(provider/kubernetes): Cache ancillary resources

* feat(search): add filters to project search (spinnaker#1878)

* feat(provider/ecs): Made EcsCloudProvider public (spinnaker#1848)

* feat(provider/kubernetes): Cache pods (spinnaker#1879)

* feat(provider/kubernetes): Annotate templates as well (spinnaker#1880)

A number of k8s resources use a spec.template to deploy a managed
resource. These need to be annotated by Spinnaker as well.

* fix(aws/tags): ensure tag sync is always performed during allow launch

* fix(search): allow SearchProvider filter exclusion (spinnaker#1883)

* allows search providers to specify an exclusion list for filter keys
to optionally be removed before executing a search.

* feat(provider/aws): Remove ability to automatically remove dependencies when deleting a security group (spinnaker#1894)

* fix(provider/aws): trim whitespace when newlines are detected (spinnaker#1844)

* fix(google): Add statusCode tag to google API metrics (spinnaker#1886)

This also refactors all the tests involving google API metrics.

* fix(provider/kubernetes): Add missing credential props (spinnaker#1897)

* feat(provider/kubernetes): add daemon & stateful set support

*  fix(provider/aws): ensure STSAssumeRoleSessionCredentialsProvider get the right endpoint (spinnaker#1888)

* feat(provider/kubernetes): v2 generic on demand caching (spinnaker#1884)

* refactor(provider/kubernetes): v2 include api version in resource id (spinnaker#1901)

* feat(provider/docker): add catalogFile option (spinnaker#1890)

adds the `catalogFile` option to the Docker Registry provider. This is
an attempt to make it easier to use registries that don't support
`v2/_catalog` (which happens to be a lot). This file can be managed by
an external process. Each caching cycle, Clouddriver reads this file to
determine the list of repositories it should cache tags for.

the format of the `configFile` is:
```
{
  "repositories":[
    "library/nginx",
    "library/busybox"
  ]
}
```

* fix(provider/docker): fix client constructor (spinnaker#1903)

Bug introduced in `f70d6ea`. I updated the constructor to include
`catalogFile` but didn't update the constructor for those using
`password` and `passwordFile` configurations. This fixes that potential
issue.

* feat(provider/kubernetes): v2 Cache lb & scg relationships (spinnaker#1902)

* Revert " fix(provider/aws): ensure STSAssumeRoleSessionCredentialsProvider get the right endpoint (spinnaker#1888)" (spinnaker#1904)

This reverts commit 828fbb7.

* feat(provider/kubernetes): v2 version deployed manifests as resources (spinnaker#1905)

* refactor(provider/kubernetes): V2 swap kind & version in cache key (spinnaker#1907)

For consistency with naming & other resources.

* fix(provider/aws): Clone spot price if not explicitly specified. (spinnaker#1908)

* refactor(provider/kubernetes): Segregate k8s by provider version (spinnaker#1909)

* feat(provider/kubernetes): Add proper support for source capacity. (spinnaker#1906)

Fully addresses: spinnaker/spinnaker#1657

  Also accounts for the possibility that the user might want to set
  their own targetSize while ignoring the source capacity.

  Deck work is still needed to expose "useSourceCapacity" in the UI.

  Fix cross namespace server groups

* feat(provider/kubernetes): Register k8s moniker for manifests (spinnaker#1910)

* feat(provider/kubernetes): V2 cluster provider (spinnaker#1911)

* refactor(core): s/version/providerVersion (spinnaker#1914)

Having the account property named 'version' was confusing the concept of
the version the provider was at, with the version the cloud environment
was at.

* refactor(provider/kubernetes): Rename duplicate classes (spinnaker#1913)

* feat(moniker): Adds moniker to all clusters & server groups (spinnaker#1915)

* chore(javadocs): Fix lt usage (spinnaker#1916)

* feat(provider/kubernetes): Attach k8s annotation monikers to v2 (spinnaker#1917)

resources

* fix(aws): CopyLastAsgAtomicOperation instance monitoring.

When not explicitly specified, instance monitoring would default to true due to groovy truth casting a non
null object to true instead of traversing into its enabled attribute

* fix(web): Guard against not finding a ClusterProvider (spinnaker#1921)

* feat(search): add fallback query param to search (spinnaker#1912)

* in an effort to speed up searching, this adds logic that attempts to
pick a "good" value to use as a search query string rather than just
using the empty string and pulling all the keys for a given type.

* feat(provider/kubernetes): V2 Cluster details (spinnaker#1918)

* chore(provider/kubernetes): Use version 0.2.0 of client-java (spinnaker#1923)

* fix(provider/kubernetes): Advanced targetSize use case. (spinnaker#1920)

The elvis operator before was not evaluating truthiness correctly for a value
of 0. A User could and may want to deploy an update ReplicaSet of 0 and Scale
it later. Or perhaps it's bundled in a Spinnaker Pipeline with multiple Server
Groups and one of the Server Groups is scaled to 0. If the User is implementing
'useSourceCapacity' we would want that 0 value respected and not overwritten
with a value of 1.

The proposal is that the desired capacity trumps targetSize.  It's
reasonable to assume that someone who wants a targetSize of 0 would also
have a desired capacity of 0.  However, we shouldn't use the targetSize
if the desired capacity is greater than its value.

* refactor(provider/kubernetes): v2 cache view lives in cache package (spinnaker#1926)

* feat(provider/kubernetes): v2 cache network policies (spinnaker#1927)

* feat(provider/kubernetes): Build v2 creds based on context & kubeconfigFile (spinnaker#1925)

* feat(provider/kubernetes): Integrate moniker into v2 (spinnaker#1929)

* fix(provider/kubernetes): v2 guard against empty cache relationships (spinnaker#1932)

* fix(provider/kuberentes): v2 guard against non-existent clusters (spinnaker#1934)

* feat(moniker): adds moniker to server group view model (spinnaker#1936)

* feat(provider/kubernetes): Cache v2 resources as artifacts (spinnaker#1931)

* fix(all): Broken k8s server group lookup breaking all components (spinnaker#1937)

@lwander I'm merging for short term fix since everyone is affected. However see my question about returning null and whether there are other implications.

* feat(provider/google): Adds support for multiple named ports in load balancers. (spinnaker#1930)

* refactor(provider/kubernetes): refactor artifact gen (spinnaker#1938)

* feat(moniker): Adds moniker to pendingOnDemandResults (spinnaker#1940)

* fix(provider/google): Fix named port defaults in server group deploy. (spinnaker#1939)

* feat(moniker): Add monikers to security groups & load balancers (spinnaker#1941)

* feat(provider/google): Support connection draining for LBs (spinnaker#1943)

* fix(provider/gce): s/it/cacheData/ in zonal svg caching agent. (spinnaker#1944)

* feat(provider/docker): Added insecure registry support (spinnaker#1887)

* feat(provider/docker): Added insecure registry support

Added an insecureRegistry flag to clouddriver-docker registry
support that will skip SSL certificate validation.

This will allow for addressing spinnaker/halyard#680.

* Update DockerRegistryNamedAccountCredentials.groovy

* Update DockerRegistryClientSpec.groovy

* feat(appengine): Deploy from Google Cloud Storage [accounts]. (spinnaker#1935)

* fix(provider/docker): Assign proper Spinnaker/<version> user agent (spinnaker#1946)

* feat(provider/kubernetes): Properly version all versioned resources (spinnaker#1945)

* fix(rollbacks): support for tolerating some instance failures (spinnaker#1922)

* fix(provider/gce): Paginate server groups when calculating next name. (spinnaker#1948)

* fix(appengine): Dont assume storage.gce.enabled=true (spinnaker#1953)

* fix(provider/kubernetes): fix k8s client configuration and image id p… (spinnaker#1933)

* fix(provider/kubernetes): fix k8s client configuration and image id problem.

* (feat/cats) Add support for an error interval for clustered agent schedulers (spinnaker#1895)

* feat(search): cull missing ASGs before returning (spinnaker#1951)

* removes ASGs that show up in the search results from the redis cache
but don't actually exist.

* fix(provider/kubernetes): Fixed an issue with jobProvider failing out other providers. (spinnaker#1690)

* fix(provider/dcos): fix cluster provider to guard against errors from invalid ids (spinnaker#1928)

* feat(core): Add provider version to creds controller (spinnaker#1958)

* feat(appengine): Add AppengineStorageController to list accounts (spinnaker#1957)

* chore(provider/kubernetes): Bump client to 1.0.0-beta1 (spinnaker#1961)

* fix(provider/kubernetes): v1 client call to list pods (spinnaker#1963)

* fix(provider/aws): Return load balancers for all target groups associated with an application (spinnaker#1955)

* refactor(provider/kubernetes): V2 deploy 1 manifest at a time (spinnaker#1965)

* feat(provider/gae): Modify deploy description and validators for artifacts. (spinnaker#1966)

* feat(provider/kubernetes): Feature-flag debug mode (spinnaker#1968)

* fix(provider/kubernetes): Don't store-unowned artifacts (spinnaker#1969)

* feat(provider/kuberentes): Application provider (spinnaker#1971)

* fix(provider/aws): fix issue with VPCZoneIdentifier containing deleted subnets (spinnaker#1972)

* chore(provider/docker): Only log trailing whitespace warning during failure (spinnaker#1973)

* feat(provider/kubernetes): V2 cluster provider finishing touches (spinnaker#1974)

* fix(provider/aws): Allow updating healthy threshold count on target groups (spinnaker#1977)

* feat(moniker): Add getServerGroup endpoint that does not require application name. (spinnaker#1967)

* feat(provider/kubernetes): v2 instance counts (spinnaker#1975)

* feat(provider/kuberentes): Register kind mapping using autowired deploy (spinnaker#1978)

* refactor(provider/kubernetes): Move manifest desc. into package (spinnaker#1979)

* refactor(provider/kubernetes): Move ops w/ kind into v1 only (spinnaker#1980)

* feat(provider/kubernetes): Support resize statefulset operation. (spinnaker#1976)

* feat(dynomite): hashtags & dynomite pipelines (spinnaker#1832)

* refactor(provider/google): Rely on cached instance details instead of calling platform MIG.listInstances() api. (spinnaker#1983)

* feat(provider/amazon): Create app elb specific security group for new ELBs (spinnaker#1982)

- Autocreate elb security group and ingress on application security group

* feat(core/search): Improve search speed: faster validation that search keys exists (spinnaker#1984)

* feat(provider/kubernetes): Support resize operations (spinnaker#1981)

* feat(provider/kubernetes): V2 instance state (spinnaker#1986)

* fix(provider/kubernetes): Fix v2 operation converter (spinnaker#1987)

* feat(provider/gce): Replace instanceTemplate get with list call. (spinnaker#1990)

* refactor(provider/kubernetes): use deployed names in result (spinnaker#1989)

* fix(core): Support the eviction of stale cache key identifiers (spinnaker#1985)

This PR attempts to clean up situations where the `:members` set for a
given cached type contains keys that no longer exist.

In particular, a `CachingAgent` can optionally return a pattern glob for
each authoritative type.

Any `:members` key matching the pattern glob BUT not included in the
most recent caching cycle will be evicted.

This has been enabled on a select few `aws` caching agents.

* fix(aws): Do not do stale key evictions of amazon load balancers (spinnaker#1993)

Seeing some mixed results when application/network and classic load
balancers are being used.

* feat(provider/kubernetes): v2 on-demand manifest caching (spinnaker#1988)

* feat(provider/aws): add executionId to user-agent for cloudtrail (spinnaker#1992)

* fix(aws): Enable stale key evictions for amazon load balancers (spinnaker#1994)

Key structures vary between classic and application/networking load
balancers.

* feat(provider/kubernetes): Re-encode api types (spinnaker#1991)

* feat(logging) compile logstashEncoder for json logging (spinnaker#1995)

* fix(provider/gae): Set path correct when application root non-empty. (spinnaker#1970)

* chore(dependencies): Bump spinnaker-dependencies to 0.115.0. (spinnaker#1996)

* fix hidden exception while pulling docker tags (spinnaker#1708)

* fix(provider/kubernetes): reads correct fields from cache refresh (spinnaker#1998)

* feat(provider/kubernetes): v2 delete operation (spinnaker#1999)

* feat(provider/dcos): Adding proxy for marathon-client to make use of spectator. (spinnaker#1960)

* fix(appengine): fix deploy failures (spinnaker#2001)

* feat(provider/kubernetes): Supply app name in attributes (spinnaker#2003)

* feat(provider/kuberentes): implement capacity (spinnaker#2004)

* feat(cats): Adding cache gzip compression (spinnaker#2000)

* feat(provider/kubernetes): Update deployable's patch behavior (spinnaker#2006)

* feat(provider/kubernetes): Show more in the error message (spinnaker#2007)

* feat(provider/kubernetes): V2 Cache services (spinnaker#2009)

* fix(web): Fixing missing property exception (spinnaker#2008)

* fix(cats): Get bytes in consistent chatset during decompress (spinnaker#2010)

* fix(cats): Remove use of unmodifiable collections (spinnaker#2011)

* feat(provider/kubernetes): cache non-obvious relationship (spinnaker#2012)

* feat(provider/kubernetes): load balancer -> sg relationship (spinnaker#2013)

* refactor(aws): Simplified ELB security group autocreation (spinnaker#1997)

- updated code that allows ingress from elb group to app security group

* fix(provider/gae): Delete gcs target directory in deploy. (spinnaker#2015)

* fix(provider/gae): Delete gcs target directory in deploy.

* fix(storage): Change the storage account endpoint to '/storage'.

* feat(provider/kubernetes): Enable setting of labels on k8s Service objects spinnaker/spinnaker#2035 (spinnaker#2014)

* fix(provider/kubernetes): allows dots in secretNames (spinnaker#2017)

The kubernetes api allows dots in secret names:
https://github.com/kubernetes/kubernetes/blob/e5ac41331119bb8c67004b8efaa0c3d942b31d3a/pkg/api/validation/validation.go#L160-L162
https://github.com/kubernetes/kubernetes/blob/8d7d7a5e0d4d7e75f5a860574346944b8cc0fc43/staging/src/k8s.io/apimachinery/pkg/util/validation/validation.go#L126-L142
https://github.com/kubernetes/kubernetes/blob/e5ac41331119bb8c67004b8efaa0c3d942b31d3a/pkg/api/validation/validation.go#L1488
This allows dots in the secretName field of volume source definitions.

* perf(cluster): Avoid prematurely fetching full server groups (spinnaker#2019)

This is an optimization for fetching target server groups by `CURRENT`,
`PREVIOUS`, `OLDEST` and `FAIL`.

For these strategies, the unexpanded server group is sufficient to
narrow down to a match.

Once a match is found, _only_ that server group will be expanded.

This only works for cluster providers that support loading both
expanded and unexpanded server groups (ie. aws). The behavior for other
providers should be unchanged.

* fix(aws): Fix NPE in elb group autocreation (spinnaker#2020)

- fixed NPE due to missing credentials in security group description request

* feat(provider/kubernetes): Enable annotations, labels and secrets for security groups (aka ingress resources) (spinnaker#2000) (spinnaker#2005)

* feat(google): Instrument individual google API calls. (spinnaker#2016)

* fix(cats): Remove unmodifiable collections from modifiable codepaths (spinnaker#2022)

* fix(provider/gce): Fix broken deploys with namedPorts. (spinnaker#2023)

* fix(aws): Only autocreate app elb security group on Create (spinnaker#2025)

- Updated to only auto create <appname>-elb group for new ELBs

* feat(provider/aws): Support specifying explicit subnet ids for deploy (spinnaker#2026)

Expectation is that a `subnetType` will still be provided and that
`subnetIds` will be a subnet of those valid for `subnetType`.

When explicit subnet ids are provided (or inherited), the newly created
server group will be tagged with:

`SPINNAKER_SUBNET_ID_OVERRIDE`: `"comma-separated list of subnet ids"`

* fix(provider/gce): Paginate instance template list in svg caching. (spinnaker#2027)

* fix(eureka): fix instanceId on eureka caching

The instanceId was intended to be the value in dataCenterInfo.metadata but due to some
Jackson behaviour the top level instanceId from the discovery registration was written
over the dataCenterInfo.metadata during deserialization.

This changes to explicitly handle the top level instanceId value in the JsonCreator factory
method to prevent the unhandled property behaviour from kicking in.

* Updating API for IAM role dropdown to work nicely with Deck
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants