error creating NodePool: googleapi: got HTTP response code 404 #282

sylver · 2020-02-06T20:09:17Z

Problem

When trying to create a gcp.container.NodePool I now got this message :

 gcp:container:NodePool ({pool-name}):
    error: error creating NodePool: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
    <html lang=en>
      <meta charset=utf-8>
      <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
      <title>Error 404 (Not Found)!!1</title>
      <style>
        *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
      </style>
      <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
      <p><b>404.</b> <ins>That's an error.</ins>
      <p>The requested URL <code>/v1beta1/projects/{project_name}/locations/europe-west1-b/clusters/projects/{project_name}/locations/europe-west1-b/clusters/{cluster_name}/nodePools?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That's all we know.</ins>

This comes from a code I use since more than a year, everything was still fine like 2 months ago.
I didn't change a thing, I just needed to create a new stack/cluster, so that's likely something outside of my code. No error is reported during update on existing stacks/clusters.

Seems like the v1beta1 URL does not work anymore ?
I don't know if that's directly related but there's a deprecation warning and shutdown notice here https://cloud.google.com/kubernetes-engine/docs/how-to/protecting-cluster-metadata?hl=en_US#disable-legacy-apis

I already talked about this issue on the Slack channel https://pulumi-community.slack.com/archives/C84L4E3N1/p1580991257027200

Code

Here's a MWE to reproduce the problem.

Create a new project in GCP and Pulumi
setup the stack :

pulumi config set gcp:project  {gcp_project_name}
pulumi config set gcp:region   europe-west1
pulumi config set gcp:zone     europe-west1-b

minimal package.json :

{
  "name": "pulumi-sandbox",
  "dependencies": {
    "@cappalyst/pulumi": "1.0.8",
    "@pulumi/gcp": "2.6.0",
    "@pulumi/pulumi": "1.10.1"
  }
}

index.ts :

import * as cappalyst from '@cappalyst/pulumi'
import * as gcp from '@pulumi/gcp'

const prefix = 'sandbox'
const name = 'test-cluster'
const nodeDefaultScopes = [
  'https://www.googleapis.com/auth/compute',
  'https://www.googleapis.com/auth/devstorage.read_only',
  'https://www.googleapis.com/auth/logging.write',
  'https://www.googleapis.com/auth/monitoring',
  'https://www.googleapis.com/auth/servicecontrol',
  'https://www.googleapis.com/auth/trace.append',
]

const projectAdminAccount = new cappalyst.gcp.ServiceAccount(
  `${prefix}-project-admin`,
  {
    accountId: `${prefix}-project-admin`,
    displayName: `${prefix} project admin`,
    roles: [
      'roles/container.admin',
      'roles/iam.serviceAccountUser',
      'roles/compute.viewer',
      'roles/compute.networkAdmin',
    ],
  },
)

const projectAdminKey = projectAdminAccount.key.privateKey.apply(
  (key: string) => (
    Buffer.from(key, 'base64').toString()
  )
)

const provider = new gcp.Provider(
  `${prefix}-gcp-provider`,
  {
    credentials: projectAdminKey,
    project: gcp.config.project,
    zone: gcp.config.zone,
    region: gcp.config.region,
  },
)

const cluster = new gcp.container.Cluster(
  name,
  {
    name,
    nodeVersion: '1.15.8-gke.3',
    minMasterVersion: '1.15.8-gke.3',
    masterAuth: {
      username: 'master',
      password: 'masterPassw0rd#$',
    },
    initialNodeCount: 1,
    nodeConfig: {
      labels: {},
      machineType: 'g1-small',
      diskSizeGb: 10,
      preemptible: true,
      oauthScopes: [
        ...nodeDefaultScopes,
      ],
    },
  },
  { provider },
)

// Node Pool

const labels = {}
const nodeCount = 1
const maxNodeCount = 1
const minNodeCount = 1
const nodeType = 'n1-standard-1'
const diskSizeGb = 10
const preemptible = false

const poolName = `${name}-pool-test`

// Here is the problem

const pool = new gcp.container.NodePool(
  poolName,
  {
    cluster: cluster.id,
    name: poolName,
    nodeCount: nodeCount,
    ...(maxNodeCount && {
      autoscaling: {
        maxNodeCount,
        minNodeCount,
      },
    }),
    nodeConfig: {
      labels,
      diskSizeGb,
      preemptible,
      machineType: nodeType,
      oauthScopes: [
        ...nodeDefaultScopes,
      ],
    },
  },
  { provider },
)

Versions

pulumi

v1.10.1

npm

"@pulumi/gcp": "2.6.0",
"@pulumi/kubernetes": "1.5.0",
"@pulumi/pulumi": "1.10.1",
"@pulumi/random": "1.5.0",

gcloud CLI

Google Cloud SDK 279.0.0
beta 2019.05.17
bq 2.0.53
core 2020.01.31
gsutil 4.47

Other

Node Version : 1.15.8-gke.3

The text was updated successfully, but these errors were encountered:

sylver · 2020-02-07T11:34:41Z

After some digging, I think the problem is that the formatted URL is wrong :

/v1beta1/projects/{project_id}/locations/{location}/clusters/projects/{project_id}/locations/{location}/clusters/{cluster_id}/nodePools

but as stated in the doc https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.locations.clusters.nodePools/list URL should be :
/v1beta1/projects/{project_id}/locations/{location}/clusters/{cluster_id}/nodePools

And this one is actually working as expected when I call it directly.

I don't know when/why it changed, and I can't find it in the pulumi source code, maybe from a third party like terraform ?

paultiplady · 2020-02-13T22:58:17Z

Looks like it comes from https://github.com/terraform-providers/terraform-provider-google/blob/31926fa61be523a515ac97cc5dfc0f498b4c3b16/google/resource_container_node_pool.go

paultiplady · 2020-02-13T23:06:15Z

I'm hitting this with a new Pulumi install, using Python and

✗ pulumi version
v1.10.1

✗ pip freeze
Arpeggio==1.9.2
attrs==19.3.0
certifi==2019.11.28
chardet==3.0.4
dill==0.3.1.1
grpcio==1.27.1
idna==2.8
parver==0.2.1
protobuf==3.11.3
pulumi==1.10.1
pulumi-gcp==2.6.0
pulumi-kubernetes==1.5.3
requests==2.21.0
semver==2.9.0
six==1.14.0
urllib3==1.24.3

I can provide more diags if that would be useful.

paultiplady · 2020-02-13T23:26:25Z

@Sylv3r I resolved this in my own deploy -- try this:

const pool = new gcp.container.NodePool(
  poolName,
  {
    cluster: cluster.name,
...

The cluster arg of the NodePort should refer to the cluster name, not sure if cluster.id in your code is returning the same thing.

lukehoban · 2020-02-13T23:43:31Z

@paultiplady Indeed - I was just testing out the same thing as that looked like it might be the cause. It seems as if the beta provider changed the id format from being just the name (like test-cluster) to being a qualified name (like projects/pulumi-development/locations/europe-west1-b/clusters/test-cluster). As far as I can tell, this hasn't change in the last 1+ years, so I'm not sure exactly how this code was working for you previously but is not any more. But it does appear to be by design (though a little unfortunate). Given that, I'm going to close this out - thanks for the help getting to the bottom of this.

sylver · 2020-02-14T07:06:48Z

Thanks @paultiplady and @lukehoban
Fun fact, I just thought during the night to check at the morning the value returned by cluster.id

I'll follow up on the slack thread about this change of returned value.

lukehoban closed this as completed Feb 13, 2020

lukehoban mentioned this issue Feb 14, 2020

MigrateState and SchemaVersion are not honored pulumi/pulumi-terraform-bridge#110

Closed

davidxia mentioned this issue Mar 9, 2020

google-beta v3.10.0 fail to modify gke node pool status hashicorp/terraform-provider-google#5767

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error creating NodePool: googleapi: got HTTP response code 404 #282

error creating NodePool: googleapi: got HTTP response code 404 #282

sylver commented Feb 6, 2020 •

edited

Loading

sylver commented Feb 7, 2020 •

edited

Loading

paultiplady commented Feb 13, 2020

paultiplady commented Feb 13, 2020

paultiplady commented Feb 13, 2020

lukehoban commented Feb 13, 2020

sylver commented Feb 14, 2020

error creating NodePool: googleapi: got HTTP response code 404 #282

error creating NodePool: googleapi: got HTTP response code 404 #282

Comments

sylver commented Feb 6, 2020 • edited Loading

Problem

Code

Versions

pulumi

npm

gcloud CLI

Other

sylver commented Feb 7, 2020 • edited Loading

paultiplady commented Feb 13, 2020

paultiplady commented Feb 13, 2020

paultiplady commented Feb 13, 2020

lukehoban commented Feb 13, 2020

sylver commented Feb 14, 2020

sylver commented Feb 6, 2020 •

edited

Loading

sylver commented Feb 7, 2020 •

edited

Loading