Not access data from Google Play

[beshkenadze]

When requesting files from the bucket "pubsite_prod_rev_ххххх" answer is always "Forbidden" (status code 403).
Example code:

var gcloud = require('gcloud');
var storage = gcloud.storage({
    projectId: '1xxxxxxxx',
    keyFilename: './json_api_key.json'
});

var bucket = storage.bucket("pubsite_prod_rev_1xxxxxxxxx");

bucket.getFiles({
    prefix: "reviews"
}, function (err, files, nextQuery, apiResponse) {
    if (nextQuery) {
        // nextQuery will be non-null if there are more results.
        bucket.getFiles(nextQuery, function (err, files, nextQ, apiResponse) {
        });
    }
    if(files == null) { // always Forbidden
        console.log(err.message); 
    }else{
        console.log(files);
    }
});

[jgeewax]

Is that bucket owned by you? If so, are you sure that particular service account (specified in keyFilename) has access to the bucket?

[beshkenadze]

Yep, the bucket and the app owned by one account.
Using gsutil everything works as expected.

[jgeewax]

Hmm. Are you able to list buckets ?

var gcloud = require('gcloud');
var storage = gcloud.storage({
    projectId: '1xxxxxxxx',
    keyFilename: './json_api_key.json'
});

storage.getBuckets(function(err, buckets, next) {
  console.log('Buckets were: ', buckets);
});

[beshkenadze]

The list is empty, the response status of 200.

{
  response: {
    debugId: 1,
    headers: {
      expires: 'Wed, 10 Jun 2015 13:28:33 GMT',
      date: 'Wed, 10 Jun 2015 13:28:33 GMT',
      'cache-control': 'private, max-age=0, must-revalidate, no-transform',
      vary: 'Origin, X-Origin',
      'content-type': 'application/json; charset=UTF-8',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'SAMEORIGIN',
      'x-xss-protection': '1; mode=block',
      'content-length': '31',
      server: 'GSE',
      'alternate-protocol': '443:quic,p=1',
      connection: 'close'
    },
    statusCode: 200,
    body: '{\n "kind": "storage#buckets"\n}\n'
  }
}

btw, APIs Explorer (https://cloud.google.com/storage/docs/json_api/v1/buckets/list) shows the same response, the request (https://cloud.google.com/storage/docs/json_api/v1/objects/list) generates a list of files.

[jgeewax]

This really seems like an access problem for that specific service account... You're 100% certain that the project ID is correct (and owns that bucket)? And that the service account is in that project?

What happens if you run gcloud auth login locally and comment out the keyFilename property? (This tells gcloud-node to pull credentials from whatever the cloud SDK is using.)

[beshkenadze]

After using the command "gcloud auth login" and comment out the keyFilename property, request responded with 200 status and brought the list of files.

It turns out that is not authenticated through a "service_account".

I doubled checked and even created another json key, but still won't work.

[jgeewax]

Gotcha. Do you see the service account in the list on the permissions page? (https://console.developers.google.com/project/_/permissions)

[beshkenadze]

Of course.

[jgeewax]

Err, that should probably look more like:

[jgeewax]

I just ran a quick test to double check this:

1. Create a bucket
2. Upload a file to that bucket
3. Create a new set of credentials (service account, with JSON keyfile)
4. Check whether the new credentials could see the bucket listing

The hope was to see if there was some bug on Google's side where maybe they weren't granting access to the bucket if you created the credentials after the fact.

It looks like this did exactly what it was supposed to do: I was able to see the bucket listing using the service account.

Any other info you can provide to help us track down whether this was a bug or not would be super useful... but I'm out of ideas (outside of simple mistakes like using the wrong .json file).

[beshkenadze]

Oh, I have a new version of the console.
Here is the correct screenshot:

[beshkenadze]

We don't have account in Google Cloud Storage and we can't create a bucket :(
We have another account for a android application.

[ryanseys]

This is weird. Just throwing out a couple ideas:

1. The project id is different than the project number. Typically it starts with a letter, so it's strange yours starts with a number unless you changed it. For example purple-spaceman-123 is a good example of what a default project id looks like.
2. Try going to https://console.developers.google.com/project/{{projectID}}/apiui/credential

You should see a screen like:

Click the "Generate new JSON key" button and set keyFilename to point to that file. The file should contain the line: "type": "service_account".

We don't have account in Google Cloud Storage and we can't create a bucket :(

What do you mean you don't have an account in Google Cloud Storage? There is no such thing as a Google Cloud Storage account. There are Google accounts and Google Service Accounts. With those two things, and the right credentials, you should be all set to use this library.

[ryanseys]

FWIW Cloud Storage did see a blip of 500's yesterday on https://status.cloud.google.com/ but, despite the overall vague-ness of the status report, it seems unrelated to the issue presented here.

[beshkenadze]

I re-generated the key:

{
  "private_key_id": "2xxxxxxxxxxxxxa",
  "private_key": "-----BEGIN PRIVATE KEY-----\nBODY\n-----END PRIVATE KEY-----\n",
  "client_email": "1xxxxxx-xxxxxx@developer.gserviceaccount.com",
  "client_id": "1xxxxxx-xxxxxx.apps.googleusercontent.com",
  "type": "service_account"
}

And nothing has changed.

All developers have a personal bucket (like pubsite_prod_rev_*) in the Cloud Storage, but do not have a full account if not buy it.

[jgeewax]

Is the Cloud Storage API turned on ? https://console.developers.google.com/project/_/apiui/apis/library -> choose your project, then search for "Cloud Storage API"

[ryanseys]

Have you enabled billing or started a free trial? You cannot use Google Cloud, including Storage, if you don't start the trial, have a credit on your account, or add a credit card to your account for billing.

[beshkenadze]

@jgeewax Of course.

[beshkenadze]

@ryanseys Yes, but I can use Google Cloud, if I'm Android Developer.

Details here: https://support.google.com/googleplay/android-developer/answer/138230?hl=en

[jgeewax]

That's interesting. So you didn't create the bucket yourself... In that case, can you look at the permissions on the bucket?

• https://console.developers.google.com/project/_/storage/browser
• See the 3 dots on the far right side next to your bucket, click it and choose Edit bucket permissions

If you didn't create it yourself it's possible that your project doesn't have all the right access... You want to make sure your project is listed as an owner and reader in this screen:

[ryanseys]

Ahhh... You're given restricted access to that bucket, i.e. cannot create files in it and it's owned by Google and shared with you to get your ratings csv etc. Additionally, you have access to Google Cloud Storage but you cannot do anything with that project because its sole purpose is to provide you your ratings. Try to enable billing or start a trial and create a new project with Google Cloud, then you should be free to create as many buckets and files as you wish.

[jgeewax]

That's fair, but still... shouldn't downloads not throw a content mismatch?

[ryanseys]

That is a separate issue that should be addressed separately.

[beshkenadze]

@jgeewax I can't access the browser did not accept a trial.

But I can read and download files through the browser: https://console.developers.google.com/storage/browser/pubsite_prod_rev_*

I don't want to create an cloud storage account, but i want to get the files via gcloud-node, not using gsutil. :)

[jgeewax]

OK wait... The issue is his service account does not have access. We're saying that that is expected because the bucket is shared with a specific Google account, not a project?

[jgeewax]

@beshkenadze : If my previous comment is accurate, then it seems that Android is specifically limiting how you can access that data (that is, Android is saying "we'll put stuff here, and you can access it, and we won't make you pay, but that means you don't get all the features of Cloud Storage")

[beshkenadze]

@jgeewax If I don't have access to the bucket, then why login using gsutil or gcloud (console) authorization works?
I mean, I can't even login through json_key, but works through the console authorization.

[beshkenadze]

Well, I can login via the console and save the token.
I guess we can close the issue.

[jgeewax]

That's exactly my point. I think Android is sharing the bucket with you (ie, you@gmail.com) -- not your project.

So you'd need a way to tell Android to share either with your entire project (described above) or with your specific service account (in the e-mail address field for your service account, ie 606734090113-sjdpv266j56ffhlugjhuatb2nhfan8j9@developer.gserviceaccount.com).

If you want a service account to have access (which, again, I'm not even sure how you'd do that), you likely can't do this for free. I'm asking around to see what I can find, but I don't think this is going to result in the answer you want :(

[jgeewax]

@beshkenadze : Any chance you can try adding your service account as a user in the Play Devconsole? That is, in the JSON keyfile you'll see an e-mail address... Can you try adding that e-mail to the list of users authorized to your Play account?

[beshkenadze]

@jgeewax Interesting idea, will definitely try!

[beshkenadze]

I'm afraid to jinx it, but seems it works.
Why I didn't think about that? :)

[jgeewax]

Cool -- Glad to hear it. This means your SA will have the same read-only access that you have. You still won't be able to configure notifications, but you can write your gcloud-node scripts in the normal way :)

[beshkenadze]

Yep 👍

[jgeewax]

Are you sure that you're not accidentally using the wrong credentials? The Play folks mentioned they're only syncing credentials once per day, so it's unlikely that the service account would have access yet.

[beshkenadze]

Folder with configs is empty, nowhere to take the keys:

akira@akira-mbp ~/P/M/R/src> ls -l ~/.config/
total 0
drwxr-xr-x  5 akira  staff  170 Aug 10  2014 Transmission Remote GUI
drwx------  7 akira  staff  238 May 26 13:11 configstore
drwx------  6 akira  staff  204 Jul 20 16:30 fish
drwxr-xr-x  3 akira  staff  102 Jan  9  2015 robomongo

[jgeewax]

Double check that your gcloud-node code (var gcloud = require('gcloud')({ ... })) explicitly uses the service account key file? (I just want to be 100% certain that it's working the right way so we can add it to the documentation ! :)

[beshkenadze]

Without the key file:

With the key file:

Now, I will remove the account from Google Play and check it again.

[jgeewax]

Ha -- maybe they are .... misinformed about how often permissions get sync'd ? :-/ I'll ask around.. Thanks @beshkenadze !

[beshkenadze]

If anything, I can give access to the personal account on GP during the test.

[beshkenadze]

Just give me the email of the service account and I will add it to itself with read access.

[hugoduraes]

Hi @beshkenadze,

I'm getting the same issues as you got before. I'm trying to authenticate with gcloud auth login and use the generated json file on gcloud-node. In this case, what projectId have you used? Should I create a new project for this?

Thanks in advance.

[beshkenadze]

@hugoduraes hey! You need to use the project_id for which you created the service account.

[hugoduraes]

@beshkenadze So, I create a service account, add its email address to the google play developer account with permissions to view reports, generate a JSON keyfile and use it to access the API? Is this correct?

[beshkenadze]

@hugoduraes Sounds right :)

[hugoduraes]

@beshkenadze Was able to fetch files for a specific bucket. Are you able to get all buckets for the service account?

[beshkenadze]

@hugoduraes I used a particular bucket, it was 'reviews'.

[hugoduraes]

Ok.
Thank you very much for your help. 👍