Error while enabling GSuite integration #245

marco-lancini · 2020-02-05T15:33:53Z

Issue: The documentation for setting up the GSuite integration is incomplete and leads towards a 400 error

Description:
Following the README, I:

Enabled Google API access (instructions)
Created a new G Suite user account and accept the Terms of Service (this account will be used as the domain-wide delegated access).
Performed G Suite Domain-Wide Delegation of Authority, as explained here
Downloaded the service account's credentials
Setup env vars for cartography

GSUITE_GOOGLE_APPLICATION_CREDENTIALS - location of the credentials file.
GSUITE_DELEGATED_ADMIN - email address that you created in step 2

After this, Cartography crashes with the following output:

cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-env-var NEO4J_PASSWORD_ENV_VAR
INFO:cartography.sync:Starting sync with update tag '1580915718'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'aws'
...
INFO:cartography.sync:Finishing sync stage 'aws'
INFO:cartography.sync:Starting sync stage 'gcp'
...
INFO:cartography.sync:Finishing sync stage 'gcp'
INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json returned "Invalid Input">
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 11, in <module>
    load_entry_point('cartography', 'console_scripts', 'cartography')()
  File "/app/cartography/cli.py", line 240, in main
    return CLI(default_sync, prog='cartography').main(argv)
  File "/app/cartography/cli.py", line 220, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/app/cartography/sync.py", line 135, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json returned "Invalid Input">

I started troubleshooting this and I found the string my_customer is hardcoded in cartography/intel/gsuite/api.py:

request = admin.users().list(customer='my_customer', maxResults=500, orderBy='email')

I decided to replace it with the customerId of my GSuite org, and then I faced a 403 - Not Authorized error:

cartography --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password-env-var NEO4J_PASSWORD_ENV_VAR
INFO:cartography.sync:Starting sync with update tag '1580916464'
INFO:cartography.sync:Starting sync stage 'create-indexes'
INFO:cartography.intel.create_indexes:Creating indexes for cartography node types.
INFO:cartography.sync:Finishing sync stage 'create-indexes'
INFO:cartography.sync:Starting sync stage 'aws'
...
INFO:cartography.sync:Finishing sync stage 'aws'
INFO:cartography.sync:Starting sync stage 'gcp'
...
INFO:cartography.sync:Finishing sync stage 'gcp'
INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "forbidden"
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=<REDACTED>&maxResults=500&orderBy=email&alt=json returned "Not Authorized to access this resource/api">
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 11, in <module>
    load_entry_point('cartography', 'console_scripts', 'cartography')()
  File "/app/cartography/cli.py", line 240, in main
    return CLI(default_sync, prog='cartography').main(argv)
  File "/app/cartography/cli.py", line 220, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/app/cartography/sync.py", line 135, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 79, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 230, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 109, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 856, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?customer=<REDACTED>&maxResults=500&orderBy=email&alt=json returned "Not Authorized to access this resource/api">

I tried to add more scopes to the service account (listed below), but I still get Not Authorized:

Please complete the following information::

Cartography release version or commit hash: 801bb58
Python version: 3.7.5
OS: python:3.7 docker image

The text was updated successfully, but these errors were encountered:

achantavy · 2020-02-05T18:30:13Z

@skiptomyliu Can you take a look at this GSuite issue when you get a moment?

skiptomyliu · 2020-02-05T19:25:19Z

the my_customer refers to the current customer/org that the application belongs to. https://developers.google.com/admin-sdk/directory/v1/guides/manage-customers , but replacing it with your customerId should also work.

I would still double check that the steps have been completed on your step 3: ("G Suite Domain-Wide Delegation of Authority").

You should have two accounts:

A service account that is created in GCP
An e-mail account that is created in GSuite that the service account you created in GCP will delegate to.

marco-lancini · 2020-02-07T10:09:15Z

Hi @skiptomyliu, thanks for replying.

I've made sure to have:

A service account in GCP
An email account in GSuite for delegation

We also tried applying these scopes:

The error we are facing now is the following:

INFO:cartography.sync:Starting sync stage 'gsuite'
INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/discovery/v1/apis/admin/directory_v1/rest
DEBUG:cartography.intel.gsuite.api:Syncing GSuite Users
INFO:googleapiclient.discovery:URL being requested: GET https://www.googleapis.com/admin/directory/v1/users?customer=my_customer&maxResults=500&orderBy=email&alt=json
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
INFO:oauth2client.client:Failed to retrieve access token: {
  "error": "unauthorized_client",
  "error_description": "Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested."
}
ERROR:cartography.sync:Unhandled exception during sync stage 'gsuite'
Traceback (most recent call last):
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 68, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 229, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 108, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 851, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 159, in new_request
    credentials._refresh(orig_request_method)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/client.py", line 749, in _refresh
    self._do_refresh_request(http)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/client.py", line 819, in _do_refresh_request
    raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
oauth2client.client.HttpAccessTokenRefreshError: unauthorized_client: Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested.
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 11, in <module>
    load_entry_point('cartography', 'console_scripts', 'cartography')()
  File "/app/cartography/cli.py", line 241, in main
    return CLI(default_sync, prog='cartography').main(argv)
  File "/app/cartography/cli.py", line 221, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/app/cartography/sync.py", line 135, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/app/cartography/sync.py", line 69, in run
    stage_func(neo4j_session, config)
  File "/app/cartography/intel/gsuite/__init__.py", line 68, in start_gsuite_ingestion
    api.sync_gsuite_users(session, resources.admin, config.update_tag, common_job_parameters)
  File "/app/cartography/intel/gsuite/api.py", line 229, in sync_gsuite_users
    resp_objs = get_all_users(admin)
  File "/app/cartography/intel/gsuite/api.py", line 108, in get_all_users
    resp = request.execute(num_retries=GOOGLE_API_NUM_RETRIES)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 851, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 159, in new_request
    credentials._refresh(orig_request_method)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/client.py", line 749, in _refresh
    self._do_refresh_request(http)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/client.py", line 819, in _do_refresh_request
    raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
oauth2client.client.HttpAccessTokenRefreshError: unauthorized_client: Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested.

skiptomyliu · 2020-02-07T17:01:00Z

Hrmm, I wonder if we mixed up the steps as it appears ordering may be required...

https://stackoverflow.com/a/59067603/914941

"Delegating domain-wide authority to the service account" MUST be enabled before you add service account and its scopes on "Manage API client access" page in G Suite Admin. Otherwise it will fail with "Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested." error and require removing the API client and adding it again.

marco-lancini · 2020-02-13T13:50:12Z

Hi @skiptomyliu,
just to be super sure we tried both ways and we are still facing the same issue

stale · 2020-02-27T14:18:31Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

marco-lancini · 2020-02-27T14:28:57Z

We are still blocked by this issue

stale · 2020-03-12T15:25:05Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

marco-lancini · 2020-03-12T15:45:26Z

Hi stale bot, this is still an ongoing issue :)

marco-lancini · 2020-03-26T14:35:09Z

Hi, some updates on this:

After some troubleshooting I managed to successfully setup domain wide delegation, and I can confirm it is working by running rbacsync (which requires an almost identical configuration).
Now, if I perform the same exact steps to setup a service account for Cartography, I end up having get the same error (client unauthorized).
As an additional test, I took the service account created for Cartography and gave it to rbacsync. I can confirm rbacsync accepts the service account and works fine.

So it really seems this issue is related to some calls made by Cartography, rather than to a faulty setup?

zamirTo1 · 2020-09-12T23:11:00Z

Hi,
Is anyone managed to overcome HTTP error 403 "Not Authorized to access this resource/api"?

marco-lancini · 2020-09-13T12:56:09Z

Unfortunately no, I had to put on hold the GSuite integration for my use cases 😟

achantavy · 2020-09-15T05:38:02Z

Ah man, I wonder what secret sauce we've got going on because this just works™️ on our deployment.

@zamirTo1 do you get the same error messages as @marco-lancini? Can you add more details?

zamirTo1 · 2020-09-15T07:03:06Z

Hi @achantavy
I do get the same error as @marco-lancini, I've also tried to build the GSuite section by myself according to this guide:
https://developers.google.com/admin-sdk/directory/v1/guides/delegation
same result :(

marco-lancini · 2020-09-15T08:03:35Z

Yeah can confirm I followed that process as well, and got it working for RBACSync.
This leads me to think it might be something related to Cartography code itself 🤔

marco-lancini · 2020-09-16T09:54:06Z

I've documented the approach I took, step-by-step, here: https://www.marcolancini.it/2020/blog-gsuite-domain-delegation/
Hope this helps!

jychp · 2023-01-06T08:29:33Z

This #1071 does not solve the issue but allow to use an other auth method (OAuth) that can be considered as a workaround.

marco-lancini changed the title ~~GSuite integration~~ Error while enabling GSuite integration Feb 5, 2020

stale bot added the stale stalebot believes this issue/PR is no longer active label Feb 27, 2020

stale bot removed the stale stalebot believes this issue/PR is no longer active label Feb 27, 2020

stale bot added the stale stalebot believes this issue/PR is no longer active label Mar 12, 2020

stale bot removed the stale stalebot believes this issue/PR is no longer active label Mar 12, 2020

achantavy added the keep fresh Disables stalebot from closing an issue label Mar 17, 2020

achantavy added the bug Something isn't working label Oct 14, 2020

chandanchowdhury mentioned this issue Jun 30, 2024

GCP access token expires for services used less frequently #1323

Open

chandanchowdhury added the GCP GCP related issues and PRs label Jun 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error while enabling GSuite integration #245

Error while enabling GSuite integration #245

marco-lancini commented Feb 5, 2020

achantavy commented Feb 5, 2020

skiptomyliu commented Feb 5, 2020

marco-lancini commented Feb 7, 2020

skiptomyliu commented Feb 7, 2020 •

edited

Loading

marco-lancini commented Feb 13, 2020

stale bot commented Feb 27, 2020

marco-lancini commented Feb 27, 2020

stale bot commented Mar 12, 2020

marco-lancini commented Mar 12, 2020

marco-lancini commented Mar 26, 2020

zamirTo1 commented Sep 12, 2020

marco-lancini commented Sep 13, 2020

achantavy commented Sep 15, 2020

zamirTo1 commented Sep 15, 2020

marco-lancini commented Sep 15, 2020

marco-lancini commented Sep 16, 2020

jychp commented Jan 6, 2023

Error while enabling GSuite integration #245

Error while enabling GSuite integration #245

Comments

marco-lancini commented Feb 5, 2020

achantavy commented Feb 5, 2020

skiptomyliu commented Feb 5, 2020

marco-lancini commented Feb 7, 2020

skiptomyliu commented Feb 7, 2020 • edited Loading

marco-lancini commented Feb 13, 2020

stale bot commented Feb 27, 2020

marco-lancini commented Feb 27, 2020

stale bot commented Mar 12, 2020

marco-lancini commented Mar 12, 2020

marco-lancini commented Mar 26, 2020

zamirTo1 commented Sep 12, 2020

marco-lancini commented Sep 13, 2020

achantavy commented Sep 15, 2020

zamirTo1 commented Sep 15, 2020

marco-lancini commented Sep 15, 2020

marco-lancini commented Sep 16, 2020

jychp commented Jan 6, 2023

skiptomyliu commented Feb 7, 2020 •

edited

Loading