GCP access token expires for services used less frequently

[soumyadipDe]

Title: GCP access token expires for services used less frequently

Description:

Describe your idea. We are running cartography for our big GCP environment. The issue we faced is that GCP token is expiring for services that are used less frequently. The issue was as below

• Cartography uses GCP python library to fetch credentials
• Credetials are service specific i.e. different for compute/storage/dns/container/servicecatalog
• If a specific credential is not used for 1hr, the token gets revoked. If it is in use, it gets refreshed.
• Now resources like GKE or GCS buckets are few and far between projects.
• So interval between two GKE/GCS API call might be more than 1hr for which credentials gets stale and errors out.

To Reproduce:

As said, same service call between projects with gap of more than an hour will cause this.
Logs:
If applicable, copy and paste your console log with the failing stack trace.

ERROR:cartography.sync:Unhandled exception during sync stage 'gcp'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 111, in run
    stage_func(neo4j_session, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 252, in start_gcp_ingestion
    _sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 210, in _sync_multiple_projects
    _sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 181, in _sync_single_project
    storage.sync_gcp_buckets(neo4j_session, resources.storage, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 247, in sync_gcp_buckets
    storage_res = get_gcp_buckets(storage, project_id)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 32, in get_gcp_buckets
    res = req.execute()
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
    headers=self.headers,
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1725, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1444, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1396, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 724, in main
    sys.exit(CLI(prog='cartography').main(argv))
  File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 705, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 189, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 111, in run
    stage_func(neo4j_session, config)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 252, in start_gcp_ingestion
    _sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 210, in _sync_multiple_projects
    _sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 181, in _sync_single_project
    storage.sync_gcp_buckets(neo4j_session, resources.storage, project_id, gcp_update_tag, common_job_parameters)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 247, in sync_gcp_buckets
    storage_res = get_gcp_buckets(storage, project_id)
  File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 32, in get_gcp_buckets
    res = req.execute()
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
    headers=self.headers,
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1725, in request
    conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1444, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1396, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)

Please complete the following information::

• Cartography release version - 0.91
• Python version: 3.10
• OS: debian

[soumyadipDe]

Added suggested PR which worked for my case

[chandanchowdhury]

Hey Saumyadip, thank you for the bug report.

Are you sure the issue is with token expiration?
Asking cause I would expect HTTP 401 (Unauthorized) when token has expired instead of SSLError with KRB5_S_TKT_NYV.

[soumyadipDe]

Yes I did get 401 in another stackthread but did not save if unfortunately. Only when I got 401 I understood the issue and could fix it.

[chandanchowdhury]

Note: Might be related to #245