Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG][NRPTI-1165] token refresh for core importer #1247

Merged
merged 7 commits into from
Jun 27, 2024

Conversation

acatchpole
Copy link
Contributor

@acatchpole acatchpole commented Jun 26, 2024

Description

The API access token granted when doing the core-documents import is only valid for 900 sec (15 minutes). While this is not normally noticeable when only importing a couple documents, an import of all documents can take over an hour to do.

This PR includes the following proposed change(s):

  • Updated the core-documents importer to set an expiration time when the api access token is gotten.
  • Added a check that gets a new token if the expiration time has passed
  • (unrelated) also deleted the migration updateBCMICollectionTypes from a previous PR that has now been deemed unnecessary. This migration was never applied to the production db.

@acatchpole acatchpole self-assigned this Jun 26, 2024
@acatchpole acatchpole linked an issue Jun 26, 2024 that may be closed by this pull request
2 tasks
@@ -66,7 +66,7 @@ Moving the **App** from `dev` (`:latest`) to `test`
```sh
oc project f00029-tools
oc tag nrpti:test nrpti:test-backup
oc tag nrpti:lastest nrpti:test
oc tag nrpti:latest nrpti:test
```
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated typo fix

return payload.access_token;
}
return payload;
};
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed this to return the whole object because it includes the ttl for the token

@@ -43,7 +43,8 @@ class CoreDataSource {

try {
// Get a new API access token.
this.client_token = await getCoreAccessToken(CORE_CLIENT_ID, CORE_CLIENT_SECRET, CORE_GRANT_TYPE);
const apiAccess = await getCoreAccessToken(CORE_CLIENT_ID, CORE_CLIENT_SECRET, CORE_GRANT_TYPE);
this.client_token = apiAccess.access_token;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored due to the change below

@acatchpole acatchpole marked this pull request as ready for review June 26, 2024 21:52
@acatchpole acatchpole requested a review from a team June 26, 2024 21:52
Copy link
Contributor

@LocalNewsTV LocalNewsTV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a lot of leftover console logs in here, not sure if it was intended, but flagging it in case, more specific to ones like "Token updating..." immediately followed by "Token updated!"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't missing migration files cause errors to run during migration steps, if this was already merged and run by the databases at different levels? It might depend on the tool, but I know NRPTI has a migrations table in it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. this migration was never run on prod

*
* @memberof CoreDocumentsDataSource
*/
async setClientToken() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember for a different but similar issue we had to request a longer token expiry time, maybe it was more specific to a users credentials. graceful solution :)

Comment on lines +316 to +318
isAPITokenExpired(expiryTime) {
return Date.now() >= expiryTime;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an edge case, would you want to artificially increase Date.now() to account for time for the API request to be made?

Copy link
Contributor Author

@acatchpole acatchpole Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's a TIME_BUFFER subtracted from the expiryTime. we request a refresh 30 seconds before we actually need it to allow for any delays

await this.taskAuditRecord.updateTaskRecord({ itemTotal: this.status.itemTotal });

const permitUtils = new PermitUtils(this.auth_payload, RECORD_TYPE.Permit);

// Push records to proccess into a Promise array to be processed in parallel.
for (let i = 0; i < permits.length; i += jobCount) {
for (let i = 0; i < numPermits; i += jobCount) {
Copy link
Contributor

@fergmac fergmac Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@L78-L89

This could be a good opportunity to pull this code out into a function and write a test for it. This kind of code can be confusing to read.

Not a blocker, I know we didn't write this code we are just updating the permit length reference.

@acatchpole acatchpole merged commit cf61de1 into master Jun 27, 2024
3 checks passed
@acatchpole acatchpole deleted the 1165-token-refresh-for-core-importer branch June 27, 2024 23:35
Copy link

sggerard pushed a commit that referenced this pull request Jul 10, 2024
* unrelated typo fix

* refactored getCoreAccessToken to return whole data obj instead of just the token

* refactoring to account for a changed return type in supporting function

* added a check for expired api token to the core-documents importer

* fixed typo

* removed migration to update bcmi collections as it is not needed.
sggerard pushed a commit that referenced this pull request Jul 17, 2024
* unrelated typo fix

* refactored getCoreAccessToken to return whole data obj instead of just the token

* refactoring to account for a changed return type in supporting function

* added a check for expired api token to the core-documents importer

* fixed typo

* removed migration to update bcmi collections as it is not needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Token Refresh for CORE Importer
3 participants