Adding [wip] documentation for syncing prod > qa #20
Conversation
|
|
||
| Note the input/output may need to be changed from "opensearch" to "elasticsearch" depending on whether the domain is actually AWS OpenSearch or a classic Elasticsearch domain. The 5.3 domains appear to be usable as "elasticsearch" targets even though AWS calls them opensearch - probably grandfathered access points. | ||
|
|
||
| Note also inclusion of "parallelPublisher" in `remove_field` to tell logstash to remove parallelPublisher when found. This was a deprecated property in favor of parallelPublisherLiteral property in any document it's |
There was a problem hiding this comment.
in any document it's..... ? gah! the suspense is killin me!
There was a problem hiding this comment.
Thanks I'll update. Too many distrac
| ./logstash-plugin install logstash-input-opensearch | ||
| ``` | ||
|
|
||
| The `LS_JAVA_OPTS` bit above invites logstash to use a reasonable amount of memory for the work it needs to do rather than fall over with a `java.lang.OutOfMemoryError` around about 7M records. A proverbial "make yourself at home" to the JVM. |
There was a problem hiding this comment.
hah. is that number just an estimate or from experience?
|
|
||
| Follow progress by checking index doc count via `GET https://[fqdn of qa domain]]/_cat/indices?v` | ||
|
|
||
| **If the pipeline fails**, you may need to kill the process via multiple `CTRL-c`s. Resume from the last indexed `uri`. Determine the last seen `uri` by POSTing the following to `{{ES_BASE_URI}}/{{ES_INDEX_NAME}}/resource/_search` |
There was a problem hiding this comment.
what would indicate that the pipeline has failed?
| } | ||
| ``` | ||
|
|
||
| The single record returned has the last indexed `uri` (which is also the `_id` value). Once obtained, cause `logstash` to begin where you left off by editing the `query` line in your logstash conf as follows: |
There was a problem hiding this comment.
can you move the first sentence of this line above the query body? i gleaned this was the case before reading it, but if you're going to say it, you should probably say it before the body.
|
|
||
| ## Syncing Production Elasticsearch to QA | ||
|
|
||
| The following describes how to copy production down to QA using logstash with no downtime. |
There was a problem hiding this comment.
is logstash a package? how does one get it running?
| Prepare the index using `index-admin prepare` to apply mapping, etc.: | ||
|
|
||
| ``` | ||
| node jobs/index-admin prepare --index resources-[YYYY-MM-DD] --profile nypl-digital-dev --envfile config/qa.env |
There was a problem hiding this comment.
where does jobs/index-admin come from?
There was a problem hiding this comment.
Added clarity. Will put this PR in Draft form until we can simplify this step.
| The pollers are controlled by EventBridge triggers: | ||
| - [SierraBibUpdatePoller-qa](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/SierraBibUpdatePoller-qa?tab=configure) | ||
| - [SierraBibDeleteUpdatePoller-qa](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/SierraBibDeleteUpdatePoller-qa?tab=configure) | ||
| - [SierraItemUpdatePoller-qa](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/SierraItemUpdatePoller-qa?tab=configure) |
There was a problem hiding this comment.
you have the update url for the item delete link
Draft documentation for syncing QA from prod data.