-
Notifications
You must be signed in to change notification settings - Fork 176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CCI][GUIDE] Added Guide for Snapshot API(snapshot.md) #486
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #486 +/- ##
=======================================
Coverage 72.12% 72.12%
=======================================
Files 89 89
Lines 7939 7939
=======================================
Hits 5726 5726
Misses 2213 2213 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will you add a working sample into samples based on this guide? Something that creates, gets, cleans up a snapshot?
CHANGELOG.md
Outdated
@@ -42,6 +42,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) | |||
- Improved CI performance of integration with unreleased OpenSearch ([#318](https://github.com/opensearch-project/opensearch-py/pull/318)) | |||
- Added k-NN guide and samples ([#449](https://github.com/opensearch-project/opensearch-py/pull/449)) | |||
- Added the ability to run tests matching a pattern to `.ci/run-tests` ([#454](https://github.com/opensearch-project/opensearch-py/pull/454)) | |||
- Added new guide: `snapshot.md` for Snapshot API. ([#486](https://github.com/opensearch-project/opensearch-py/pull/429)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Say "Added a guide for taking snapshots"
guides/snapshot.md
Outdated
"location": "/path/to/repo", | ||
} | ||
} | ||
client.snapshot.create_repository(repository='my_repository', body=repo_body) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the response from this? Should print something out as a result.
samples/snapshot/snapshot_sample.py
Outdated
repo_body = { | ||
"type": "fs", # Replace 'fs' with the appropriate repository type | ||
"settings": { | ||
"location": "/path/to/repo", # Replace with the desired repository location |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not really sure if I need to keep this as a location example for it to be replaced by the client or if is there a way to retrieve the default location for a file system repository.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would tempfile.TemporaryDirectory
work for the sample against a locally running / docker OpenSearch?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For windows may be use cd opensearch-py/samples
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I apologize for the wrong screenshot. Yes, I have tried it within samples directory. I get the following error:
$ poetry run hello/hello.py
"hello" is not an internal or external
command, executable program, or batch file.
(base)
And I am running this on my main branch, which is up-to-date with my remote origin. Could you please let me know if there are any pre-steps I need to take besides the ones on ./samples/README.md?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dblock Could you please take a look at this issue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
relevant doc. https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#shared-file-system opensearch.yml file will be found in opensearch docker container files. https://opensearch.org/docs/latest/install-and-configure/configuration/#configuration-file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am trying to run snapshot
volume following the steps specified here: https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/,
but the poetry run python snapshot/snapshot_sample
fails with the following error:
$ poetry run python snapshot/snapshot_sample.py
Traceback (most recent call last):
File "C:\Users\HP\Desktop\AmazonOpenSearchService\Opensearch-py-ml\opensearch-py\samples\snapshot\snapshot_sample.py", line 39, in <module>
response = client.snapshot.create_repository(repository = repository_name, body = repo_body)
File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\client\utils.py", line 179, in _wrapped
return func(*args, params=params, headers=headers, **kwargs)
File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\client\snapshot.py", line 185, in create_repository
return self.transport.perform_request(
File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\transport.py", line 409, in perform_request
raise e
File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\transport.py", line 370, in perform_request
status, headers_response, data = connection.perform_request(
File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\connection\http_urllib3.py", line 266, in perform_request
self._raise_error(
File "C:\Users\HP\AppData\Local\pypoetry\Cache\virtualenvs\package-yvpjSqWs-py3.9\lib\site-packages\opensearchpy\connection\base.py", line 301, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
opensearchpy.exceptions.TransportError: TransportError(500, 'repository_verification_exception', 'failed to create blob container')
I believe this is because docker is denying the write permission.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know the details of how this feature works, but I imagine that the docker container will need something like --env path.repo=/tmp
. It would be totally fine to document that to run this sample you need to create a docker container in a certain way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I nitpick? :)
We have a 2 variables that aren't reused temp_repo
and temp_repo_location
in more than 1 place, and 2 names for the same thing: temp_repo
, temp_repo_location
.
Is a 1-liner cleaner?
repo_body = {
...
"location": tempfile.TemporaryDirectory().name
}
Ignore me if you don't agree ;)
@roma2023 Want to finish this? |
Sure, but I got stuck with this issue: How can I run the docker with the OpenSearch server write permission? Here is more about the details: #486 (comment). Would appreciate your guidance) |
@roma2023 Do you need more help? Let's finish this one? |
I am trying to use shared file system as a snapshot in
It seems to me that the OpenSearch process does not have read and write permissions to I do not know how to solve this issue. If someone could pick this up from here that would be amazing. |
Yes, you will need to chmod it inside the container. Here's what I did.
Code ran: $ poetry run python snapshot/snapshot_sample.py
/usr/share/opensearch/backups
{'acknowledged': True}
{'accepted': True}
{'snapshots': [{'snapshot': 'my_snapshot', 'uuid': 'Q4_hA4jATWmsb6ex38qRsQ', 'version_id': 136327827, 'version': '2.11.0', 'remote_store_index_shallow_copy': False, 'indices': ['test-snapshot'], 'data_streams': [], 'include_global_state': True, 'state': 'IN_PROGRESS', 'start_time': '2023-11-21T20:28:44.288Z', 'start_time_in_millis': 1700598524288, 'end_time': '1970-01-01T00:00:00.000Z', 'end_time_in_millis': 0, 'duration_in_millis': 0, 'failures': [], 'shards': {'total': 0, 'failed': 0, 'successful': 0}}]} LMK if this doesn't unblock you. |
@roma2023 want to finish this? |
@dblock Yes, I will finish this soon. |
Apologize for the delay |
cf0faba
to
fd1c9df
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rebase, update nit?
USER_GUIDE.md
Outdated
@@ -153,6 +153,7 @@ print(response) | |||
- [Search](guides/search.md) | |||
- [Point in Time](guides/point_in_time.md) | |||
- [Using a Proxy](guides/proxy.md) | |||
- [Taking a Snapshot](guides/snapshot.md) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This doc is more than taking, let's say "Working with Snapshots" or "Snapshot Actions" to match what's in the doc?
a88044f
to
3f635b2
Compare
Signed-off-by: Raman Saparkhan <romasaparhan19@gmail.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…#588) Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
* Prepare for next developer iteration, 2.4.1. Signed-off-by: dblock <dblock@amazon.com> * Fix: sync opensearchpy without iohttp. Signed-off-by: dblock <dblock@amazon.com> * Use nox to run tests. Signed-off-by: dblock <dblock@amazon.com> --------- Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
* Fix: TypeError on calling parallel_bulk. Signed-off-by: dblock <dblock@amazon.com> * Added a sample that uses a bulk function generator. Signed-off-by: dblock <dblock@amazon.com> --------- Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…rch-project#603) Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…roject#604) * Fix invalid value type. Signed-off-by: dblock <dblock@amazon.com> * Workaround Incompatible types in assignment (expression has type float, variable has type Double) [assignment] Signed-off-by: dblock <dblock@amazon.com> --------- Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
* Added pylint. Signed-off-by: dblock <dblock@amazon.com> * Enforce pylint:invalid-name. Signed-off-by: dblock <dblock@amazon.com> * Updated the generated code header to prevent broken links. Signed-off-by: dblock <dblock@amazon.com> * Swapped order of messages. Signed-off-by: dblock <dblock@amazon.com> --------- Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…ensearch-project#579) * Added a guide & sample for a custom logger client implementation. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Black formatter Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> * Changes from PR review Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Fixed import formatting in sample code for gudie. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Fixed nox formatting of log collection sample module. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Added types to log_collection_sample.py Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Added type ignore to StramHandler class Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Added formatting change Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> * Added PR review changes. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Fixed typo in CHANGELOG. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Requested changes. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Requested changes again. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Added link in USER_GUIDE.md. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> --------- Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…earch-project#614) * Add GitHub action for opensearch-py release Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Generate GitHub release at the end Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Update CHANGELOG Signed-off-by: Zelin Hao <zelinhao@amazon.com> * Update CHANGELOG Signed-off-by: Zelin Hao <zelinhao@amazon.com> --------- Signed-off-by: Zelin Hao <zelinhao@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
* remove unnecessary utf-8 header in .py files Signed-off-by: samuel orji <awesomeorji@gmail.com> * review feedback: add link to changelog Signed-off-by: samuel orji <awesomeorji@gmail.com> --------- Signed-off-by: samuel orji <awesomeorji@gmail.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…pensearch-project#615) (opensearch-project#617) * License tools: remove utf-8 coding declaration from license_headers.py check/fix script; since opensearch-project#615 it is no longer used in the library's codebase UTF-8 is the default encoding used to read source code files for Python3 - see https://docs.python.org/3/howto/unicode.html#unicode-literals-in-python-source-code Signed-off-by: James Addison <james@reciperadar.com> * Cleanup: remove utf-8 coding declaration from the license_headers.py script itself Signed-off-by: James Addison <james@reciperadar.com> * Update CHANGELOG.md Signed-off-by: James Addison <james@reciperadar.com> --------- Signed-off-by: James Addison <james@reciperadar.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
* Add OpenSearch 2.11.1 integration tests. Signed-off-by: dblock <dblock@amazon.com> * Exclude flaky integration tests with OpenSearch 2.0.1. Signed-off-by: dblock <dblock@amazon.com> --------- Signed-off-by: dblock <dblock@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…oject#616) Updated CHANGELOG.md. nox formatting applied. Added new unit test for actions scan function. Added type hints & nox formatting. Added fix to async scan function & added matching unit tests for async. Signed-off-by: Djcarrillo6 <djcarrillo6@yahoo.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…ch-project#625) * Update pytest-asyncio requirement from <=0.21.1 to <=0.23.2 Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version. - [Release notes](https://github.com/pytest-dev/pytest-asyncio/releases) - [Commits](pytest-dev/pytest-asyncio@v0.1.1...v0.23.2) --- updated-dependencies: - dependency-name: pytest-asyncio dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> * Update changelog Signed-off-by: dependabot[bot] <support@github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
Signed-off-by: saimedhi <saimedhi@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…pensearch-project#640) * updated files with docstrings to pass pylint Signed-off-by: Mark Cohen <markcoh@amazon.com> * updated samples to prepare for enabling missing-docstring linter; will continue to work on this before committing setup.cfg Signed-off-by: Mark Cohen <markcoh@amazon.com> * removed missing-function-docstring from setup.cfg so the linter doesn't fail while work on docstrings continues Signed-off-by: Mark Cohen <markcoh@amazon.com> * corrected unnecessary return docstring values Signed-off-by: Mark Cohen <markcoh@amazon.com> * fixing failure in 'black' on reformatting Signed-off-by: Mark Cohen <markcoh@amazon.com> --------- Signed-off-by: Mark Cohen <markcoh@amazon.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…il.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…il.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
…il.com> Signed-off-by: roma2023 <romasaparhan19@gmail.com>
Signed-off-by: roma2023 <romasaparhan19@gmail.com>
Signed-off-by: roma2023 <romasaparhan19@gmail.com>
4df482b
to
b22a37b
Compare
Signed-off-by: Raman Saparkhan <109868980+roma2023@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. I think we need to explain how to use Dockerfile that you created.
- Move samples/Dockerfile into samples/snapshot.
- Add a "Working Sample" section to the guide with something like "See .../snapshot for a working sample. To run this sample, use the associated Dockerfile as follows." and include some bash output of how one would start the docker container and run the sample.
Add something to guides/snapshot.md
Hello @roma2023, could you please take care of the failing DCO and make the requested changes? Thank you for your patience. |
Description
Added Snapshot Actions Guide (guides/snapshot.md) with code examples
Issues Resolved
issue #406 resolved
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.