OpenSearch DSL #7068

jslay-excella · 2022-05-17T17:49:58Z

Changes

django-elasticsearch-dsl -> django-opensearch-dsl, and all of the other underlying ES -> OS libraries.
Updates cfgov to have compatibility with django-opensearch-dsl
- Moves auto_refresh from class Django to class Index
- Updates rebuild management command calls in scripts and cfgov python.
Added opensearch and opensearch-dashboards charts.

How to test this PR

Screenshots

Notes and todos

Checklist

PR has an informative and human-readable title
- PR titles are used to generate the change log in releases; good ones make that easier to scan.
- Consider prefixing, e.g., "Mega Menu: fix layout bug", or "Docs: Update Docker installation instructions".
Changes are limited to a single goal (no scope creep)
Code follows the standards laid out in the CFPB development guidelines
Future todos are captured in comments and/or tickets
Project documentation has been updated, potentially one or more of:
- This repo’s docs (edit the files in the /docs folder) – for basic, close-to-the-code docs on working with this repo
- CFGOV/platform wiki on GHE – for internal CFPB developer guidance
- CFPB/hubcap wiki on GHE – for internal CFPB design and content guidance

Front-end testing

Browser testing

Visually tested in the following supported browsers:

Accessibility

Keyboard friendly (navigable with tab, space, enter, arrow keys, etc.)
Screen reader friendly
Does not introduce new errors or warnings in WAVE

Other

Is useable without CSS
Is useable without JS
Does not introduce new lint warnings
Flexible from small to large screens

jslay-excella · 2022-05-23T12:03:04Z

OpenSearch v1/v2

v2 releases today. I have tag 2.0.0-rc1 set on GHA for the moment until there is a proper 2.0.0/2 tag. I figure by the time we get to merging this in and migrating to OpenSearch, v2 will be out for a little bit.

Security

OpenSearch really does not like to run without security or SSL. So instead of fighting it, I have embraced it, and worked it in to settings. Dashboards serves on HTTP still, and the default username/password for the UI and the cluster, is admin/admin.

Elasticsearch Helper/Call Command/2 commands vs 1

I have most of OpenSearch working now.

The helper search.elasticsearch_helpers.ElasticsearchTestsMixin.rebuild_elasticsearch_index seems to still be broken/flaky.

django-opensearch-dsl, breaks out rebuilding indices from documents, unlike django-elasticearch-dsl, which does it all in the single command (why I wasn't getting any docs in OpenSearch before, cause i was just doing half the commands). I have tried updating the aforementioned method, and it "seems" to work (doesn't call errors, and when passing in an index, doesn't complain anymore); alas the tests fail out with search results not returning.

`Document.[Django/Index].auto_refresh`

The auto_refresh attribute moved from class Django in django-elasticsearch-dsl, to class Index in django-opensearch-dsl. It is not defined for these two Documents in main, but the tests lead me to thinking there is some other intention, or maybe not because of settings.test.OPENSEARCH_DSL_AUTO_REFRESH. I added the field (primarily for testing), but I think it should default to global setting unless overridden on the instance or the document already (ergo, we actually do want auto_refresh), no?
For the following Documents:

[main] [opensearch] v1.documents.FilterablePageDocument.Index.auto_refresh -> [main] [opensearch] v1.tests.test_documents.FilterablePagesDocumentTest.test_auto_refresh_default
[main] [opensearch] ask_cfpb.documents.AnswerPageDocument.Index.auto_refresh -> [main] [opensearch] ask_cfpb.tests.test_documents.AnswerPageDocumentTest.test_auto_refresh_default

Subsequently, I had a go at messing around with this test

GitHub Actions

I got the service swapped out for OpenSearch, and it works fine for cypress test (at least, it passes). However, something fishy is going on with the backend, where it doesn't seem like it can make a connection (need to start outputting some values like ES_HOST, and ES_PORT). Getting instant connection refused, but tox and OpenSearch connect and work fine in local helm/k8s; they error because I just need to figure out that aforementioned mixin.

Locally, here is what I get (ignore the 2 timestamp failures, unrelated and not real failures, something with docker and local dev timezones):

======================================================================
ERROR: test_index_route (v1.tests.models.test_filterable_list_mixins.FilterableRoutesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/models/test_filterable_list_mixins.py", line 143, in test_index_route
    response.context_data["filter_data"]["page_set"][0].title, "Test"
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/django/core/paginator.py", line 189, in __getitem__
    return self.object_list[index]
IndexError: list index out of range

======================================================================
FAIL: test_base_data_properties (data_research.tests.test_models.MortgageModelTests)
Test basic calculation functions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/data_research/tests/test_models.py", line 413, in test_base_data_properties
    self.assertEqual(data_record.epoch, 1472702400000)
AssertionError: 1472688000000 != 1472702400000

======================================================================
FAIL: test_base_data_properties_time_series (data_research.tests.test_models.MortgageModelTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/data_research/tests/test_models.py", line 427, in test_base_data_properties_time_series
    self.assertEqual(
AssertionError: Lists differ: [0.0, 1472688000000] != [0.0, 1472702400000]

First differing element 1:
1472688000000
1472702400000

- [0.0, 1472688000000]
?           ^^^

+ [0.0, 1472702400000]
?           ^ ++


======================================================================
FAIL: test_cache_gets_called_when_visiting_filterable_page (v1.tests.jinja2tags.test_fragment_cache.TestFragmentCacheExtension)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/jinja2tags/test_fragment_cache.py", line 41, in test_cache_gets_called_when_visiting_filterable_page
    self.assertTrue(add_to_cache.called)
AssertionError: False is not true

======================================================================
FAIL: test_archive_default_opts (v1.tests.management.commands.test_archive_pages.InactiveUsersTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/freezegun/api.py", line 789, in wrapper
    result = func(*args, **kwargs)
  File "/src/consumerfinance.gov/cfgov/v1/tests/management/commands/test_archive_pages.py", line 82, in test_archive_default_opts
    self.assertTrue(self.page1.archived)
AssertionError: False is not true

======================================================================
FAIL: test_archive_last_publish_date (v1.tests.management.commands.test_archive_pages.InactiveUsersTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/freezegun/api.py", line 789, in wrapper
    result = func(*args, **kwargs)
  File "/src/consumerfinance.gov/cfgov/v1/tests/management/commands/test_archive_pages.py", line 124, in test_archive_last_publish_date
    self.assertTrue(self.page2.archived)
AssertionError: False is not true

======================================================================
FAIL: test_archive_months_days (v1.tests.management.commands.test_archive_pages.InactiveUsersTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/freezegun/api.py", line 789, in wrapper
    result = func(*args, **kwargs)
  File "/src/consumerfinance.gov/cfgov/v1/tests/management/commands/test_archive_pages.py", line 103, in test_archive_months_days
    self.assertTrue(self.page1.archived)
AssertionError: False is not true

======================================================================
FAIL: test_restore_default_opts (v1.tests.management.commands.test_restore_archive_pages.TestRestoreArchivePages)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/freezegun/api.py", line 789, in wrapper
    result = func(*args, **kwargs)
  File "/src/consumerfinance.gov/cfgov/v1/tests/management/commands/test_restore_archive_pages.py", line 85, in test_restore_default_opts
    self.assertFalse(self.page1.archived)
AssertionError: True is not false

======================================================================
FAIL: test_restore_last_publish_date (v1.tests.management.commands.test_restore_archive_pages.TestRestoreArchivePages)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/freezegun/api.py", line 789, in wrapper
    result = func(*args, **kwargs)
  File "/src/consumerfinance.gov/cfgov/v1/tests/management/commands/test_restore_archive_pages.py", line 127, in test_restore_last_publish_date
    self.assertFalse(self.page2.archived)
AssertionError: True is not false

======================================================================
FAIL: test_restore_months_days (v1.tests.management.commands.test_restore_archive_pages.TestRestoreArchivePages)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/.tox/unittest/lib/python3.8/site-packages/freezegun/api.py", line 789, in wrapper
    result = func(*args, **kwargs)
  File "/src/consumerfinance.gov/cfgov/v1/tests/management/commands/test_restore_archive_pages.py", line 106, in test_restore_months_days
    self.assertFalse(self.page1.archived)
AssertionError: True is not false

======================================================================
FAIL: test_page_matches_categories (v1.tests.models.test_browse_filterable_page.TestNewsroomLandingPage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/models/test_browse_filterable_page.py", line 79, in test_page_matches_categories
    self.assertTrue(result.exists())
AssertionError: False is not true

======================================================================
FAIL: test_get_context_has_archived_posts (v1.tests.models.test_filterable_list_mixins.FilterableListContextTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/models/test_filterable_list_mixins.py", line 111, in test_get_context_has_archived_posts
    self.assertTrue(context["has_archived_posts"])
AssertionError: False is not true

======================================================================
FAIL: test_get_filterable_children_pages (v1.tests.models.test_filterable_list_mixins.FilterableListRelationsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/models/test_filterable_list_mixins.py", line 192, in test_get_filterable_children_pages
    self.assertEqual(qs.count(), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_search_blog_dates (v1.tests.test_documents.FilterablePagesDocumentSearchTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_documents.py", line 239, in test_search_blog_dates
    self.assertTrue(results.filter(title=self.blog.title).exists())
AssertionError: False is not true

======================================================================
FAIL: test_search_enforcement_actions (v1.tests.test_documents.FilterablePagesDocumentSearchTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_documents.py", line 259, in test_search_enforcement_actions
    self.assertTrue(results.filter(title=self.enforcement.title).exists())
AssertionError: False is not true

======================================================================
FAIL: test_search_enforcement_actions_no_statuses (v1.tests.test_documents.FilterablePagesDocumentSearchTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_documents.py", line 279, in test_search_enforcement_actions_no_statuses
    self.assertTrue(results.filter(title=self.enforcement.title).exists())
AssertionError: False is not true

======================================================================
FAIL: test_search_event_all_fields (v1.tests.test_documents.FilterablePagesDocumentSearchTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_documents.py", line 221, in test_search_event_all_fields
    self.assertTrue(results.filter(title=self.event.title).exists())
AssertionError: False is not true

======================================================================
FAIL: test_search_title_uses_multimatch (v1.tests.test_documents.FilterablePagesDocumentSearchTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_documents.py", line 292, in test_search_title_uses_multimatch
    self.assertTrue(results.filter(title=self.blog_title_match).exists())
AssertionError: False is not true

======================================================================
FAIL: test_index_reflects_page_moves_and_deletions (v1.tests.test_documents.TestThatWagtailPageSignalsUpdateIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_documents.py", line 327, in test_index_reflects_page_moves_and_deletions
    self.assertEqual(results.count(), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_enforcement_action_elasticsearch (v1.tests.test_filterable_list_form.TestEnforcementActionsFilterForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 274, in test_enforcement_action_elasticsearch
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_event_archive_elasticsearch (v1.tests.test_filterable_list_form.TestEventArchiveFilterForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 248, in test_event_archive_elasticsearch
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_filter_by_category (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 77, in test_filter_by_category
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_filter_by_language (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 114, in test_filter_by_language
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_filter_by_tags (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 88, in test_filter_by_tags
    self.assertEqual(len(page_set_pks), 2)
AssertionError: 0 != 2

======================================================================
FAIL: test_filter_by_title (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 120, in test_filter_by_title
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_filter_doesnt_return_drafts (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 98, in test_filter_doesnt_return_drafts
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_filterable_categories_sets_initial_category_list (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 174, in test_filterable_categories_sets_initial_category_list
    self.assertEqual(len(page_set), 1)
AssertionError: 0 != 1

======================================================================
FAIL: test_first_page_date (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 179, in test_first_page_date
    self.assertEqual(form.first_page_date(), self.blog1.date_published)
AssertionError: datetime.date(2010, 1, 1) != datetime.date(2022, 5, 23)

======================================================================
FAIL: test_form_language_choices (v1.tests.test_filterable_list_form.TestFilterableListForm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 103, in test_form_language_choices
    self.assertEqual(
AssertionError: Lists differ: [] != [('en', 'English'), ('es', 'Spanish')]

Second list contains 2 additional elements.
First extra element 0:
('en', 'English')

- []
+ [('en', 'English'), ('es', 'Spanish')]

======================================================================
FAIL: test_filter_by_archived_exclude (v1.tests.test_filterable_list_form.TestFilterableListFormArchive)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 215, in test_filter_by_archived_exclude
    self.assertEqual(len(pages), 2)
AssertionError: 0 != 2

======================================================================
FAIL: test_filter_by_archived_include (v1.tests.test_filterable_list_form.TestFilterableListFormArchive)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 211, in test_filter_by_archived_include
    self.assertEqual(len(pages), 3)
AssertionError: 0 != 3

======================================================================
FAIL: test_filter_by_archived_only (v1.tests.test_filterable_list_form.TestFilterableListFormArchive)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/src/consumerfinance.gov/cfgov/v1/tests/test_filterable_list_form.py", line 220, in test_filter_by_archived_only
    self.assertEqual(len(pages), 1)
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 1879 tests in 377.397s

FAILED (failures=31, errors=1)

jslay-excella · 2022-08-24T16:36:52Z

Closing in favor of #7192

jslay-excella and others added 2 commits May 17, 2022 11:48

Initial OpenSearch swapout

899e6fa

Initial OpenSearch swapout

4e4c5db

jslay-excella force-pushed the opensearch branch from 4df604d to 4e4c5db Compare May 17, 2022 17:54

jslay-excella and others added 5 commits May 22, 2022 23:32

Fixed opensearch document index, swapped out kibana

0821d24

Document update vs index

278b405

models -> indices

29e18ca

Fixed some more tests

f876960

Merge branch 'main' into opensearch

5e856ea

jslay-excella force-pushed the opensearch branch from d5f9cf5 to f7fdfee Compare May 23, 2022 10:39

Swapout for OpenSearch

d5f7d61

jslay-excella force-pushed the opensearch branch from f7fdfee to d5f7d61 Compare May 23, 2022 10:40

jslay-excella added 5 commits May 23, 2022 04:45

Removed debug prints

6466100

Testing hard coding port

5cbc35a

Testing hard coding host

d0b2e87

Testing matching functional-tests.yml

af4cee4

Testing matching functional-tests.yml

7b52b20

jslay-excella added 2 commits May 25, 2022 11:54

Added elasticsearch back, but using opensearch-dsl

7826d05

Use elasticsearch for backend for until ready to go to opensearch

7e66228

jslay-excella changed the title ~~OpenSearch~~ OpenSearch DSL May 25, 2022

jslay-excella added 5 commits May 25, 2022 12:13

Add ES_SCHEMA

e5bbf98

Merge main

978622b

Default back to http

cdfd689

Add ES back to init container

84a6409

Add --refresh to document command

dddd0f6

jslay-excella force-pushed the opensearch branch from 613a47e to dddd0f6 Compare May 26, 2022 05:24

jslay-excella added the Back-end label Jul 25, 2022

jslay-excella closed this Aug 24, 2022

wpears deleted the opensearch branch November 1, 2022 20:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenSearch DSL #7068

OpenSearch DSL #7068

jslay-excella commented May 17, 2022 •

edited

Loading

jslay-excella commented May 23, 2022 •

edited

Loading

jslay-excella commented Aug 24, 2022

OpenSearch DSL #7068

OpenSearch DSL #7068

Conversation

jslay-excella commented May 17, 2022 • edited Loading

Changes

How to test this PR

Screenshots

Notes and todos

Checklist

Front-end testing

Browser testing

Accessibility

Other

jslay-excella commented May 23, 2022 • edited Loading

OpenSearch v1/v2

Security

Elasticsearch Helper/Call Command/2 commands vs 1

Document.[Django/Index].auto_refresh

GitHub Actions

jslay-excella commented Aug 24, 2022

jslay-excella commented May 17, 2022 •

edited

Loading

jslay-excella commented May 23, 2022 •

edited

Loading

`Document.[Django/Index].auto_refresh`