[Meta] Remove mapping types

[reta]

Is your feature request related to a problem? Please describe.
The mapping types have been deprecated in Elasticsearch (starting from 7.0) [1] with breaking changes to the index creation, put mapping, get mapping, put template, get template and get field mappings APIs and on track to be removed in 8.0 [2]. This is the debt Opensearch has inherited and the code related to mapping types support has to be removed.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
[2] elastic/elasticsearch#41059

Describe the solution you'd like
Remove mapping types support from the core and APIs.

Describe alternatives you've considered

1. Keep the code around (nonetheless it is dead)
2. Resurrect mapping types support (what is the case for it?)

Additional context
[Removal of mapping types] (https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html)

OpenSearch Engine Subtasks

• [Remove] Type from Search Internals #2109
• [Remove] Types from DocWrite Request and Response #2239
• [Remove] Type from Client.prepare(Index,Delete,Update) #2253
• [Remove] remaining type usage in Client and AbstractClient #2258
• [Remove] deprecated getMapping API from IndicesClient #2262
• [Remove] types from PutMappingRequest #2335
• Handling multiple mapping type index templates #2356
• Remove type from nested documents; where types used to store nested path information #2357
• Type removal from API end-points
  • RestIndexAction Remove type mapping from document index API #2026
  • RestUpdateAction [Remove] Type mapping parameter from document update API #2204
  • RestDeleteAction [Remove] Types from DocWrite Request and Response #2239
  • RestGetAction [Remove] Types from GET/MGET #2168
  • RestMultiGetAction [Remove] Types from GET/MGET #2168
  • RestExplainAction [Remove] types from SearchHit and Explain API #2205
  • RestBulkAction [Remove] type support from Bulk API #2215
  • RestNoopBulkAction Remove type end-points from NOOP bulk and search action #2261
  • RestSearchAction Remove type end-points from search and related APIs #2263 and [Remove] Types from GET/MGET #2168
  • RestMultiSearchAction Remove type end-points from search and related APIs #2263
  • RestSearchTemplateAction Remove type end-points from search and related APIs #2263
  • RestMultiSearchTemplateAction [Remove] Type mapping end-points from RestMultiSearchTemplateAction #2433 & Remove type end-points from search and related APIs #2263
  • RestNoopSearchAction Remove type end-points from NOOP bulk and search action #2261
  • RestCountAction Remove type end-points from count action #2379
  • RestUpdateByQueryAction Remove type end-points from search and related APIs #2263
  • RestDeleteByQueryAction Remove type end-points from search and related APIs #2263
  • RestGetFieldMappingAction Removes types from mapping APIs #2238
  • RestGetMappingAction Removes types from mapping APIs #2238
  • RestPutMappingAction Removes types from mapping APIs #2238
  • RestValidateQueryAction Remove type from validate query API #2255
  • RestGetSourceAction [Remove] Types from GET/MGET #2168
  • RestTermVectorsAction [Remove] Type parameter from TermVectors API #2104 and Remove type end-points from search and related APIs #2263
  • RestMultiTermVectorsAction Remove type end-points from search and related APIs #2263
• Remove include_type_name param from API end-points #2195
• Add new tests to verify bwc scenarios #2240
• Documentation #2267
• Rest layer bwc compatibility post type removal #2296

Plugins:

• alerting-dashboards-plugin Remove mapping types alerting-dashboards-plugin#181
• alerting Remove mapping types alerting#317
• anomaly-detection-dashboards-plugin: Remove mapping types anomaly-detection-dashboards-plugin#184
• anomaly-detection Remove mapping types anomaly-detection#394
• asynchronous-search Remove mapping types asynchronous-search#98
• common-utils Remove mapping types common-utils#123
• dashboards-reports Remove mapping types reporting#292
• dashboards-visualizations Remove mapping types dashboards-visualizations#52
• index-management-dashboards-plugin Remove mapping types index-management-dashboards-plugin#155
• index-management Remove mapping types index-management#282
• job-scheduler Remove mapping types job-scheduler#131
• K-NN Remove mapping types #155 k-NN#303
• notifications Remove mapping types notifications#375
• performance-analyzer Remove mapping types performance-analyzer#145
• performance-analyzer-rca Remove mapping types performance-analyzer-rca#134
• security-dashboards-plugin Remove mapping types security-dashboards-plugin#914
• security Remove mapping types security#1658
• sql Remove mapping types sql#452
• observability Remove mapping types observability#523
• cross-cluster-replication Remove mapping types cross-cluster-replication#318
• OpenSearch-Dashboards [BWC] Verify impact for type and include_type_name no longer being supported OpenSearch-Dashboards#1290

Clients

• data-prepper Remove mapping types data-prepper#1139
• logstash-output-opensearch Remove mapping types logstash-output-opensearch#128
• opensearch-clients Remove mapping type support  opensearch-clients#6

Other

• opensearch-benchmark Remove mapping types opensearch-benchmark#150
• opensearch-benchmark-workloads Remove mapping types opensearch-benchmark-workloads#37
• opensearch-cli Remove mapping types opensearch-cli#47
• ml-commons: Remove mapping types ml-commons#158
• opensearch-oci-object-storage Remove mapping types opensearch-oci-object-storage#8
• documentation-website Remove mapping types documentation-website#429
• project-website Remove mapping type project-website#709

[CEHENKLE]

Thanks @reta This seems to make sense to actually remove in 2.0. What do you think?'

Thanks again,
/C

[anasalkouz]

More discussion already here: https://discuss.opendistrocommunity.dev/t/is-opensearch-planning-to-remove-the-mapping-types-too/8341?u=searchymcsearchface

[reta]

@CEHENKLE +1 to include it into 2.0 roadmap, thank you!

[dreamer-89]

Looking into it

[nknize]

We may want to convert this to a meta issue. This is a biggie, especially for bwc testing. If we try to remove everything there's a possibility of it not being completed in time for 2.0. Types is already deprecated so I don't think we need to make it a blocker for 2.0; we should be able to continue during the 2.x line.

[anasalkouz]

We may want to convert this to a meta issue. This is a biggie, especially for bwc testing. If we try to remove everything there's a possibility of it not being completed in time for 2.0. Types is already deprecated so I don't think we need to make it a blocker for 2.0; we should be able to continue during the 2.x line.

@nknize my understanding that removing mapping types is breaking changes and we should target a major a release. Otherwise, we can target any minor release.

[dreamer-89]

Current state:

1. Remove type references from rest end-points [In-Progress]
2. Remove internal references of type mapping e.g. Transport layer, builders, serialized classes, terms lookup etc
3. Update plugin in respective repositories
4. Client Update - Java/Python/Go
5. Testing
   a. Automated - covered by UT & ITs in OpenSearch repo
   b. Manual - Verify various upgrades paths from 1.x -> 2.0
   c. Plugins integration - Build OpenSearch main branch with default plugins; following existing build process. Sample build manifest file
   d. Clients - Java/Python/Go

[Edit]
Had a quick sync up with @nknize @andrross @anasalkouz and we decided to first evaluate impact of chanages on rolling upgrades, index templates, bwc testing. Identify which changes can be delivered in 2.0 without delaying the release.

[dreamer-89]

Proposal for v2.0 release

• Rest end-points updates
  • Document APIs
    • Ingestion, Update, Delete, Get APIs
    • UpdateByQuery, DeleteByQuery
    • Bulk Request. Request and response
    • Term, multi-term vectors
  • Index APIs
    • Create Index
    • Index mapping (put/get)
    • Field mapping (put/get)
    • Source API
  • Search APIs
    • Search, multi Search, explain
    • Query
      • Terms lookup
      • Type Query - Remove
    • Validate API
  • Query DSL
• Search templates
• include_type_name param from APIs
• Backward compatibility changes necessary for upgrade
• Update sdk clients (java/python/go/php/rust etc)
• Update plugins (if needed)
• Testing
  • Automated.
    • Upgrade/bwc tests. Verify older OS versions are compatible with OS 2.0. Sample upgrades to test:
      • Basic. Test basic features
        • Index. Verify indices created with types (or without) allows ingestion in OS 2.0
        • Templates. Verify templates created on OS 1.x with types, allows new index creation.
        • Searching. Verify older indices created with types, are searchable on OS 2.0 without returning type information.
        • Snapshots. Verify older indices are getting snapshots successfully in OS 2.0. It should be possible to recover older index from snapshot in OS 2.0.
    • Integration Tests.
  • Manual.
    • Verify main builds with different plugin repos
    • Verify advance test cases (TBD)
• Documentation

[dblock]

@dreamer-89 What's left for 2.0 here?

[dreamer-89]

@dreamer-89 What's left for 2.0 here?

@dblock : We have completed all changes for 2.0.

We are targeting remaining pending changes post 2.0 release. Created #2979 to capture remaining changes.

[dblock]

Let's close this, then.

We have #2357 and #2979 for 2.1.