[Backport 1.x] Add support for Point In Time APIs (#405) (#422)

* Add `CreatePit` & `DeletePit` methods Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add `DeleteAllPits` and `GetAllPits` methods Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add PIT search support Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add integration tests Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add docs Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add changelog entry Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Correct Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Fix changelog typo Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add tests for CreatePit Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add tests for DeletePit Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Fixing tests Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Cleanup Signed-off-by: Thomas Farr <tsfarr@amazon.com> * Add tests for GetAllPits & DeleteAllPits Signed-off-by: Thomas Farr <tsfarr@amazon.com> * PitSearch tests Signed-off-by: Thomas Farr <tsfarr@amazon.com> * call isolated Signed-off-by: Thomas Farr <tsfarr@amazon.com> * writable cluster Signed-off-by: Thomas Farr <tsfarr@amazon.com> --------- Signed-off-by: Thomas Farr <tsfarr@amazon.com> (cherry picked from commit 5111427)
opensearch-project · Nov 13, 2023 · 1fce876 · 1fce876
1 parent c4b846c
commit 1fce876
Show file tree

Hide file tree

Showing 48 changed files with 2,312 additions and 21 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,9 @@
 Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased]
+### Added
+- Added support for point-in-time search and associated APIs ([#405](https://github.com/opensearch-project/opensearch-net/pull/405))
+
 ### Dependencies
 - Bumps `FSharp.Data` from 6.2.0 to 6.3.0
 - Bumps `BenchMarkDotNet` from 0.13.7 to 0.13.10
@@ -98,4 +101,4 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 [Unreleased]: https://github.com/opensearch-project/opensearch-net/compare/v1.5.0...1.x
 [1.5.0]: https://github.com/opensearch-project/opensearch-net/compare/v1.4.0...v1.5.0
 [1.4.0]: https://github.com/opensearch-project/opensearch-net/compare/v1.3.0...v1.4.0
-[1.3.0]: https://github.com/opensearch-project/opensearch-net/compare/v1.2.0...v1.3.0
+[1.3.0]: https://github.com/opensearch-project/opensearch-net/compare/v1.2.0...v1.3.0
diff --git a/guides/search.md b/guides/search.md
@@ -0,0 +1,181 @@
+# Search
+OpenSearch provides a powerful search API that allows you to search for documents in an index. The search API supports a number of parameters that allow you to customize the search operation. In this guide, we will explore the search API and its parameters.
+
+## Setup
+Let's start by creating an index and adding some documents to it:
+
+```csharp
+using OpenSearch.Client;
+using OpenSearch.Net;
+
+var node = new Uri("https://localhost:9200");
+var config = new ConnectionSettings(node)
+    .ThrowExceptions()
+    .ServerCertificateValidationCallback(CertificateValidations.AllowAll)
+    .BasicAuthentication("admin", "admin");
+var client = new OpenSearchClient(config);
+
+class Movie
+{
+    public string Title { get; set; }
+    public string Director { get; set; }
+    public int Year { get; set; }
+    public override string ToString()
+    {
+        return $"{nameof(Title)}: {Title}, {nameof(Director)}: {Director}, {nameof(Year)}: {Year}";
+    }
+}
+
+await client.Indices.CreateAsync("movies", c => c
+    .Settings(s => s
+        .NumberOfShards(1)
+        .NumberOfReplicas(0)
+    )
+    .Map<Movie>(m => m
+        .Properties(p => p
+            .Text(t => t
+                .Name(o => o.Title))
+            .Text(t => t
+                .Name(o => o.Director))
+            .Number(n => n
+                .Name(o => o.Year)
+                .Type(NumberType.Integer)))));
+
+var movies = new List<Movie>
+{
+    new Movie { Title = "The Godfather", Director = "Francis Ford Coppola", Year = 1972 },
+    new Movie { Title = "The Shawshank Redemption", Director = "Frank Darabont", Year = 1994 },
+};
+
+for (var i = 0; i < 10; ++i)
+    movies.Add(new Movie { Title = "The Dark Knight " + i, Director = "Christopher Nolan", Year = 2008 + i });
+
+// Index the movies
+var indexResponse = await client.BulkAsync(b => b
+    .Index("movies")
+    .IndexMany(movies)
+    .Refresh(Refresh.WaitFor));
+Console.WriteLine($"Indexed {indexResponse.Items.Count} movies"); // Indexed 12 movies
+```
+
+
+## Search API
+
+### Basic Search
+The search API allows you to search for documents in an index. The following example searches for ALL documents in the `movies` index:
+
+```csharp
+var searchResponse = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(q => q.MatchAll()));
+Console.WriteLine(string.Join('\n', searchResponse.Documents));
+```
+
+You can also search for documents that match a specific query. The following example searches for documents that match the query `dark knight`:
+```csharp
+var searchResponse = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(q => q
+        .Match(m => m
+            .Field(f => f.Title)
+            .Query("dark knight"))));
+Console.WriteLine(string.Join('\n', searchResponse.Documents));
+```
+
+OpenSearch query DSL allows you to specify complex queries. Check out the [OpenSearch query DSL documentation](https://opensearch.org/docs/latest/query-dsl/) for more information.
+
+
+### Basic Pagination
+The search API allows you to paginate through the search results. The following example searches for documents that match the query `dark knight`, sorted by `year` in in ascending order, and returns the first 2 results after skipping the first 5 results:
+
+```csharp
+var query = Query<Movie>.Match(m => m
+    .Field(f => f.Title)
+    .Query("dark knight"));
+var sort = new SortDescriptor<Movie>()
+    .Ascending(f => f.Year);
+var searchResponse = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .From(5)
+    .Size(2));
+Console.WriteLine(string.Join('\n', searchResponse.Documents));
+```
+
+With sorting, you can also use the `SearchAfter` parameter to paginate through the search results. Let's say you have already displayed the first page of results, and you want to display the next page. You can use the `SearchAfter` parameter to paginate through the search results. The following example will demonstrate how to get the first 3 pages of results using the search query of the previous example:
+
+```csharp
+var page1 = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2));
+var page2 = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2)
+    .SearchAfter(page1.Hits.Last().Sorts));
+var page3 = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2)
+    .SearchAfter(page2.Hits.Last().Sorts));
+Console.WriteLine(string.Join('\n', page3.Documents));
+```
+
+### Pagination with Scroll API
+When retrieving large amounts of non-real-time data, you can use the `scroll` parameter to paginate through the search results:
+
+```csharp
+var page1 = await client.SearchAsync<Movie>(s => s
+    .Index("movies")
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2)
+    .Scroll("1m"));
+var page2 = await client.ScrollAsync<Movie>("1m", page1.ScrollId);
+var page3 = await client.ScrollAsync<Movie>("1m", page2.ScrollId);
+Console.WriteLine(string.Join('\n', page3.Documents));
+```
+
+### Pagination with Point in Time
+The scroll example above has one weakness: if the index is updated while you are scrolling through the results, they will be paginated inconsistently. To avoid this, you should use the "Point in Time" feature. The following example demonstrates how to use the `point_in_time` and `pit_id` parameters to paginate through the search results:
+
+```csharp
+var pitResp = await client.CreatePitAsync("movies", p => p.KeepAlive("1m"));
+
+var page1 = await client.SearchAsync<Movie>(s => s
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2)
+    .PointInTime(p => p.Id(pitResp.PitId).KeepAlive("1m")));
+var page2 = await client.SearchAsync<Movie>(s => s
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2)
+    .PointInTime(p => p.Id(pitResp.PitId).KeepAlive("1m"))
+    .SearchAfter(page1.Hits.Last().Sorts));
+var page3 = await client.SearchAsync<Movie>(s => s
+    .Query(_ => query)
+    .Sort(_ => sort)
+    .Size(2)
+    .PointInTime(p => p.Id(pitResp.PitId).KeepAlive("1m"))
+    .SearchAfter(page2.Hits.Last().Sorts));
+
+foreach (var doc in page1.Documents.Concat(page2.Documents).Concat(page3.Documents))
+{
+    Console.WriteLine(doc.Title);
+}
+
+await client.DeletePitAsync(p => p.PitId(pitResp.PitId));
+```
+
+Note that a point-in-time is associated with an index or a set of index. So, when performing a search with a point-in-time, you DO NOT specify the index in the search.
+
+## Cleanup
+```csharp
+await client.Indices.DeleteAsync("movies");
+```
diff --git a/src/OpenSearch.Client/Aggregations/Bucket/Composite/CompositeAggregationSource.cs b/src/OpenSearch.Client/Aggregations/Bucket/Composite/CompositeAggregationSource.cs
@@ -30,6 +30,7 @@
 using System.Collections.Generic;
 using System.Linq.Expressions;
 using System.Runtime.Serialization;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 using OpenSearch.Net.Utf8Json.Resolvers;

diff --git a/src/OpenSearch.Client/Cat/CatHelpResponseBuilder.cs b/src/OpenSearch.Client/Cat/CatHelpResponseBuilder.cs
@@ -31,6 +31,7 @@
 using System.Threading;
 using System.Threading.Tasks;
 using OpenSearch.Net;
+using OpenSearch.Net.Extensions;
 
 namespace OpenSearch.Client
 {

diff --git a/src/OpenSearch.Client/CommonAbstractions/Extensions/Extensions.cs b/src/OpenSearch.Client/CommonAbstractions/Extensions/Extensions.cs
@@ -114,13 +114,6 @@ internal static string ToEnumValue<T>(this T enumValue) where T : struct
 			return null;
 		}
 
-		internal static string Utf8String(this ref ArraySegment<byte> segment) =>
-			StringEncoding.UTF8.GetString(segment.Array, segment.Offset, segment.Count);
-
-		internal static string Utf8String(this byte[] bytes) => bytes == null ? null : Encoding.UTF8.GetString(bytes, 0, bytes.Length);
-
-		internal static byte[] Utf8Bytes(this string s) => s.IsNullOrEmpty() ? null : Encoding.UTF8.GetBytes(s);
-
 		internal static bool IsNullOrEmpty(this IndexName value) => value == null || value.GetHashCode() == 0;
 
 		internal static bool IsNullable(this Type type) =>

diff --git a/src/OpenSearch.Client/CommonOptions/Sorting/SortFormatter.cs b/src/OpenSearch.Client/CommonOptions/Sorting/SortFormatter.cs
@@ -27,6 +27,7 @@
 */
 
 using System.Collections.Generic;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 using OpenSearch.Net.Utf8Json.Resolvers;

diff --git a/src/OpenSearch.Client/Indices/MappingManagement/GetFieldMapping/FieldMappingFormatter.cs b/src/OpenSearch.Client/Indices/MappingManagement/GetFieldMapping/FieldMappingFormatter.cs
@@ -27,6 +27,7 @@
 */
 
 using System.Collections.Generic;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 using OpenSearch.Net.Utf8Json.Resolvers;

diff --git a/...t/Modules/SnapshotAndRestore/Repositories/GetRepository/GetRepositoryResponseFormatter.cs b/...t/Modules/SnapshotAndRestore/Repositories/GetRepository/GetRepositoryResponseFormatter.cs
@@ -29,6 +29,7 @@
 using System;
 using System.Collections.Generic;
 using OpenSearch.Net;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Resolvers;
 

diff --git a/src/OpenSearch.Client/OpenSearch.Client.csproj b/src/OpenSearch.Client/OpenSearch.Client.csproj
@@ -19,6 +19,7 @@
     <ProjectReference Include="..\OpenSearch.Net\OpenSearch.Net.csproj" />
   </ItemGroup>
   <ItemGroup>
+    <InternalsVisibleTo Include="Tests" Key="$(ExposedPublicKey)" />
     <InternalsVisibleTo Include="OpenSearch.Net.CustomDynamicObjectResolver" Key="$(ExposedPublicKey)" />
     <InternalsVisibleTo Include="OpenSearch.Net.DynamicCompositeResolver" Key="$(ExposedPublicKey)" />
     <InternalsVisibleTo Include="OpenSearch.Net.DynamicObjectResolverAllowPrivateFalseExcludeNullFalseNameMutateOriginal" Key="$(ExposedPublicKey)" />

diff --git a/...OpenSearch.Client/QueryDsl/Compound/FunctionScore/Functions/ScoreFunctionJsonFormatter.cs b/...OpenSearch.Client/QueryDsl/Compound/FunctionScore/Functions/ScoreFunctionJsonFormatter.cs
@@ -28,6 +28,7 @@
 
 using System;
 using OpenSearch.Net;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/QueryDsl/Geo/BoundingBox/GeoBoundingBoxQuery.cs b/src/OpenSearch.Client/QueryDsl/Geo/BoundingBox/GeoBoundingBoxQuery.cs
@@ -27,6 +27,7 @@
 */
 
 using System;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/QueryDsl/Geo/Distance/GeoDistanceQuery.cs b/src/OpenSearch.Client/QueryDsl/Geo/Distance/GeoDistanceQuery.cs
@@ -26,6 +26,7 @@
 *  under the License.
 */
 
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/QueryDsl/Geo/Polygon/GeoPolygonQuery.cs b/src/OpenSearch.Client/QueryDsl/Geo/Polygon/GeoPolygonQuery.cs
@@ -27,6 +27,7 @@
 */
 
 using System.Collections.Generic;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/QueryDsl/Geo/Shape/GeoShapeQueryFormatter.cs b/src/OpenSearch.Client/QueryDsl/Geo/Shape/GeoShapeQueryFormatter.cs
@@ -27,6 +27,7 @@
 */
 
 using System;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/QueryDsl/Specialized/Shape/CartesianPoint.cs b/src/OpenSearch.Client/QueryDsl/Specialized/Shape/CartesianPoint.cs
@@ -32,6 +32,7 @@
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 using OpenSearch.Client;
+using OpenSearch.Net.Extensions;
 
 namespace OpenSearch.Client
 {

diff --git a/src/OpenSearch.Client/QueryDsl/Specialized/Shape/ShapeQueryFormatter.cs b/src/OpenSearch.Client/QueryDsl/Specialized/Shape/ShapeQueryFormatter.cs
@@ -27,6 +27,7 @@
 */
 
 using System;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/QueryDsl/TermLevel/Terms/TermsQueryFormatter.cs b/src/OpenSearch.Client/QueryDsl/TermLevel/Terms/TermsQueryFormatter.cs
@@ -27,6 +27,7 @@
 */
 
 using System.Collections.Generic;
+using OpenSearch.Net.Extensions;
 using OpenSearch.Net.Utf8Json;
 using OpenSearch.Net.Utf8Json.Internal;
 

diff --git a/src/OpenSearch.Client/Search/PointInTime/CreatePitRequest.cs b/src/OpenSearch.Client/Search/PointInTime/CreatePitRequest.cs
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: Apache-2.0
+*
+* The OpenSearch Contributors require contributions made to
+* this file be licensed under the Apache-2.0 license or a
+* compatible open source license.
+*/
+
+namespace OpenSearch.Client;
+
+[MapsApi("create_pit")]
+[ReadAs(typeof(CreatePitRequest))]
+public partial interface ICreatePitRequest { }
+
+public partial class CreatePitRequest { }
+
+public partial class CreatePitDescriptor { }
diff --git a/src/OpenSearch.Client/Search/PointInTime/CreatePitResponse.cs b/src/OpenSearch.Client/Search/PointInTime/CreatePitResponse.cs
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: Apache-2.0
+*
+* The OpenSearch Contributors require contributions made to
+* this file be licensed under the Apache-2.0 license or a
+* compatible open source license.
+*/
+
+using System.Runtime.Serialization;
+
+namespace OpenSearch.Client;
+
+[DataContract]
+public class CreatePitResponse : ResponseBase
+{
+	[DataMember(Name = "pit_id")]
+	public string PitId { get; internal set; }
+
+	[DataMember(Name = "_shards")]
+	public ShardStatistics Shards { get; internal set; }
+
+	[DataMember(Name = "creation_time")]
+	public long CreationTime { get; internal set; }
+}
diff --git a/src/OpenSearch.Client/Search/PointInTime/DeleteAllPitsRequest.cs b/src/OpenSearch.Client/Search/PointInTime/DeleteAllPitsRequest.cs
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: Apache-2.0
+*
+* The OpenSearch Contributors require contributions made to
+* this file be licensed under the Apache-2.0 license or a
+* compatible open source license.
+*/
+
+namespace OpenSearch.Client;
+
+[MapsApi("delete_all_pits")]
+[ReadAs(typeof(DeleteAllPitsRequest))]
+public partial interface IDeleteAllPitsRequest { }
+
+public partial class DeleteAllPitsRequest { }
+
+public partial class DeleteAllPitsDescriptor { }