Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add minimal FTS support #3282

Merged
merged 12 commits into from
May 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,14 @@
* Removed `PermissionDeniedException` as it was no longer possible to get it. (Issue [#3272](https://github.com/realm/realm-dotnet/issues/3272))
* Removed some obsolete error codes from the `ErrorCode` enum. All codes removed were obsolete and no longer emitted by the server. (PR [3273](https://github.com/realm/realm-dotnet/issues/3273))
* Removed `IncompatibleSyncedFileException` as it was no longer possible to get it. (Issue [#3167](https://github.com/realm/realm-dotnet/issues/3167))
* The `Realms.Schema.Property` API now use `IndexType` rather than a boolean indicating whether a property is indexed. (Issue [#3281](https://github.com/realm/realm-dotnet/issues/3281))

### Enhancements
* Added nullability annotations to the Realm assembly. Now methods returning reference types are correctly annotated to indicate whether the returned value may or may not be null. (Issue [#3248](https://github.com/realm/realm-dotnet/issues/3248))
* Replacing a value at an index (i.e. `myList[1] = someObj`) will now correctly `CollectionChange` notifications with the `Replace` action. (Issue [#2854](https://github.com/realm/realm-dotnet/issues/2854))
* It is now possible to change the log level at any point of the application's lifetime. (PR [#3277](https://github.com/realm/realm-dotnet/pull/3277))
* Some log messages have been added to the Core database. Events, such as opening a Realm or committing a transaction will now be logged. (Issue [#2910](https://github.com/realm/realm-dotnet/issues/2910))
* Added support for Full-Text search (simple term) queries. (Issue [#3281](https://github.com/realm/realm-dotnet/issues/3281))

### Fixed

Expand Down
1 change: 1 addition & 0 deletions Realm/Realm.Fody/Realm.Fody.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
<NoWarn>$(NoWarn),1591, NU5100, NU5128</NoWarn>
<LangVersion>8.0</LangVersion>
<IncludeSymbols>False</IncludeSymbols>
<DefineConstants>$(DefineConstants);PRIVATE_INDEXTYPE</DefineConstants>
</PropertyGroup>
<ItemGroup Label="References">
<PackageReference Include="FodyHelpers" Version="6.*" PrivateAssets="All" />
Expand Down
4 changes: 2 additions & 2 deletions Realm/Realm.SourceGenerator/ClassCodeBuilder.cs
Original file line number Diff line number Diff line change
Expand Up @@ -196,9 +196,9 @@ private string GeneratePartialClass(string interfaceString, string managedAccess
{
var realmValueType = GetRealmValueType(property.TypeInfo);
var isPrimaryKey = property.IsPrimaryKey.ToCodeString();
var isIndexed = property.IsIndexed.ToCodeString();
var indexType = property.Index.ToCodeString();
var isNullable = property.IsRequired ? "false" : property.TypeInfo.IsNullable.ToCodeString();
schemaProperties.AppendLine(@$"Realms.Schema.Property.Primitive(""{property.GetMappedOrOriginalName()}"", {realmValueType}, isPrimaryKey: {isPrimaryKey}, isIndexed: {isIndexed}, isNullable: {isNullable}, managedName: ""{property.Name}""),");
schemaProperties.AppendLine(@$"Realms.Schema.Property.Primitive(""{property.GetMappedOrOriginalName()}"", {realmValueType}, isPrimaryKey: {isPrimaryKey}, indexType: {indexType}, isNullable: {isNullable}, managedName: ""{property.Name}""),");

// The rules for determining whether to always set the property value are:
// 1. If the property has [Required], always set it - this is only the case for string and byte[] properties.
Expand Down
30 changes: 29 additions & 1 deletion Realm/Realm.SourceGenerator/Diagnostics.cs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ private enum Id
TypeNotSupported = 24,
RealmObjectWithoutAutomaticProperty = 25,
NotPersistedPropertyWithRealmAttributes = 26,
ParentOfNestedClassIsNotPartial = 27
ParentOfNestedClassIsNotPartial = 27,
IndexedPrimaryKey = 28,
}

#region Errors
Expand Down Expand Up @@ -119,6 +120,33 @@ public static Diagnostic IndexedWrongType(string className, string propertyName,
location);
}

public static Diagnostic FullTextIndexedWrongType(string className, string propertyName, string propertyType, Location location)
{
return CreateDiagnosticError(
Id.IndexedWrongType,
"[Indexed(IndexType.FullText)] is only allowed on string properties",
$"{className}.{propertyName} is marked as [Indexed(IndexType.FullText)] which is only allowed on string properties, not on {propertyType}.",
location);
}

public static Diagnostic IndexedModeNone(string className, string propertyName, Location location)
{
return CreateDiagnosticError(
Id.IndexedWrongType,
"[Indexed(IndexType.None)] is not allowed",
$"{className}.{propertyName} is annotated as [Indexed(IndexType.None)] which is not allowed. If you don't wish to index the property, removed the [Indexed] attribute.",
location);
}

public static Diagnostic IndexPrimaryKey(string className, string propertyName, Location location)
{
return CreateDiagnosticError(
Id.IndexedPrimaryKey,
"[Indexed] is not allowed in combination with [PrimaryKey]",
$"{className}.{propertyName} is marked has both [Indexed] and [PrimaryKey] attributes which is not allowed. PrimaryKey properties are indexed by default so the [Indexed] attribute is redundant.",
location);
}

public static Diagnostic RequiredWrongType(string className, string propertyName, string propertyType, Location location)
{
return CreateDiagnosticError(
Expand Down
9 changes: 4 additions & 5 deletions Realm/Realm.SourceGenerator/InfoClasses.cs
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ internal record EnclosingClassInfo(string Name, Accessibility Accessibility);

internal record PropertyInfo(string Name)
{
public bool IsIndexed { get; set; }
public IndexType? Index { get; set; }

public bool IsRequired { get; set; }

Expand Down Expand Up @@ -236,10 +236,9 @@ public bool IsSupportedIndexType()
return _indexableTypes.Contains(ScalarType);
}

public bool IsSupportedPrimaryKeyType()
{
return _primaryKeyTypes.Contains(ScalarType);
}
public bool IsSupportedFullTextType() => ScalarType == ScalarType.String;

public bool IsSupportedPrimaryKeyType() => _primaryKeyTypes.Contains(ScalarType);

public bool IsSupportedRequiredType()
{
Expand Down
18 changes: 15 additions & 3 deletions Realm/Realm.SourceGenerator/Parser.cs
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ private IEnumerable<PropertyInfo> GetProperties(ClassInfo classInfo, IEnumerable
var info = new PropertyInfo(propSymbol.Name)
{
Accessibility = propSymbol.DeclaredAccessibility,
IsIndexed = propSymbol.HasAttribute("IndexedAttribute"),
Index = propSymbol.GetIndexType(),
IsRequired = propSymbol.HasAttribute("RequiredAttribute"),
IsPrimaryKey = propSymbol.HasAttribute("PrimaryKeyAttribute"),
MapTo = (string?)propSymbol.GetAttributeArgument("MapToAttribute"),
Expand Down Expand Up @@ -239,12 +239,24 @@ private IEnumerable<PropertyInfo> GetProperties(ClassInfo classInfo, IEnumerable
{
classInfo.Diagnostics.Add(Diagnostics.PrimaryKeyWrongType(classInfo.Name, info.Name, info.TypeInfo.TypeString, propSyntax.GetLocation()));
}
}

if (info.IsIndexed && !info.TypeInfo.IsSupportedIndexType())
if (info.Index != null)
{
classInfo.Diagnostics.Add(Diagnostics.IndexPrimaryKey(classInfo.Name, info.Name, propSyntax.GetLocation()));
}
}
else if (info.Index == IndexType.General && !info.TypeInfo.IsSupportedIndexType())
{
classInfo.Diagnostics.Add(Diagnostics.IndexedWrongType(classInfo.Name, info.Name, info.TypeInfo.TypeString, propSyntax.GetLocation()));
}
else if (info.Index == IndexType.FullText && !info.TypeInfo.IsSupportedFullTextType())
{
classInfo.Diagnostics.Add(Diagnostics.FullTextIndexedWrongType(classInfo.Name, info.Name, info.TypeInfo.TypeString, propSyntax.GetLocation()));
}
else if (info.Index == IndexType.None)
{
classInfo.Diagnostics.Add(Diagnostics.IndexedModeNone(classInfo.Name, info.Name, propSyntax.GetLocation()));
}

if (info.IsRequired)
{
Expand Down
2 changes: 2 additions & 0 deletions Realm/Realm.SourceGenerator/Realm.SourceGenerator.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
<IncludeSymbols>false</IncludeSymbols>
<Nullable>enable</Nullable>
<EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
<DefineConstants>$(DefineConstants);PRIVATE_INDEXTYPE</DefineConstants>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="7.0.1">
Expand Down Expand Up @@ -60,6 +61,7 @@

<ItemGroup>
<AdditionalFiles Include="$(ProjectDir)..\..\stylecop.json" />
<Compile Include="..\Realm\Attributes\IndexType.cs" Link="IndexType.cs" />
</ItemGroup>

<ItemGroup>
Expand Down
32 changes: 29 additions & 3 deletions Realm/Realm.SourceGenerator/Utils.cs
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,31 @@ public static bool HasAttribute(this ISymbol symbol, string attributeName)
return symbol.GetAttributes().Any(a => a.AttributeClass?.Name == attributeName);
}

public static object? GetAttributeArgument(this ISymbol symbol, string attributeName)
public static object? GetAttributeArgument(this ISymbol symbol, string attributeName, int index = 0)
{
var attribute = symbol.GetAttributes().FirstOrDefault(a => a.AttributeClass?.Name == attributeName);
return attribute?.ConstructorArguments[0].Value;
var arguments = symbol.GetAttributes().FirstOrDefault(a => a.AttributeClass?.Name == attributeName)?.ConstructorArguments;
if (arguments != null && arguments.Value.Length > index)
{
return arguments.Value[index].Value;
}

return null;
}

public static IndexType? GetIndexType(this ISymbol symbol)
{
var attribute = symbol.GetAttributes().FirstOrDefault(a => a.AttributeClass?.Name == "IndexedAttribute");
if (attribute == null)
{
return null;
}

if (attribute.ConstructorArguments.Length == 0)
{
return IndexType.General;
}

return (IndexType)(int)attribute.ConstructorArguments[0].Value!;
}

public static bool IsValidIntegerType(this ITypeSymbol symbol)
Expand Down Expand Up @@ -228,6 +249,11 @@ public static string ToCodeString(this bool boolean)
return boolean.ToString().ToLower();
}

public static string ToCodeString(this IndexType? index)
{
return $"IndexType.{index ?? IndexType.None}";
}

#region Formatting

public static string Indent(this string str, int indents = 1, bool trimNewLines = false)
Expand Down
1 change: 1 addition & 0 deletions Realm/Realm.UnityWeaver/Realm.UnityWeaver.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
<AssemblyName>Realm.UnityWeaver</AssemblyName>
<NoWarn>1701;1702;NU1701</NoWarn>
<LangVersion>8.0</LangVersion>
<DefineConstants>$(DefineConstants);PRIVATE_INDEXTYPE</DefineConstants>
</PropertyGroup>

<ItemGroup>
Expand Down
1 change: 1 addition & 0 deletions Realm/Realm.Weaver/Realm.Weaver.projitems
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,6 @@
</PropertyGroup>
<ItemGroup>
<Compile Include="$(MSBuildThisFileDirectory)**\*.cs" />
<Compile Include="..\Realm\Attributes\IndexType.cs" Link="IndexType.cs" />
</ItemGroup>
</Project>
28 changes: 24 additions & 4 deletions Realm/Realm.Weaver/RealmWeaver.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,13 @@

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Mime;
using System.Threading.Tasks;
using Mono.Cecil;
using Mono.Cecil.Cil;
using Mono.Cecil.Rocks;
using Realms;

namespace RealmWeaver
{
Expand Down Expand Up @@ -531,10 +532,27 @@ private WeavePropertyResult WeaveProperty(PropertyDefinition prop, TypeDefinitio
}

var backingField = prop.GetBackingField();
var isIndexed = prop.CustomAttributes.Any(a => a.AttributeType.Name == "IndexedAttribute");
if (isIndexed && !prop.IsIndexable(_references))
var indexedAttribute = prop.CustomAttributes.FirstOrDefault(a => a.AttributeType.Name == "IndexedAttribute");
if (indexedAttribute != null)
{
return WeavePropertyResult.Error($"{type.Name}.{prop.Name} is marked as [Indexed] which is only allowed on integral types as well as string, bool and DateTimeOffset, not on {prop.PropertyType.FullName}.");
if (!prop.IsIndexable(_references))
{
return WeavePropertyResult.Error($"{type.Name}.{prop.Name} is marked as [Indexed] which is only allowed on integral types as well as string, bool, DateTimeOffset, ObjectId, and Guid not on {prop.PropertyType.FullName}.");
}

if (indexedAttribute.ConstructorArguments.Count > 0)
{
var mode = (IndexType)(int)indexedAttribute.ConstructorArguments[0].Value;
if (mode == IndexType.None)
{
return WeavePropertyResult.Error($"{type.Name}.{prop.Name} is marked as [Indexed(IndexType.None)] which is not allowed. If you don't wish to index the property, remove the IndexedAttribute.");
}

if (mode == IndexType.FullText && prop.PropertyType.FullName != StringTypeName)
{
return WeavePropertyResult.Error($"{type.Name}.{prop.Name} is marked as [Indexed(IndexType.FullText)] which is only allowed on string properties, not on {prop.PropertyType.FullName}.");
}
}
}

var isPrimaryKey = prop.IsPrimaryKey(_references);
Expand Down Expand Up @@ -719,6 +737,8 @@ private WeavePropertyResult WeaveProperty(PropertyDefinition prop, TypeDefinitio
prop.CustomAttributes.Add(wovenPropertyAttribute);

var primaryKeyMsg = isPrimaryKey ? "[PrimaryKey]" : string.Empty;

var isIndexed = indexedAttribute != null;
var indexedMsg = isIndexed ? "[Indexed]" : string.Empty;
_logger.Debug($"Woven {type.Name}.{prop.Name} as a {prop.PropertyType.FullName} {primaryKeyMsg} {indexedMsg}.");
return WeavePropertyResult.Success(prop, backingField, isPrimaryKey, isIndexed);
Expand Down
90 changes: 90 additions & 0 deletions Realm/Realm/Attributes/IndexType.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
////////////////////////////////////////////////////////////////////////////
//
// Copyright 2023 Realm Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
////////////////////////////////////////////////////////////////////////////

namespace Realms
{
#if PRIVATE_INDEXTYPE
internal enum IndexType : int

#else
/// <summary>
/// Describes the indexing mode for properties annotated with the <see cref="IndexedAttribute"/>.
/// </summary>
public enum IndexType : int
#endif
{
/// <summary>
/// Indicates that the property is not indexed.
/// </summary>
None = 0,

/// <summary>
/// Describes a regular index with no special capabilities.
/// </summary>
General = 1,

/// <summary>
/// Describes a Full-Text index on a string property.
/// </summary>
/// <remarks>
/// Only <see cref="string"/> properties can be marked with this attribute.
/// <br/>
/// The full-text index currently support this set of features:
/// <list type="bullet">
/// <item>
/// Only token or word search, e.g. <c>QueryMethods.FullTextSearch(o.Bio, "computer dancing")</c>
/// will find all objects that contains the words <c>computer</c> and <c>dancing</c> in their <c>Bio</c> property.
/// </item>
/// <item>
/// Tokens are diacritics- and case-insensitive, e.g. <c>QueryMethods.FullTextSearch(o.Bio, "cafe dancing")</c>
/// and <c>QueryMethods.FullTextSearch(o.Bio, "café DANCING")</c> will return the same set of matches.
/// </item>
/// <item>
/// Ignoring results with certain tokens are done using `-`, e.g. <c>QueryMethods.FullTextSearch(o.Bio, "computer -dancing")</c>
/// will find all objects that contain <c>computer</c> but not <c>dancing</c> .
/// </item>
/// <item>
/// Tokens only consist of alphanumerical characters from ASCII and the Latin-1 supplement. All other characters
/// are considered whitespace. In particular words using `-` like <c>full-text</c> are split into two tokens.
/// </item>
/// <item>
/// </item>
/// </list>
/// <br/>
/// Note the following constraints before using full-text search:
/// <list type="bullet">
/// <item>
/// Token prefix or suffix search like <c>QueryMethods.FullTextSearch(o.Bio, "comp* *cing")</c> is not supported.
/// </item>
/// <item>
/// Only ASCII and Latin-1 alphanumerical chars are included in the index (most western languages).
/// </item>
/// <item>
/// Only boolean match is supported, i.e. "found" or "not found". It is not possible to sort results by "relevance".
/// </item>
/// </list>
/// </remarks>
/// <example>
/// <code>
/// var cars = realm.All&lt;Car&gt;().Where(c => QueryMethods.FullTextSearch(o.Description, "vintage sport red"));
/// var cars = realm.All&lt;Car&gt;().Filter("Description TEXT $0", "vintage sport red");
/// </code>
/// </example>
FullText = 2,
}
}
Loading