Add nanosecond field mapper #37755

spinscale · 2019-01-23T10:20:58Z

This adds a dedicated field mapper that supports nanosecond resolution -
at the price of a reduced date range.

When using the date field mapper, the time is stored as milliseconds since the epoch
in a long in lucene. This field mapper stores the time in nanoseconds
since the epoch - which means its range is much smaller, ranging roughly from
1677 to 2262.

If you are searching across indices where a date field is in nanoseconds
and another date field in milliseconds, then an internal mechanism will
convert the nanoseconds to milliseconds losing precision and then merge
the search results.

Relates #27330
Closes #32601

elasticmachine · 2019-01-23T10:21:00Z

Pinging @elastic/es-search

spinscale · 2019-01-23T12:43:54Z

@elasticmachine run elasticsearch-ci/1

jimczi

Thanks @spinscale , I did a first pass and added some comments.

jimczi · 2019-01-23T12:57:50Z

docs/reference/mapping/types/nanosecond.asciidoc

+This means that it will accept dates with optional timestamps, which conform
+to the formats supported by
+<<strict-date-time,`strict_date_optional_time_nanos`>> or
+nanoseconds-since-the-epoch.


Is it nanoseconds-since-the-epoch or epoch_millis ? I don't understand which one is used by default.

jimczi · 2019-01-23T12:58:08Z

docs/reference/mapping/types/nanosecond.asciidoc

+<1> The `date` field uses the default `format`.
+<2> This document uses a plain date.
+<3> This document includes a time.
+<4> This document uses milliseconds-since-the-epoch.


Why, shouldn't this use the nanoseconds-since-the-epoch ?

jimczi · 2019-01-23T13:00:08Z

docs/reference/mapping/types/nanosecond.asciidoc

+<2> This document uses a plain date.
+<3> This document includes a time.
+<4> This document uses milliseconds-since-the-epoch.
+<5> Note that the `sort` values that are returned are all in milliseconds-since-the-epoch.


It should be in nanoseconds since epoch if there is a single index (the field is declared as a nanosecond date) ?

jimczi · 2019-01-23T13:07:27Z

server/src/main/java/org/elasticsearch/index/mapper/NanosecondDateFieldMapper.java

+import static org.elasticsearch.common.time.DateUtils.toLong;
+
+/** A {@link FieldMapper} for dates. */
+public class NanosecondDateFieldMapper extends FieldMapper {


I wonder if the nanosecond type could be included in DateFieldMapper directly. The code is almost the same, only the parsing changes so I don't think we need a separate class ?

valid, looking if that works out!

i have not worked on the other items but just on this one. Maybe you can take a look if my change makes sense. I created an enum, that has different methods for converting a string to a long for milliseconds and nanoseconds. Does that make sense to you?

++, it looks good

jimczi · 2019-01-23T13:10:46Z

server/src/test/java/org/elasticsearch/search/sort/NanosecondIT.java

+import static org.hamcrest.Matchers.arrayWithSize;
+import static org.hamcrest.Matchers.is;
+
+public class NanosecondIT extends ESIntegTestCase {


Let's add rest tests rather than ESIntegTestCase. These tests shouldn't bee too difficult to migrate to yaml ?

jpountz

Woohoo, great to see that all the work that has been done wrt java-time support now allows to support nanosecond timestamps with so little additional code. Some comments/questions:

I'm not a fan of the nanosecond name because it doesn't tell whether this is a date/time or a duration. I'd rather like something like timestamp, timestamp_nanos or even date_nanos.
I left a note about some computations that are wrong with negative timestamps (ie. before 1970). While we could fix it, I wonder that a better option might be to reject nanosecond timestamps before 1970 at index time? I can't think of a use-case that requires nanosecond precision before 1970.
I agree with Jim it's a bit confusing that this new field supports nanoseconds but treats numbers as millis at index time.
Does this field work with date_histogram aggregations already?

jpountz · 2019-01-24T08:29:24Z

docs/reference/mapping/types/nanosecond.asciidoc

+same mapping parameters as the `date` formatter can be used.
+
+NOTE: You can query across indices that have a date formatter and a
+nanosecond formatter. Internally the nanosecond based dates will be


maybe s/formatter/field/? "date formatter" reads to me as "nanosecond field with a format that doesn't include nanos" but I think you mean date fields?

jpountz · 2019-01-24T08:39:43Z

server/src/main/java/org/elasticsearch/common/time/DateUtils.java

+
+        long seconds = nanoSecondsSinceEpoch / 1_000_000_000;
+        long nanos = nanoSecondsSinceEpoch % 1_000_000_000;
+        return Instant.ofEpochSecond(seconds, nanos);


I think this is incorrect for negative timestamps? you might want to use Math.floorDiv/floorMod instead.

jpountz · 2019-01-24T08:39:57Z

server/src/main/java/org/elasticsearch/common/time/DateUtils.java

+            return 0;
+        }
+
+        return nanoSecondsSinceEpoch / 1_000_000;


This is incorrect for negative times too.

spinscale · 2019-01-24T08:54:08Z

I'd favor date_nanos here, easier to remember than anything else...

spinscale · 2019-01-24T08:57:38Z

regarding treating the field differently with its default formatters: I do think that it makes sense to have it the same as the date field. Otherwise just switching from date to this one will suddenly index a completely different date when indexing just a number - I think that is more confusing?

Also, there is no epoch_nanos date formatter currently - I'd prefer people using human readable dates when indexing.

I am currently working on the fact that both can use the same default date formatter, so that the only difference would be in indexing, but not in configuration.

Does that make sense?

jpountz · 2019-01-24T09:00:23Z

Otherwise just switching from date to this one will suddenly index a completely different date when indexing just a number - I think that is more confusing?

It might be indeed. We might want to add a big warning to the docs that this field will interpret numbers as millis.

Does that make sense?

Yes, thanks.

spinscale · 2019-01-24T09:15:15Z

If everyone is good with rejecting pre 1970 dates, I'm happy to add that :-)

spinscale · 2019-01-24T10:07:38Z

regarding aggregations: those are still fully in milliseconds (I already updated the docs with this limitation, but havent pushed it yet, working on Jim's review comments at the moments).

This adds a dedicated field mapper that supports nanosecond resolution - at the price of a reduced date range. When using the date field mapper, the time is stored as milliseconds since the epoch in a long in lucene. This field mapper stores the time in nanoseconds since the epoch - which means its range is much smaller, ranging roughly from 1677 to 2262. If you are searching across indices where a date field is in nanoseconds and another date field in milliseconds, then an internal mechanism will convert the nanoseconds to milliseconds losing precision and then merge the search results. Relates elastic#27330

- fix docs - switch integration test to YAML test - fix missing numeric types/fielddata types - remove dead code in SearchPhaseController - rename mapper from nanosecond to date_nanos - only allow positive nanosecond dates (after 1970) - add more tests, ensure that doc value fields are working - add more tests, ensure that aggregations are working This is just a super dirty prototype I hashed together on friday afternoon, I will come up with a cleaner implementation next.

spinscale · 2019-01-28T16:49:26Z

@jimczi I made some more changes that we talked about.

One more open thing is to throw an exception when search_after is used in combination with nanosecond/millisecond indices. Can you point me to the code I should take a look at to do this?

spinscale · 2019-01-31T10:49:36Z

@jimczi I reverted the merging code, the custom nanosecond sort field, and kept the fieldmapper, the custom fielddata for script doc values and field doc values plus tests

jimczi

It looks great @spinscale . I left some comments and questions but it's getting close ;).

server/src/main/java/org/elasticsearch/index/fielddata/plain/SortedNumericDVIndexFieldData.java

jimczi · 2019-01-31T12:11:38Z

server/src/main/java/org/elasticsearch/index/fielddata/plain/AtomicLongFieldData.java

@@ -56,7 +56,27 @@ public long ramBytesUsed() {
    public final ScriptDocValues<?> getLegacyFieldValues() {


This function is unused since #30831. I'll open a separate pr to remove it.

I left the code in here for now, but can also remove it, whatever you prefer

I opened #38087

server/src/main/java/org/elasticsearch/index/fielddata/plain/SortedNumericDVIndexFieldData.java

docs/reference/mapping/types/date_nanos.asciidoc

…ano test

jimczi

I left a minor comment but it looks ready to me.
Thanks for iterating @spinscale

jimczi · 2019-02-01T09:05:19Z

server/src/main/java/org/elasticsearch/index/mapper/DateFieldMapper.java

@@ -181,21 +247,24 @@ public TypeParser() {
        }
    }

-    public static final class DateFieldType extends MappedFieldType {


This class can remain final ?

spinscale · 2019-02-01T14:34:34Z

@elasticmachine run tests elasticsearch-ci/1

spinscale · 2019-02-01T17:50:04Z

@elasticmachine run tests elasticsearch-ci/1

spinscale added >feature :Search Foundations/Mapping Index mappings, including merging and defining field types v7.0.0 labels Jan 23, 2019

jimczi requested changes Jan 23, 2019

View reviewed changes

jpountz requested changes Jan 24, 2019

View reviewed changes

spinscale added 6 commits January 25, 2019 17:34

fix docs

4357eff

review comment: fold nanosecond field mapper into existing field mapper

e6ffc6e

remove nanosecond date field mapper

05d5c26

remove nanosecond field mapper in indicesmodule

1cc1929

spinscale force-pushed the 1901-nanosecond-fieldmapper branch from 1599452 to ec6ef2a Compare January 25, 2019 17:15

spinscale added 5 commits January 28, 2019 13:54

Merge branch 'master' into 1901-nanosecond-fieldmapper

3e7633b

Merge branch 'master' into 1901-nanosecond-fieldmapper

3189d6a

minor refactoring, make code more readable

d1b08af

Merge branch 'master' into 1901-nanosecond-fieldmapper

d3a5da3

fix compilation/test issues

e7f4f48

spinscale added 5 commits January 28, 2019 23:38

run nanosecond yaml test on 7.0 only

dc708e9

Merge branch 'master' into 1901-nanosecond-fieldmapper

5627d9e

Merge branch 'master' into 1901-nanosecond-fieldmapper

98518ba

Merge branch 'master' into 1901-nanosecond-fieldmapper

92c079e

Merge branch 'master' into 1901-nanosecond-fieldmapper

ff77ce6

spinscale added 3 commits January 30, 2019 12:01

rename and document

c879621

Merge branch 'master' into 1901-nanosecond-fieldmapper

67f7cb5

Merge branch 'master' into 1901-nanosecond-fieldmapper

ab2a575

spinscale mentioned this pull request Jan 30, 2019

Ensure 6.x BWC compatibility when searching indices with nanosecond fields #38028

Closed

spinscale added 3 commits January 31, 2019 09:03

Merge branch 'master' into 1901-nanosecond-fieldmapper

65e47e9

revert back to old behaviour after discussion, no fancy merging for now

59f426f

add date/date_nanos to field caps tests

1aee465

jimczi reviewed Jan 31, 2019

View reviewed changes

spinscale added 2 commits January 31, 2019 14:20

Merge branch 'master' into 1901-nanosecond-fieldmapper

f7c89ab

incorporate review comments, fix field cap tests by having own date n…

5316c91

…ano test

jimczi approved these changes Feb 1, 2019

View reviewed changes

spinscale added 3 commits February 1, 2019 11:29

Merge branch 'master' into 1901-nanosecond-fieldmapper

cc5eac3

review comment: make class final again

3b3ac95

Merge branch 'master' into 1901-nanosecond-fieldmapper

5daa950

spinscale added 4 commits February 2, 2019 21:03

Merge branch 'master' into 1901-nanosecond-fieldmapper

a81e986

fix rest tests

384e572

Merge branch 'master' into 1901-nanosecond-fieldmapper

4d10b1e

revert changes in files that have not been touched

68dbe75

spinscale merged commit 87f3579 into elastic:master Feb 4, 2019

astefan mentioned this pull request Feb 5, 2019

SQL: support for nanoseconds precision for SQL INTERVALs #38402

Open

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

jimczi mentioned this pull request Feb 11, 2019

Date type has not enough precision for the logging use case. #10005

Closed

t-8ch mentioned this pull request Mar 6, 2019

Elasticsearch 7.0 will add nanosecond datemapper jaegertracing/jaeger#1406

Open

cosmo0920 mentioned this pull request Mar 13, 2019

time resolution, is there a way to change it ? fluent/fluentd-kubernetes-daemonset#267

Closed

codebrain mentioned this pull request Jul 12, 2019

Date Nanos implementation elastic/elasticsearch-net#3933

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nanosecond field mapper #37755

Add nanosecond field mapper #37755

spinscale commented Jan 23, 2019 •

edited

Loading

elasticmachine commented Jan 23, 2019

spinscale commented Jan 23, 2019

jimczi left a comment

jimczi Jan 23, 2019

jimczi Jan 23, 2019

jimczi Jan 23, 2019

jimczi Jan 23, 2019

spinscale Jan 23, 2019

spinscale Jan 23, 2019

jimczi Jan 23, 2019

jimczi Jan 23, 2019

jpountz left a comment

jpountz Jan 24, 2019

jpountz Jan 24, 2019

jpountz Jan 24, 2019

spinscale commented Jan 24, 2019

spinscale commented Jan 24, 2019

jpountz commented Jan 24, 2019

spinscale commented Jan 24, 2019

spinscale commented Jan 24, 2019

spinscale commented Jan 28, 2019

spinscale commented Jan 31, 2019

jimczi left a comment

jimczi Jan 31, 2019

spinscale Jan 31, 2019

jimczi Feb 1, 2019

jimczi left a comment

jimczi Feb 1, 2019

spinscale Feb 1, 2019

spinscale commented Feb 1, 2019

spinscale commented Feb 1, 2019

		@@ -56,7 +56,27 @@ public long ramBytesUsed() {
		public final ScriptDocValues<?> getLegacyFieldValues() {

Add nanosecond field mapper #37755

Add nanosecond field mapper #37755

Conversation

spinscale commented Jan 23, 2019 • edited Loading

elasticmachine commented Jan 23, 2019

spinscale commented Jan 23, 2019

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spinscale commented Jan 24, 2019

spinscale commented Jan 24, 2019

jpountz commented Jan 24, 2019

spinscale commented Jan 24, 2019

spinscale commented Jan 24, 2019

spinscale commented Jan 28, 2019

spinscale commented Jan 31, 2019

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimczi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spinscale commented Feb 1, 2019

spinscale commented Feb 1, 2019

spinscale commented Jan 23, 2019 •

edited

Loading