Synthetic Source

[nik9000]

This shrinks the index by implementing a "synthetic" _source field. Instead of saving the field to disk we reconstruct it on the fly using our column store, doc values.

Before removing the feature flag

• Initial implementation Synthetic source #85649
• Figure out how much performance we'd get from using synthetic source for recovery - thus removing the `_recovery_source field. Synthetic source #85649 (comment)
• Figure out final API to turn this on (@qhoxie)
• If it stays synthetic: true then we'll have to fail mappings that contain enabled: false, synthetic: true Synthetic source: don't allow disabling source #87270
• Flip API from synthetic: true to synthetic: strict so we have room to add more later. We totally will. @romseygeek
• Rally track Support synthetic source in tsdb and nyc_taxis tracks rally-tracks#268 + Add fetch operations for synthetic source rally-tracks#270
• Resolve remaining round trip tests Synethetic Source: Fix scaled float #86760
• Realtime GET should synthesize on load to return consistent results. (Tests for synthetic _source from translog #87578)
• Add an option to _search to simulate synthetic source (Add an option to _search to force synthetic source #87068)
• Support the simulate option on GET and MGET (Add force_synthetic_source to GET #87536 + Add force_synthetic_source to mget #87574)
• Make sure we throw an error if you try to enable or disable synthetic source on an index Synthetic source: paranoid tests for configuration #87182
• Support "subobjects" Add support for dots in field names for metrics usecases #86166 (Synthetic source: tests for disabling subobjects #87261)
• Docs Docs for synthetic source #87416
• Figure out highlighting Fixup highlighting with synthetic source #87667

Later

• Make sure there is a nice error message when scripts try to access synethic source - it won't be there. They should use doc values or the fancy new fields API. @tmgordeeva Synthetic source error on script loads #88334
• Add support for more field types
  • aggregate_metric_double field type Add synthetic_source support to aggregate_metric_double fields #88909
  • constant_keyword Enable synthetic source support on constant keyword fields #88603
  • dense_vector Synthetic _source: support dense_vector #89840
  • histogram Synthetic _source: support histogram field #89833
  • keyword fields with ignore_above (Synthetic source: load text from stored fields #87480 + Synthetic _source: support ignore_above #89466)
  • match_only_text Synthetic _source: support match_only_text #89516
  • version Synthetic _source: support version field type #89706
  • _doc_count Support synthetic _source for _doc_count field #91465
• Support fields in runtime fields scripts
  • Numbers REST tests fetching fields with synthetic _source #89888
  • ip REST tests fetching fields with synthetic _source #89888
  • text (Synthetic _source: support field in many cases #89950 + more)
  • keyword (Synthetic _source: support field in many cases #89950)
  • match_only_text (Synthetic _source: support field in many cases #89950)
• Support loading from stored fields (text would love it!) Synthetic source: load text from stored fields #87480
• Rally tests for random fetch
• Look into the enrich processor (More tests for enrich processor #89554)
• Improve performance of synthesis
  • General Speed up synthetic source #87882
  • Load column-wise Speed up synthetic keyword, ip, and text fields #87930 Synthetic source numbers in columns #88025
  • Parallel loading?
• Make fields API aware of synthetic-ness and go to doc values rather than rebuilding _source if _source isn't separately needed.
• Document best practices for load over synthetic source
• Support for ignore_malformed Synthetic _source: support ignore_malformed #90007
  • ip Synthetic _source: ignore_malformed for ip #90038
  • numeric Support malformed numbers in synthetic _source #90428
  • scaled_float Support synthetic source for scaled_float and unsigned_long when ignore_malformed is used #109506
  • date/date_nanos Support synthetic source for date fields when ignore_malformed is used #109410
  • geo_point Support synthetic source for geo_point when ignore_malformed is used #109651
  • histogram Support synthetic source together with ignore_malformed in histogram fields #109882
  • aggregate_double_metric Support synthetic source for aggregate_metric_double when ignore_malformed is used #108746 Simplify ignore_malformed handling for synthetic souce in aggregate_metric_double #109888
  • text type family Text fields are stored by default in TSDB indices #106338
  • keyword Synthetic _source: support ignore_above #89466

Much later

• Synthesize instead of using _recovery_source - we find that it'd improve write performance by ~11%. We'd have to synthesize on load instead. That's pretty slow. We'd love the 11% but we have to be careful here.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[jsoriano]

@nik9000 does synthetic source leverage _source_include/_source_exclude for the fields it has to synthesize?

[nik9000]

@nik9000 does synthetic source leverage _source_include/_source_exclude for the fields it has to synthesize?

It does not. There is no support at the moment for any kind of partial synthesis.

[rocco8620]

Awesome feature, can't wait to have this in GA!!

[Kiriakos1998]

Hello @nik9000 , can I pick some of the unchecked subtasks?

[nik9000]

Hello @nik9000 , can I pick some of the unchecked subtasks?

I think all of the unchecked tasks are quick difficult to be honest. ignore_malformed are maybe easier, but I wouldn't suggest picking it up.

Also you'd need a committer buddy and I've had to move on to other tasks sadly. That might be quite difficult to find too.

[iby-dev]

@nik9000 does synthetic source leverage _source_include/_source_exclude for the fields it has to synthesize?

It does not. There is no support at the moment for any kind of partial synthesis.

Hi @nik9000 - just for my own clarity. You can either use mode: synthetic on its own or use the _source_include/_source_exclude ? But the two cannot be combined ? Is this correct ?

[nik9000]

Hi @nik9000 - just for my own clarity. You can either use mode: synthetic on its own or use the _source_include/_source_exclude ? But the two cannot be combined ? Is this correct ?

Right. I honestly didn't know how to combine them so I just declared combining them to be incompatible.

Keep in mind synthetic source is only GA for time series indices and data streams. I've had to move on to other things but expect folks will get back to working on getting synthetic source good in more contexts at some point soon.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

I'm closing this issue, because most of the mentioned items have been completed. Most of the uncompleted items have dedicated issues.