Docs for synthetic source

[nik9000]

This adds some basic docs for synthetic source both to get us started
documenting it and to show how I'd like to get it documented - with a
central section in the docs for _source and "satellite" sections in
each of the supported field types that link back to the central section.

Preview

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[nik9000]

I've pushed docs for the remaining fields.

[nik9000]

One thing I forgot to ask... near the bottom of the main _source field page there's the warning "Think before disabling the _source field". If those restrictions apply to synthetic source as well, we may want to move that warning up to the top of the page, or something like that.

For the most part none of those things are true for synthetic source. The debugging one is a little complicated and so is the corruption. But for the most part, no, synthetic source exists to solve those problems.

[nik9000]

I pushed some more stuff into the "text" page as well.

[javanna]

reading this, as much as I understand why you mention it, keys ordering is never a guarantee in JSON land. I wonder if it's then needed to provide an example. Could we shorten it and say "You should never rely on keys ordering, but if you do beware you'll lose that with synthetic source as what you'll get back is not 100% what you sent"?

[nik9000]

I get that. I'm providing an example for everything so I like the.... balance? I don't know the right word. I could link to the spec and mention that ordering isn't supported by the spec.

[javanna]

I don't have a strong opinion but while the examples with arrays ought to be mentioned, taking this one out would make the docs page a little more compact without losing much?

[nik9000]

We need a tie breaker. @romseygeek, break a tie!

[javanna]

No pressure :)

[romseygeek]

I'm with @javanna, we don't guarantee key ordering anywhere so I don't think an example is necessary here.

[nik9000]

Democracy wins. I'll remove the example. I'll reduce this to a note.

[javanna]

I wonder what is specific of synthetic source about how fields are mapped: isn't this all about dynamic mappings? Did you mean to say that synthetic _source recreate the objects structure or not depending on how fields are mapped?

[nik9000]

It recreates the structure precisely as the objects are mapped. At first I tried to explicitly create a field mapped as foo.bar.baz but the mapping infrastructure unraveled it so I went with this. I'll make an example, one moment.

[nik9000]

Yeah, check this out:

$ curl -uelastic:password -XDELETE localhost:9200/test
$ curl -uelastic:password -XPUT -HContent-Type:application/json localhost:9200/test -d'{
  "mappings": {
    "properties": {
      "foo.bar.baz": {
        "type": "keyword"
      }
    }
  }
}'
$ curl -uelastic:password localhost:9200/test/_mappings?pretty
{"acknowledged":true}{"acknowledged":true,"shards_acknowledged":true,"index":"test"}{
  "test" : {
    "mappings" : {
      "properties" : {
        "foo" : {
          "properties" : {
            "bar" : {
              "properties" : {
                "baz" : {
                  "type" : "keyword"
                }
              }
            }
          }
        }
      }
    }
  }
}

Compare:

$ curl -uelastic:password -XDELETE localhost:9200/test
$ curl -uelastic:password -XPUT -HContent-Type:application/json localhost:9200/test -d'{
  "mappings": {
    "subobjects": false,
    "properties": {
      "foo.bar.baz": {
        "type": "keyword"
      }
    }
  }
}'
$ curl -uelastic:password localhost:9200/test/_mappings?pretty
{"acknowledged":true}{"acknowledged":true,"shards_acknowledged":true,"index":"test"}{
  "test" : {
    "mappings" : {
      "subobjects" : false,
      "properties" : {
        "foo.bar.baz" : {
          "type" : "keyword"
        }
      }
    }
  }
}

So I wrote the example this way because it was the shortest way to get objects with dots in the name. But maybe it's unclear?

[javanna]

Ok, I guess I was wondering why this needs to be specifically explained, isn't it the expected behaviour? :)

[nik9000]

I figured it'd be useful to explain "this does the right thing with that thing Luca just built". But maybe it's not worth it because it's not something folks want much?

[javanna]

Scratch that, maybe this is not so obvious :)
One thing to mention could be that you always recreate the object structure despite dots in fields names were provided in the first place (unless subobjects are disabled). Why not always flatten by the way? After all, there are two ways to provide that document and that leads to the same mapping, you have 50% chances to pick the variant that was sent in the first place.

By the way we may start accepting docs with subobjects even when subobjects are disabled and treat them like dotted fields, but I think the current behaviour of "flattening" would still be good in that case.

[nik9000]

I've removed the section

[nik9000]

Oh no! I removed the section!

Why not always flatten by the way?

We asked some kibana friends and they said folks liked things shaped this way instead of flattened. The objects feel right to folks. Also, I figured if we were guessing anyway the "more objecty" approach was probably more likely to match what users sent. I figured flattened fields with dots in them is more rare.

[javanna]

fine with me. I think you removed only the section with subobjects false and you could make the remaining section about recreating objects shorter by saying that you follow the mappings structure hence prefer nested objects over dots in fields names but the opposite when subobjects are disabled.

[nik9000]

I think you removed only the section with subobjects false and you could make the remaining section about recreating objects shorter by saying that you follow the mappings structure hence prefer nested objects over dots in fields names but the opposite when subobjects are disabled.

Two things:

1. Don't use the word nested unless you mean it. I get scared.
2. Do you want me to change the words above? Could you suggest a change? I think the "dynamic mappings work like BLAH" bit is useful because it explains where those objects come from. I don't mean to explain all of dynamic mappings, just enough to give folks a hint.

[kilfoyle]

LGTM! 🚗
Thanks a bunch for adding these Nik. They look great.

[javanna]

Is it worth quickly describing the difference between these two options, or people need to go to their corresponding pages to find out?

[javanna]

Elasticsearch can reconstruct source content on the fly upon retrieval?

[javanna]

, while dots in field names are preserved within objects that have subobjects disabled (with a link to the docs for the subobjects paramater)

[javanna]

I left a couple small comments, LGTM otherwise. Thanks!

[nik9000]

I left a couple small comments, LGTM otherwise. Thanks!

🤘

I'll integrate.