Storing embedded entity, it's ignoring exclude_from_indexes from embedded

[igama]

I'm trying to have a embedded entity with a field that is larger than 1500bytes. I add that field to the exclude_from_indexes key of the embedded entity. When I try and save the parent Entity it tells me the field is bigger than 1500bytes. If I save the embedded entity independently, it works.

Is exclude_from_indexes ignored on embedded entity?

client = datastore.Client(dataset_id=projectID)
record_key = client.key('Record', my_id)
record_entity = datastore.Entity(record_key)

embedded_key = client.key('Data', another_id)
embedded_entity = datastore.Entity(key=embedded_key,exclude_from_indexes=('big_field',))
embedded_entity['field1']='1234'
embedded_entity['big_field']='large string bigger than 1500bytes'

record_entity['RandomFieldName']=embedded_entity

client.put(record_entity)
#Error: gcloud.exceptions.BadRequest: 400 The value of property "big_field" is longer than 1500 bytes.

client.put(embedded_entity)
#No Error

[dhermes]

@pcostell What's the expected behavior here?

[pcostell]

This should succeed. I am able to get this to work successfully directly from the API (in a commit):

{
 "mode": "NON_TRANSACTIONAL",
 "mutation": {
  "insertAutoId": [
   {
    "properties": {
     "embedded": {
      "entityValue": {
       "properties": {
        "long_prop": {
         "stringValue": "a string longer than 1500 bytes", # I just used 'a'*1501
         "indexed": false
        }
       }
      }
     }
    },
    "key": {
     "path": [
      {
       "kind": "ModelWithEmbedded"
      }
     ]
    }
   }
  ]
 }
}

Gives me a 200.

[pcostell]

It looks like information about exclude_from_indexes is not propagated in embedded entities: https://github.com/GoogleCloudPlatform/gcloud-python/blob/c8d231ba7462e2f35d689b526a70cef4712c565a/gcloud/datastore/batch.py#L282

[dhermes]

Thanks @pcostell!

[igama]

@pcostell thanks for debugging this.

[dhermes]

@igama This has been fixed:

>>> from gcloud import datastore
>>> from gcloud.environment_vars import TESTS_DATASET
>>> from gcloud.datastore import client
>>> client.DATASET = TESTS_DATASET
>>> CLIENT = datastore.Client()
>>> record_key = CLIENT.key('Record', 1234)
>>> record_entity = datastore.Entity(record_key)
>>> embedded_key = CLIENT.key('Data', 5678)
>>> embedded_entity = datastore.Entity(key=embedded_key, exclude_from_indexes=('big_field',))
>>> embedded_entity['field1'] = '1234'
>>> embedded_entity['big_field'] = 'large string bigger than 1500bytes ' * 50
>>> len(embedded_entity['big_field'])
1750
>>> record_entity['RandomFieldName'] = embedded_entity
>>> CLIENT.put(record_entity)
>>> CLIENT.put(embedded_entity)
>>> 
>>> CLIENT.delete(record_entity.key)
>>> CLIENT.delete(embedded_entity.key)

[pcostell]

Note -- This should probably use . to qualify the name (it matches a proposed field mask design):

exclude_from_indexes=('RandomFieldName.big_field')

Otherwise, there is no way to exclude nested field vs non-nested fields.

[dhermes]

@pcostell I'm not sure what you mean RE: "there is no way" --- see the above. The way is to exclude them on the nested Entity and then we propagate that into the protobuf created. Maybe you mean it would be useful to differentiate between properties in queries?

As for the name used, see the reported snippet at the top of this issue.

[pcostell]

Ah sorry definitely read recorded_entity instead of embedded_entity at embedded creation. Please ignore my comment :-)

[dhermes]

No worries. Thanks for taking the time to read the notification

[igama]

@dhermes thank you, Will try it in the next days.