add additional checks to adaptor for live chain sync

[oxarbitrage]

Elasticsearch plugin with adaptor included at #1356 start to fail at around block 24M due to error:

java.lang.IllegalArgumentException: Can't merge a non object mapping [operation_history.op_object.policy] with an object mapping [operation_history.op_object.policy]

This pull attempts to fix it, testing now. Will check the full logs of all ES looking for errors until sync.

[abitmore]

Thanks for your hard work. Seeing you adding those I think it's better if you find all related fields and add them all. For example, active_special_authority comes from account options and I'm sure there are other similar fields there E.G. owner_special_authority, policy comes from vesting related operations and there may be related fields, and etc. Wish it helps.

[pmconrad]

IMO converting to a string is a bad fix.
What do the existing fields look like?

[oxarbitrage]

Ill try to find them all by following the ES logs. Service drops a failure each time one of this issues occur. Then we can try to do something better for some of them.

[oxarbitrage]

Currently at block 21M and no mapping errors in the elasticsearch logs with the last commit.

[oxarbitrage]

My server went down for maintenance overnight so i had to start over this morning. Currently in 21M blocks again and no errors.

[ryanRfox]

My node is now fully synced based on (most) of these changes. I did not have the two owner_special_authority entries in my build. However, it log these as [DEBUG] entries in the elasticsearch.log file (only two entries) and continued to function. I feel this PR should be merged.

[cogutvalera]

just as an idea about next code here

bitshares-core/libraries/plugins/elasticsearch/include/graphene/elasticsearch/elasticsearch_plugin.hpp

Lines 249 to 272 in ff0629c

	if (o.find("proposed_ops") != o.end())
	{
	o["proposed_ops"] = fc::json::to_string(o["proposed_ops"]);
	}
	if (o.find("initializer") != o.end())
	{
	o["initializer"] = fc::json::to_string(o["initializer"]);
	}
	if (o.find("policy") != o.end())
	{
	o["policy"] = fc::json::to_string(o["policy"]);
	}
	if (o.find("predicates") != o.end())
	{
	o["predicates"] = fc::json::to_string(o["predicates"]);
	}
	if (o.find("active_special_authority") != o.end())
	{
	o["active_special_authority"] = fc::json::to_string(o["active_special_authority"]);
	}
	if (o.find("owner_special_authority") != o.end())
	{
	o["owner_special_authority"] = fc::json::to_string(o["owner_special_authority"]);
	}

it looks very similar, just with different fields, perhaps would be better to iterate through collection of fields.

Thanks !

[oxarbitrage]

@cogutvalera i agree. I didn't know they were going to be that much to make something better.
Still, i think we can postpone that change for the next normal release as it is hard to sync everything again and we are running out of time, i know that with the changes in this release the chain synchronize fully without any errors.

@pmconrad i found some hints about some of the new fields with problems.

the case of operation_history.op_object.extensions.active_special_authority and operation_history.op_object.extensions.owner_special_authority seems to be the format:

["1",{"asset":"1.3.743","num_top_holders":5}]

This is not supported by elasticsearch.

Researching the others.

[oxarbitrage]

similar issue with policy, example:

["1",{"start_claim":"2018-01-25T17:59:28","vesting_seconds":31536000}]

[oxarbitrage]

the predicates is a similar issue in the assert_operation:

[["0",{"account_id":"1.2.96397","name":"openledger-wallet"}]]

[oxarbitrage]

I think this can be merged as it is and open a new issue for code style and a possible way to insert some of those problematic fields.

The information lost here is minimal and we know is fully working. I am interested on having a working version in master and testnet and continue the work in the develop.

Also to discuss, plugin need to be checked after new operations are added, could have an unsupported field.

[pmconrad]

https://www.elastic.co/guide/en/elasticsearch/reference/current/array.html :

...however, all values in the array must be of the same datatype...

That sucks. I'm surprised this doesn't bite us more often, since all static_variants serialize like this IIRC.
Then again, static variant serialization sucks even more. :-/

Ok for now, but we should change this in the next release IMO. E. g.

[ tag_no, { object:data }] -> { _type: type_name, object:data }

[abitmore]

Ok for now, but we should change this in the next release IMO. E. g.
[ tag_no, { object:data }] -> { _type: type_name, object:data }

If you mean to change the overall jsonification, I disagree, because it may break all the clients. It's a big change so I'd rather live with the sucking one. If change ES only, then perhaps acceptable.

[pmconrad]

I mean it sucks generally, but we should only adapt this for ES.