Optimize mongodb plugin #7207

UMU618 · 2019-04-25T06:09:12Z

mongodb plugin performance greatly enhanced

before:
fc::variant -> JSON string -> bson

now:
fc::variant -> bson

the conversion performance is 40~50x than before, when processes big blocks.

Just add bson.hpp, hope it will pass all checks and tests.

Improve for MongoDB sharding

UMU618 · 2019-04-25T06:49:24Z

2 failing and 15 successful checks
I just add bson.hpp, not even modify cpp, and the checks fail?

UMU618 · 2019-04-25T07:24:52Z

mongo_db_plugin.cpp modified

convert fc::variant to BSON directly

heifner · 2019-04-25T13:45:30Z

The nodeos_run_test-mongodb test is failing. It looks like because the format that the python expects for NumberLong has changed. Can you look at updating the test?

heifner · 2019-04-25T13:49:02Z

    cmd returned transaction id: 687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08, status: executed, (possible) block num: 48 

          cmd: echo 'db.transactions.findOne( { "trx_id": "687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08" } )' | mongo --host localhost --port 27017 EOStest

           ERROR: JSONDecodeError

           Raw MongoDB response: > MongoDB shell version v3.6.3

connecting to: mongodb://localhost:27017/EOStest

MongoDB server version: 3.6.3

{

	"_id" : ObjectId("5cc1646325fb28b2a8b31761"),

	"trx_id" : "687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08",

	"accepted" : true,

	"actions" : [

		{

			"account" : "eosio",

			"name" : "newaccount",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"creator" : "defproducera",

				"name" : "currency1111",

				"owner" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS72VGHvEb8Rwh3tYvNei3NeMRMewPJot3UXojHSv1PUDAiQdvzZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				},

				"active" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS7RkUJbfvyhWPJSFQrPpiCgX5SFvi9DWdssZn2Hn8gVE3K1WzVZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				}

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4601000000010003198dbddbf5f1afe834488c2c60291156860cf4ab3344ee9ee01c75c4fa607f5201000000010000000100034e5f928558c398ecfef966c2e9b0ca3e27ed088cf73dff8419ad7a38651ed44c01000000"

		},

		{

			"account" : "eosio",

			"name" : "buyram",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"payer" : "defproducera",

				"receiver" : "currency1111",

				"quant" : "200000.0000 SYS"

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4600943577000000000453595300000000"

		},

		{

			"account" : "eosio",

			"name" : "delegatebw",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"from" : "defproducera",

				"receiver" : "currency1111",

				"stake_net_quantity" : "100.0000 SYS",

				"stake_cpu_quantity" : "100.0000 SYS",

				"transfer" : NumberLong(0)

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4640420f0000000000045359530000000040420f0000000000045359530000000000"

		}

	],

	"context_free_actions" : [ ],

	"context_free_data" : [ ],

	"createdAt" : ISODate("2019-04-25T07:40:19.933Z"),

	"delay_sec" : NumberLong(0),

	"expiration" : "2019-04-25T07:40:49",

	"implicit" : false,

	"max_cpu_usage_ms" : NumberLong(0),

	"max_net_usage_words" : NumberLong(0),

	"ref_block_num" : NumberLong(30),

	"ref_block_prefix" : NumberLong(939910211),

	"scheduled" : false,

	"signatures" : [

		"SIG_K1_KgdrasgxWZHQSMyUVPfQGogfConEQppufWVb3oUSkbL5EJXYkVFNRvmwchPBAiAVU48sKnmMoPwP4V4XN74XjZmWDz2TcY"

	],

	"signing_keys" : [

		"EOS6mV8wVdsLpvh7M1TfGK4cQapt8HZMtgXEA5MuK9EyGZfv4cPu1"

	],

	"transaction_extensions" : [ ]

}

bye



           Normalized MongoDB response: > {

	"_id" : "ObjectId-5cc1646325fb28b2a8b31761",

	"trx_id" : "687c855999527e50c63c283078778e734c716f7f0fe9771690a6e1c0cff36f08",

	"accepted" : true,

	"actions" : [

		{

			"account" : "eosio",

			"name" : "newaccount",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"creator" : "defproducera",

				"name" : "currency1111",

				"owner" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS72VGHvEb8Rwh3tYvNei3NeMRMewPJot3UXojHSv1PUDAiQdvzZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				},

				"active" : {

					"threshold" : NumberLong(1),

					"keys" : [

						{

							"key" : "EOS7RkUJbfvyhWPJSFQrPpiCgX5SFvi9DWdssZn2Hn8gVE3K1WzVZ",

							"weight" : NumberLong(1)

						}

					],

					"accounts" : [ ],

					"waits" : [ ]

				}

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4601000000010003198dbddbf5f1afe834488c2c60291156860cf4ab3344ee9ee01c75c4fa607f5201000000010000000100034e5f928558c398ecfef966c2e9b0ca3e27ed088cf73dff8419ad7a38651ed44c01000000"

		},

		{

			"account" : "eosio",

			"name" : "buyram",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"payer" : "defproducera",

				"receiver" : "currency1111",

				"quant" : "200000.0000 SYS"

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4600943577000000000453595300000000"

		},

		{

			"account" : "eosio",

			"name" : "delegatebw",

			"authorization" : [

				{

					"actor" : "defproducera",

					"permission" : "active"

				}

			],

			"data" : {

				"from" : "defproducera",

				"receiver" : "currency1111",

				"stake_net_quantity" : "100.0000 SYS",

				"stake_cpu_quantity" : "100.0000 SYS",

				"transfer" : NumberLong(0)

			},

			"hex_data" : "60ae423ad15b974a1042081e4d75af4640420f0000000000045359530000000040420f0000000000045359530000000000"

		}

	],

	"context_free_actions" : [ ],

	"context_free_data" : [ ],

	"createdAt" : "ISODate-2019-04-25T07:40:19.933Z",

	"delay_sec" : NumberLong(0),

	"expiration" : "2019-04-25T07:40:49",

	"implicit" : false,

	"max_cpu_usage_ms" : NumberLong(0),

	"max_net_usage_words" : NumberLong(0),

	"ref_block_num" : NumberLong(30),

	"ref_block_prefix" : NumberLong(939910211),

	"scheduled" : false,

	"signatures" : [

		"SIG_K1_KgdrasgxWZHQSMyUVPfQGogfConEQppufWVb3oUSkbL5EJXYkVFNRvmwchPBAiAVU48sKnmMoPwP4V4XN74XjZmWDz2TcY"

	],

	"signing_keys" : [

		"EOS6mV8wVdsLpvh7M1TfGK4cQapt8HZMtgXEA5MuK9EyGZfv4cPu1"

	],

	"transaction_extensions" : [ ]

}

 Test failed.

heifner · 2019-04-25T17:14:13Z

2 failing and 15 successful checks
I just add bson.hpp, not even modify cpp, and the checks fail?

You likely caught develop in a bad state. What I posted above seems to be the only issue.

UMU618 · 2019-04-26T10:03:01Z

2 failing and 15 successful checks
I just add bson.hpp, not even modify cpp, and the checks fail?

You likely caught develop in a bad state. What I posted above seems to be the only issue.

Fixed. All checks have passed @heifner

plugins/mongo_db_plugin/include/eosio/mongo_db_plugin/bson.hpp

UMU618 and others added 3 commits March 25, 2019 15:33

Merge pull request #1 from UMU618/optimize-mongodb-plugin

9b03693

Improve for MongoDB sharding

Merge branch 'develop' of github.com:UMU618/eos into develop

3987447

fc::variant to BSON

3f2b658

convert fc::variant to BSON directly

f0014e2

* fix normalizeJsonObject

806338e

heifner suggested changes Apr 30, 2019

View reviewed changes

UMU618 and others added 3 commits May 6, 2019 12:07

+ use bson_oid_is_valid()

86a8a07

Merge branch 'develop' into optimize-mongodb-plugin

0840a79

Remove explicit version specification

afd997a

heifner approved these changes May 6, 2019

View reviewed changes

heifner merged commit d458d72 into EOSIO:develop May 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize mongodb plugin #7207

Optimize mongodb plugin #7207

UMU618 commented Apr 25, 2019

UMU618 commented Apr 25, 2019 •

edited

Loading

UMU618 commented Apr 25, 2019

heifner commented Apr 25, 2019

heifner commented Apr 25, 2019

heifner commented Apr 25, 2019

UMU618 commented Apr 26, 2019

Optimize mongodb plugin #7207

Optimize mongodb plugin #7207

Conversation

UMU618 commented Apr 25, 2019

UMU618 commented Apr 25, 2019 • edited Loading

UMU618 commented Apr 25, 2019

heifner commented Apr 25, 2019

heifner commented Apr 25, 2019

heifner commented Apr 25, 2019

UMU618 commented Apr 26, 2019

UMU618 commented Apr 25, 2019 •

edited

Loading