Ref !3/peerplays-network#376: Graphene Updates

This adds the most important updates to Graphene from BitShares. Most notably,
bitshares/bitshares-core#1506

Second most notably, it updates Peerplays' FC to be in sync with BitShares FC.

This is a squash commit of several subcommits. The subcommit messages are
reproduced below:

Replace fc::uint128 with boost::multiprecision::uint128_t

replace smart_ref with shared_ptr

Fixes/Remove Unused

Remove NTP time

Remove old macro

This macro is now in FC, so no need to define it here anymore

Replaced fc::array with std::array

Separate exception declaration and implementation

Adapted to fc promise changes

Fixes

Add back in some of Peter's fixes that got lost in the cherry pick

_hash endianness fixes

Remove all uses of fc/smart_ref

It's gone, can't use it anymore

Replace improper static_variant operator overloads with comparators

Fixes

Remove boost::signals from build system; it's header-only so it's not
listed in cmake anymore.

Also remove some unused hashing code

Impl. pack/unpack functions for extension class

Ref #1506: Isolate chain/protocol to its own library

Ref #1506: Add object_downcast_t

Allows the more concise expression `object_downcast_t<xyz>` instead of
the old `typename object_downcast<xyz>::type`

Ref #1506: Move ID types from db to protocol

The ID types, object_id and object_id_type, were defined in the db
library, and the protocol library depends on db to get these types.
Technically, the ID types are defined by the protocol and used by the
database, and not vice versa. Therefore these types should be in the
protocol library, and db should depend on protocol to get them.

This commit makes it so.

Ref #1506: Isolate chain/protocol to its own library

Remove commented-out index code

Wrap overlength line

Remove unused key types

Probably fix Docker build

Fix build after rebase

Ref #1506/#1737: Some requested changes

Ref #1506/#1737: Macro-fy ID type definitions

Define macros to fully de-boilerplate ID type definitions.

Externalities:
 - Rename transaction_object -> transaction_history_object
 - Rename impl_asset_dynamic_data_type ->
impl_asset_dynamic_data_object_type
 - Rename impl_asset_bitasset_data_type ->
impl_asset_bitasset_data_object_type

The first is to avoid a naming collision on transaction_id_type, and the
other two are to maintain consistency with the naming of the other
types.

Ref #1506/#1737: Fix clean_name()

Ref #1506/#1737: Oops

Fix .gitignore

Externalized serialization in protocol library

Fix compile sets

Delete a couple of ghost files that were in the tree but not part
of the project (I accidentally added them to CMakeLists while
merging, but they're broken and not part of the Peerplays code), and
add several files that got dropped from the build during merge.

General fixes

Fix warnings, build issues, unused code, etc.

Fix #1772 by decprecating cli_wallet -H

More fixes

Fix errors and warnings and generally coax it to build

Fix test

I'm pretty sure this didn't break from what I did... But I can't build
the original code, so I can't tell. Anyways, this one now passes...
Others still fail...

Small fix

Fix crash in auth checks

Final fixes

Last round of fixes following the rebase to Beatrice

Rename project in CMakeLists.txt

The CMakeLists.txt declared this project as BitShares and not Peerplays,
which makes it confusing in IDEs. Rename it to be clear which project is
open.

Resolve peerplays-network#374

Replace all object refs in macros with IDs, and fix affected tests to look
up objects by ID rather than using invalidated refs.

A full audit of all tests should be performed to eliminate any further
usage of invalidated object references.

Resolve peerplays-network#373: Add object notifiers

Various fixes

Fixes to various issues, primarily reflections, that cropped up
during merge conflict resolution

Fix startup bug in Bookie plugin

Bookie plugin was preventing the node from starting up because it
registered its secondary indexes to create objects in its own primary
indexes to track objects being created in other primary indexes, and did
so during its `initialize()` step, which is to say, before the database
was loaded from disk at startup. This caused the secondary indexes to
create tracker objects when the observed indexes were loading objects
from disk. This then caused a failure when these tracker indexes were
later loaded from disk, and the first object IDs collided.

This is fixed by refraining from defining secondary indexes until the
`startup()` stage rather than the `initialize()` stage. Primary indexes
are registered in `initialize()`, secondary indexes are registered in
`startup()`.

This also involved adding a new method, "add_secondary_index()", to
`object_database`, as before there was no way to do this because you
couldn't get a non-const index from a non-const database.

I have no idea how this was working before I got here...

Fix egenesis install

Fixes after updates

Rebase on updated develop branch and fix conflicts

.gitmodules

@@ -4,6 +4,6 @@
     ignore = dirty
 [submodule "libraries/fc"]
     path = libraries/fc
-    url = https://github.com/peerplays-network/peerplays-fc.git
+    url = https://github.com/nathanhourt/peerplays-fc
     branch = latest-fc
     ignore = dirty

CMakeLists.txt

@@ -8,6 +8,7 @@ set( CLI_CLIENT_EXECUTABLE_NAME graphene_client )
 set( GUI_CLIENT_EXECUTABLE_NAME Peerplays )
 set( CUSTOM_URL_SCHEME "gcs" )
 set( INSTALLER_APP_ID "68ad7005-8eee-49c9-95ce-9eed97e5b347" )
+set( CMAKE_CXX_STANDARD 14 )
 
 # http://stackoverflow.com/a/18369825
 if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
@@ -46,7 +47,6 @@ LIST(APPEND BOOST_COMPONENTS thread
                              system
                              filesystem
                              program_options
-                             signals
                              serialization
                              chrono
                              unit_test_framework

libraries/CMakeLists.txt

@@ -5,6 +5,6 @@ add_subdirectory( egenesis )
 add_subdirectory( fc )
 add_subdirectory( net )
 add_subdirectory( plugins )
-add_subdirectory( time )
 add_subdirectory( utilities )
 add_subdirectory( wallet )
+add_subdirectory( protocol )

libraries/app/CMakeLists.txt

@@ -12,9 +12,8 @@ add_library( graphene_app
            )
 
 # need to link graphene_debug_witness because plugins aren't sufficiently isolated #246
-#target_link_libraries( graphene_app graphene_market_history graphene_account_history graphene_chain fc graphene_db graphene_net graphene_utilities graphene_debug_witness )
 target_link_libraries( graphene_app
-                       PUBLIC graphene_net graphene_utilities
+                       PUBLIC graphene_chain graphene_net graphene_utilities
                        graphene_account_history graphene_accounts_list graphene_affiliate_stats graphene_bookie graphene_debug_witness graphene_elasticsearch graphene_es_objects graphene_generate_genesis graphene_market_history )
 
 target_include_directories( graphene_app
@@ -43,7 +42,7 @@ add_library( graphene_plugin
            )
 
 target_link_libraries( graphene_plugin
-                       PUBLIC graphene_net graphene_utilities )
+                       PUBLIC graphene_chain graphene_net graphene_utilities )
 
 target_include_directories( graphene_plugin
                             PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include" )

libraries/app/api.cpp

@@ -29,14 +29,15 @@
 #include <graphene/chain/database.hpp>
 #include <graphene/chain/get_config.hpp>
 #include <graphene/utilities/key_conversion.hpp>
-#include <graphene/chain/protocol/fee_schedule.hpp>
+#include <graphene/protocol/fee_schedule.hpp>
 #include <graphene/chain/confidential_object.hpp>
 #include <graphene/chain/market_object.hpp>
-#include <graphene/chain/transaction_object.hpp>
+#include <graphene/chain/transaction_history_object.hpp>
 #include <graphene/chain/withdraw_permission_object.hpp>
 #include <graphene/chain/worker_object.hpp>
 #include <graphene/chain/tournament_object.hpp>
 
+#include <fc/crypto/base64.hpp>
 #include <fc/crypto/hex.hpp>
 #include <fc/rpc/api_connection.hpp>
 #include <fc/thread/future.hpp>
@@ -169,7 +170,7 @@ namespace graphene { namespace app {
              {
                 auto block_num = b.block_num();
                 auto& callback = _callbacks.find(id)->second;
-                fc::async( [capture_this,this,id,block_num,trx_num,trx,callback]() {
+                fc::async( [capture_this,id,block_num,trx_num,trx,callback]() {
                    callback( fc::variant( transaction_confirmation{ id, block_num, trx_num, trx },
                                           GRAPHENE_MAX_NESTED_OBJECTS ) );
                 } );
@@ -190,10 +191,10 @@ namespace graphene { namespace app {
     fc::variant network_broadcast_api::broadcast_transaction_synchronous(const signed_transaction& trx)
     {
        _app.chain_database()->check_tansaction_for_duplicated_operations(trx);
-
-       fc::promise<fc::variant>::ptr prom( new fc::promise<fc::variant>() );
-       broadcast_transaction_with_callback( [=]( const fc::variant& v ){
-        prom->set_value(v);
+        
+       fc::promise<fc::variant>::ptr prom = fc::promise<fc::variant>::create();
+       broadcast_transaction_with_callback( [prom]( const fc::variant& v ){
+          prom->set_value(v);
        }, trx );
 
        return fc::future<fc::variant>(prom).wait();

libraries/app/application.cpp

@@ -26,8 +26,10 @@
 #include <graphene/app/application.hpp>
 #include <graphene/app/plugin.hpp>
 
-#include <graphene/chain/protocol/fee_schedule.hpp>
-#include <graphene/chain/protocol/types.hpp>
+#include <graphene/chain/db_with.hpp>
+#include <graphene/chain/genesis_state.hpp>
+#include <graphene/protocol/fee_schedule.hpp>
+#include <graphene/protocol/types.hpp>
 
 #include <graphene/egenesis/egenesis.hpp>
 
@@ -378,7 +380,6 @@ namespace detail {
             _chain_db->enable_standby_votes_tracking( _options->at("enable-standby-votes-tracking").as<bool>() );
          }
 
-         bool replay = false;
          std::string replay_reason = "reason not provided";
 
          if( _options->count("replay-blockchain") )
@@ -552,7 +553,7 @@ namespace detail {
          _chain_db->push_transaction( transaction_message.trx );
       } FC_CAPTURE_AND_RETHROW( (transaction_message) ) }
 
-      virtual void handle_message(const message& message_to_process) override
+      virtual void handle_message(const message&) override
       {
          // not a transaction, not a block
          FC_THROW( "Invalid Message Type" );

libraries/app/config_util.cpp

@@ -210,7 +210,7 @@ static void load_config_file(const fc::path& config_ini_path, const bpo::options
 {
    deduplicator dedup;
    bpo::options_description unique_options("Graphene Witness Node");
-   for( const boost::shared_ptr<bpo::option_description> opt : cfg_options.options() )
+   for( const boost::shared_ptr<bpo::option_description>& opt : cfg_options.options() )
    {
       const boost::shared_ptr<bpo::option_description> od = dedup.next(opt);
       if( !od ) continue;
@@ -259,7 +259,7 @@ static void create_new_config_file(const fc::path& config_ini_path, const fc::pa
    deduplicator dedup(modify_option_defaults);
    std::ofstream out_cfg(config_ini_path.preferred_string());
    std::string plugin_header_surrounding( 78, '=' );
-   for( const boost::shared_ptr<bpo::option_description> opt : cfg_options.options() )
+   for( const boost::shared_ptr<bpo::option_description>& opt : cfg_options.options() )
    {
       const boost::shared_ptr<bpo::option_description> od = dedup.next(opt);
       if( !od ) continue;

libraries/app/database_api.cpp

@@ -26,8 +26,10 @@
 #include <graphene/chain/get_config.hpp>
 #include <graphene/chain/tournament_object.hpp>
 #include <graphene/chain/account_object.hpp>
-#include <graphene/chain/protocol/address.hpp>
-#include <graphene/chain/pts_address.hpp>
+#include <graphene/chain/hardfork.hpp>
+
+#include <graphene/protocol/address.hpp>
+#include <graphene/protocol/pts_address.hpp>
 
 #include <fc/bloom_filter.hpp>
 
@@ -37,7 +39,6 @@
 
 #include <boost/range/iterator_range.hpp>
 #include <boost/rational.hpp>
-#include <boost/multiprecision/cpp_int.hpp>
 
 #include <cctype>
 
@@ -2594,10 +2595,9 @@ graphene::app::gpos_info database_api_impl::get_gpos_info(const account_id_type
 #endif
 
    vector<vesting_balance_object> account_vbos;
-   const time_point_sec now = _db.head_block_time();
    auto vesting_range = _db.get_index_type<vesting_balance_index>().indices().get<by_account>().equal_range(account);
    std::for_each(vesting_range.first, vesting_range.second,
-                  [&account_vbos, now](const vesting_balance_object& balance) {
+                  [&account_vbos](const vesting_balance_object& balance) {
                      if(balance.balance.amount > 0 && balance.balance_type == vesting_balance_type::gpos
                         && balance.balance.asset_id == asset_id_type())
                         account_vbos.emplace_back(balance);

libraries/app/include/graphene/app/api.hpp

@@ -25,8 +25,8 @@
 
 #include <graphene/app/database_api.hpp>
 
-#include <graphene/chain/protocol/types.hpp>
-#include <graphene/chain/protocol/confidential.hpp>
+#include <graphene/protocol/types.hpp>
+#include <graphene/protocol/confidential.hpp>
 
 #include <graphene/market_history/market_history_plugin.hpp>
 #include <graphene/accounts_list/accounts_list_plugin.hpp>

libraries/app/include/graphene/app/database_api.hpp

@@ -25,7 +25,7 @@
 
 #include <graphene/app/full_account.hpp>
 
-#include <graphene/chain/protocol/types.hpp>
+#include <graphene/protocol/types.hpp>
 
 #include <graphene/chain/database.hpp>
 

libraries/app/include/graphene/app/full_account.hpp

@@ -26,6 +26,8 @@
 #include <graphene/chain/account_object.hpp>
 #include <graphene/chain/vesting_balance_object.hpp>
 #include <graphene/chain/market_evaluator.hpp>
+#include <graphene/chain/market_object.hpp>
+#include <graphene/chain/proposal_object.hpp>
 #include <graphene/chain/withdraw_permission_object.hpp>
 
 namespace graphene { namespace app {
@@ -67,7 +69,6 @@ FC_REFLECT( graphene::app::full_account,
             (limit_orders)
             (call_orders)
             (settle_orders)
-            (proposals)
             (assets)
             (withdraws)
             (proposals) 

libraries/app/plugin.cpp

@@ -23,7 +23,7 @@
  */
 
 #include <graphene/app/plugin.hpp>
-#include <graphene/chain/protocol/fee_schedule.hpp>
+#include <graphene/protocol/fee_schedule.hpp>
 
 namespace graphene { namespace app {
 

libraries/chain/CMakeLists.txt

@@ -5,11 +5,9 @@ set_source_files_properties( "${CMAKE_CURRENT_BINARY_DIR}/include/graphene/chain
 
 add_dependencies( build_hardfork_hpp cat-parts )
 
-file(GLOB HEADERS "include/graphene/chain/*.hpp")
-file(GLOB PROTOCOL_HEADERS "include/graphene/chain/protocol/*.hpp")
+file(GLOB HEADERS "include/graphene/chain/*.hpp" "${CMAKE_CURRENT_BINARY_DIR}/include/graphene/chain/hardfork.hpp")
 
 file(GLOB CPP_FILES "*.cpp")
-file(GLOB PROTOCOL_CPP_FILES "protocol/*.cpp")
 
 #if( GRAPHENE_DISABLE_UNITY_BUILD )
    list(FILTER CPP_FILES EXCLUDE REGEX "[/]database[.]cpp$")
@@ -23,14 +21,12 @@ file(GLOB PROTOCOL_CPP_FILES "protocol/*.cpp")
 
 add_library( graphene_chain
              ${CPP_FILES}
-             ${PROTOCOL_CPP_FILES}
              ${HEADERS}
-             ${PROTOCOL_HEADERS}
              "${CMAKE_CURRENT_BINARY_DIR}/include/graphene/chain/hardfork.hpp"
            )
 
 add_dependencies( graphene_chain build_hardfork_hpp )
-target_link_libraries( graphene_chain graphene_db )
+target_link_libraries( graphene_chain fc graphene_db graphene_protocol )
 target_include_directories( graphene_chain
                             PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include" "${CMAKE_CURRENT_BINARY_DIR}/include" )
 
@@ -46,4 +42,3 @@ INSTALL( TARGETS
    ARCHIVE DESTINATION lib
 )
 INSTALL( FILES ${HEADERS} DESTINATION "include/graphene/chain" )
-INSTALL( FILES ${PROTOCOL_HEADERS} DESTINATION "include/graphene/chain/protocol" )

libraries/chain/account_evaluator.cpp

@@ -29,7 +29,7 @@
 #include <graphene/chain/exceptions.hpp>
 #include <graphene/chain/hardfork.hpp>
 #include <graphene/chain/internal_exceptions.hpp>
-#include <graphene/chain/special_authority.hpp>
+#include <graphene/chain/special_authority_evaluation.hpp>
 #include <graphene/chain/special_authority_object.hpp>
 #include <graphene/chain/worker_object.hpp>
 
@@ -214,7 +214,7 @@ object_id_type account_create_evaluator::do_apply( const account_create_operatio
    if( dynamic_properties.accounts_registered_this_interval % global_properties.parameters.accounts_per_fee_scale == 0
          && global_properties.parameters.account_fee_scale_bitshifts != 0 )
    {
-      d.modify(global_properties, [&dynamic_properties](global_property_object& p) {
+      d.modify(global_properties, [](global_property_object& p) {
          p.parameters.current_fees->get<account_create_operation>().basic_fee <<= p.parameters.account_fee_scale_bitshifts;
       });
    }

libraries/chain/account_object.cpp

@@ -36,10 +36,10 @@ share_type cut_fee(share_type a, uint16_t p)
    if( p == GRAPHENE_100_PERCENT )
       return a;
 
-   fc::uint128 r(a.value);
+   fc::uint128_t r = a.value;
    r *= p;
    r /= GRAPHENE_100_PERCENT;
-   return r.to_uint64();
+   return r;
 }
 
 void account_balance_object::adjust_balance(const asset& delta)

libraries/chain/affiliate_payout.cpp

@@ -62,15 +62,15 @@ namespace graphene { namespace chain {
              //ilog("Paying ${p} of ${P} for ${s} of ${r}", ("p",payout.to_uint64())("P",to_pay.value)("s",share)("r",remaining) );
              remaining -= share;
           }
-          FC_ASSERT( payout.to_uint64() <= to_pay );
+          FC_ASSERT( payout <= to_pay );
           if( payout > 0 )
           {
              if ( accumulator.find(affiliate) == accumulator.end() )
-                accumulator[affiliate] = payout.to_uint64();
+                accumulator[affiliate] = payout;
              else
-                accumulator[affiliate] += payout.to_uint64();
-             to_pay -= payout.to_uint64();
-             paid += payout.to_uint64();
+                accumulator[affiliate] += payout;
+             to_pay -= payout;
+             paid += payout;
           }
       }
       FC_ASSERT( to_pay == 0 );

libraries/chain/asset_evaluator.cpp

@@ -596,7 +596,6 @@ void_result asset_update_dividend_evaluator::do_apply( const asset_update_divide
             obj.referrer = op.issuer;
             obj.lifetime_referrer = op.issuer(db()).lifetime_referrer;
 
-            auto& params = db().get_global_properties().parameters;
             obj.network_fee_percentage = GRAPHENE_DEFAULT_NETWORK_PERCENT_OF_FEE;
             obj.lifetime_referrer_fee_percentage = GRAPHENE_DEFAULT_LIFETIME_REFERRER_PERCENT_OF_FEE;
             obj.referrer_rewards_percentage = GRAPHENE_DEFAULT_LIFETIME_REFERRER_PERCENT_OF_FEE;

libraries/chain/asset_object.cpp

@@ -38,10 +38,11 @@ share_type asset_bitasset_data_object::max_force_settlement_volume(share_type cu
    if( options.maximum_force_settlement_volume == GRAPHENE_100_PERCENT )
       return current_supply + force_settled_volume;
 
-   fc::uint128 volume = current_supply.value + force_settled_volume.value;
+   fc::uint128_t volume = current_supply.value;
+   volume += force_settled_volume.value;
    volume *= options.maximum_force_settlement_volume;
    volume /= GRAPHENE_100_PERCENT;
-   return volume.to_uint64();
+   return volume;
 }
 
 void asset_bitasset_data_object::update_median_feeds(time_point_sec current_time)

libraries/chain/balance_evaluator.cpp

@@ -22,7 +22,7 @@
  * THE SOFTWARE.
  */
 #include <graphene/chain/balance_evaluator.hpp>
-#include <graphene/chain/pts_address.hpp>
+#include <graphene/protocol/pts_address.hpp>
 
 namespace graphene { namespace chain {
 

libraries/chain/betting_market_object.cpp

@@ -75,7 +75,7 @@ namespace mpl = boost::mpl;
          amount_to_match_128 += backer_multiplier - GRAPHENE_BETTING_ODDS_PRECISION - 1;
        amount_to_match_128 /= backer_multiplier - GRAPHENE_BETTING_ODDS_PRECISION;
    }
-   return amount_to_match_128.to_uint64();
+   return amount_to_match_128;
 }
 
 share_type bet_object::get_approximate_matching_amount(bool round_up /* = false */) const

libraries/chain/block_database.cpp

@@ -22,7 +22,7 @@
  * THE SOFTWARE.
  */
 #include <graphene/chain/block_database.hpp>
-#include <graphene/chain/protocol/fee_schedule.hpp>
+#include <graphene/protocol/fee_schedule.hpp>
 #include <fc/io/raw.hpp>
 
 namespace graphene { namespace chain {

libraries/chain/buyback.cpp

@@ -22,7 +22,7 @@
  * THE SOFTWARE.
  */
 
-#include <graphene/chain/protocol/buyback.hpp>
+#include <graphene/protocol/buyback.hpp>
 #include <graphene/chain/database.hpp>
 #include <graphene/chain/exceptions.hpp>
 #include <graphene/chain/hardfork.hpp>

libraries/chain/committee_member_evaluator.cpp

@@ -25,9 +25,8 @@
 #include <graphene/chain/committee_member_object.hpp>
 #include <graphene/chain/database.hpp>
 #include <graphene/chain/account_object.hpp>
-#include <graphene/chain/hardfork.hpp>
-#include <graphene/chain/protocol/fee_schedule.hpp>
-#include <graphene/chain/protocol/vote.hpp>
+#include <graphene/protocol/fee_schedule.hpp>
+#include <graphene/protocol/vote.hpp>
 #include <graphene/chain/transaction_evaluation_state.hpp>
 
 namespace graphene { namespace chain {
@@ -42,7 +41,7 @@ object_id_type committee_member_create_evaluator::do_apply( const committee_memb
 { try {
    vote_id_type vote_id;
    db().modify(db().get_global_properties(), [&vote_id](global_property_object& p) {
-      vote_id = get_next_vote_id(p, vote_id_type::committee);
+      vote_id = vote_id_type(vote_id_type::committee, p.next_available_vote_id++);
    });
 
    const auto& new_del_object = db().create<committee_member_object>( [&]( committee_member_object& obj ){

libraries/chain/confidential_evaluator.cpp

@@ -22,7 +22,7 @@
  * THE SOFTWARE.
  */
 #include <graphene/chain/exceptions.hpp>
-#include <graphene/chain/protocol/confidential.hpp>
+#include <graphene/protocol/confidential.hpp>
 #include <graphene/chain/confidential_evaluator.hpp>
 #include <graphene/chain/confidential_object.hpp>
 #include <graphene/chain/database.hpp>