Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reimplement chainbase using intrusive instead of multi_index #58

Merged
merged 67 commits into from
Nov 5, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
583b0f2
Initial reimplementation of chainbase.
swatanabe-b1 Oct 2, 2019
13be5c9
Fix indentation
swatanabe-b1 Oct 2, 2019
0358735
Add members to undo_index as required by nodeos.
swatanabe-b1 Oct 2, 2019
045f651
More updates
swatanabe-b1 Oct 3, 2019
67a3387
Make the value_type correct.
swatanabe-b1 Oct 3, 2019
82ecb49
Track down and fix the bug that was causing the nodeos unittests to f…
swatanabe-b1 Oct 3, 2019
c8d6aa2
Fix the tests and reduce rebalancing in modify.
swatanabe-b1 Oct 3, 2019
4969f60
Use a custom node_type, to avoid the overhead of using offset_ptr for…
swatanabe-b1 Oct 3, 2019
9c85067
Use node_allocator.
swatanabe-b1 Oct 3, 2019
0df2648
Add tests and fix memory leak.
swatanabe-b1 Oct 4, 2019
ae0b82f
Fix memory leak
swatanabe-b1 Oct 4, 2019
00b7958
Switch to AVL tree.
swatanabe-b1 Oct 4, 2019
7a94865
Put the undo stack in a separate pool from the main container.
swatanabe-b1 Oct 4, 2019
83a78d9
Use separate node pools for each container.
swatanabe-b1 Oct 4, 2019
21e30be
Use node_allocator in chainbase_tests.
swatanabe-b1 Oct 4, 2019
865dfea
Implement cow for shared_blob
swatanabe-b1 Oct 7, 2019
605368f
An id can never be added again after it is removed.
swatanabe-b1 Oct 7, 2019
dc95d88
Reduce binary trees in the undo stack.
swatanabe-b1 Oct 8, 2019
57350d6
Prune some dead code
swatanabe-b1 Oct 8, 2019
5967854
avl_set seems to be slightly faster, and makes a fairly significant i…
swatanabe-b1 Oct 8, 2019
eaa42dc
Use optimized insertion functions.
swatanabe-b1 Oct 8, 2019
445df32
Use avltree and temporarily allow non-unique elements in undo.
swatanabe-b1 Oct 9, 2019
c26f2d5
Fix regular replay with replay opts.
swatanabe-b1 Oct 9, 2019
9015362
Fix issues from valgrind
swatanabe-b1 Oct 9, 2019
514cc77
Mostly eliminate new_ids.
swatanabe-b1 Oct 10, 2019
39fac1d
Simplify tracking of mtime/erased items
swatanabe-b1 Oct 10, 2019
2979834
Move deduplication from squash to undo.
swatanabe-b1 Oct 10, 2019
b45b09b
constant time squash.
swatanabe-b1 Oct 10, 2019
16d978e
Make undo_index fully exception safe.
swatanabe-b1 Oct 10, 2019
28a6730
Cleanup dead code.
swatanabe-b1 Oct 10, 2019
1aebc8e
Implement the ability to prune duplicate modifies from the undo stack.
swatanabe-b1 Oct 15, 2019
744bf1f
Also prune new/removed items.
swatanabe-b1 Oct 15, 2019
439fc98
session::revision is unused and doesn't work. Remove it.
swatanabe-b1 Oct 15, 2019
51c566f
Get the deallocation size correct
swatanabe-b1 Oct 15, 2019
0f7ba4d
Don't accidentally wipe out the entire undo stack.
swatanabe-b1 Oct 15, 2019
4211bcb
Support for state_history_plugin.
swatanabe-b1 Oct 15, 2019
40e59c9
Compress the undo stack on squash by default.
swatanabe-b1 Oct 16, 2019
ddf53db
Port validate from generic_index.
swatanabe-b1 Oct 16, 2019
8a3bebf
Remove the original generic_index
swatanabe-b1 Oct 18, 2019
6558c13
Split the node allocator and COW string into separate files.
swatanabe-b1 Oct 25, 2019
4f42acf
Add comment describing the representation.
swatanabe-b1 Oct 25, 2019
c3153ff
We need C++17
swatanabe-b1 Oct 25, 2019
861d351
Add range checking to match std::basic_string.
swatanabe-b1 Oct 25, 2019
4095f0c
Minor cleanup to address review feedback.
swatanabe-b1 Oct 25, 2019
94daf5d
Delete copy operations of scope_exit
swatanabe-b1 Oct 29, 2019
9d6a5c4
Add missing dec_refcount.
swatanabe-b1 Oct 29, 2019
565513f
Use std::allocator_traits.
swatanabe-b1 Oct 29, 2019
e560f50
Verify size and alignment of list item.
swatanabe-b1 Oct 29, 2019
f1885f3
Update C++ version.
swatanabe-b1 Oct 29, 2019
77897d5
chainbase::undo_state is dead.
swatanabe-b1 Oct 29, 2019
41ad213
Rename template parameters to be more clear.
swatanabe-b1 Oct 29, 2019
03b61ca
Fix project for end iterators.
swatanabe-b1 Oct 29, 2019
1cbcb36
Remove unused overload of find.
swatanabe-b1 Oct 29, 2019
c11992f
Add int overload of project.
swatanabe-b1 Oct 29, 2019
541f8ef
More renaming.
swatanabe-b1 Oct 29, 2019
bcfaabc
Remove unnecessary scope_exit.
swatanabe-b1 Oct 29, 2019
258c6f2
Avoid memcpy/memset when it's just going to be overwritten later.
swatanabe-b1 Oct 29, 2019
37b54ce
Remove c_str. We should never be depending on null-termination.
swatanabe-b1 Oct 29, 2019
dcea84e
Verify requirements on indices.
swatanabe-b1 Oct 31, 2019
6fc3c7b
Require ordered_unique only
swatanabe-b1 Nov 1, 2019
42c5cfd
Fix memory leak.
swatanabe-b1 Nov 1, 2019
ab29a23
Avoid dereferencing end iterators.
swatanabe-b1 Nov 1, 2019
568328a
Renaming for consistency
swatanabe-b1 Nov 1, 2019
3ea72d9
Forwarding.
swatanabe-b1 Nov 1, 2019
874b8e3
Handle committing nothing.
swatanabe-b1 Nov 1, 2019
6524d21
Check that the id does not change.
swatanabe-b1 Nov 1, 2019
5471547
shared_vector is unused.
swatanabe-b1 Nov 1, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ FIND_PACKAGE(Boost 1.57 REQUIRED COMPONENTS filesystem unit_test_framework)
SET(PLATFORM_LIBRARIES)

if(CMAKE_CXX_STANDARD EQUAL 98)
message(FATAL_ERROR "chainbase requires c++14 or newer")
message(FATAL_ERROR "chainbase requires c++17 or newer")
swatanabe-b1 marked this conversation as resolved.
Show resolved Hide resolved
elseif(NOT CMAKE_CXX_STANDARD)
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
endif()

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

## Dependencies

- C++14
- C++17
- [Boost](http://www.boost.org/)
- CMake Build Process
- Supports Linux, Mac OS X (no Windows Support)
Expand Down
466 changes: 11 additions & 455 deletions include/chainbase/chainbase.hpp

Large diffs are not rendered by default.

65 changes: 65 additions & 0 deletions include/chainbase/chainbase_node_allocator.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
#pragma once

#include <cstddef>
#include <boost/interprocess/offset_ptr.hpp>

#include <chainbase/pinnable_mapped_file.hpp>

namespace chainbase {

namespace bip = boost::interprocess;

template<typename T, typename S>
class chainbase_node_allocator {
public:
using value_type = T;
using pointer = bip::offset_ptr<T>;
using segment_manager = pinnable_mapped_file::segment_manager;
chainbase_node_allocator(segment_manager* manager) : _manager{manager} {}
chainbase_node_allocator(const chainbase_node_allocator& other) : _manager(other._manager) {}
template<typename U>
chainbase_node_allocator(const chainbase_node_allocator<U, S>& other) : _manager(other._manager) {}
pointer allocate(std::size_t num) {
if (num == 1) {
if (_freelist == nullptr) {
get_some();
}
list_item* result = &*_freelist;
_freelist = _freelist->_next;
result->~list_item();
return pointer{(T*)result};
} else {
return pointer{(T*)_manager->allocate(num*sizeof(T))};
}
}
void deallocate(const pointer& p, std::size_t num) {
if (num == 1) {
_freelist = new (&*p) list_item{_freelist};
swatanabe-b1 marked this conversation as resolved.
Show resolved Hide resolved
} else {
_manager->deallocate(&*p);
}
}
bool operator==(const chainbase_node_allocator& other) const { return this == &other; }
bool operator!=(const chainbase_node_allocator& other) const { return this != &other; }
segment_manager* get_segment_manager() const { return _manager.get(); }
private:
template<typename T2, typename S2>
friend class chainbase_node_allocator;
void get_some() {
static_assert(sizeof(T) >= sizeof(list_item), "Too small for free list");
static_assert(sizeof(T) % alignof(list_item) == 0, "Bad alignment for free list");
char* result = (char*)_manager->allocate(sizeof(T) * 64);
_freelist = bip::offset_ptr<list_item>{(list_item*)result};
for(int i = 0; i < 63; ++i) {
char* next = result + sizeof(T);
new(result) list_item{bip::offset_ptr<list_item>{(list_item*)next}};
result = next;
}
new(result) list_item{nullptr};
}
struct list_item { bip::offset_ptr<list_item> _next; };
bip::offset_ptr<pinnable_mapped_file::segment_manager> _manager;
bip::offset_ptr<list_item> _freelist{};
};

} // namepsace chainbase
143 changes: 143 additions & 0 deletions include/chainbase/shared_cow_string.hpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
#pragma once

#include <boost/container/container_fwd.hpp>
#include <boost/interprocess/offset_ptr.hpp>
#include <boost/interprocess/allocators/allocator.hpp>

#include <cstddef>
#include <cstring>
#include <algorithm>
#include <string>

#include <chainbase/pinnable_mapped_file.hpp>

namespace chainbase {

namespace bip = boost::interprocess;

class shared_cow_string {
struct impl {
uint32_t reference_count;
uint32_t size;
char data[0];
};
public:
using allocator_type = bip::allocator<char, pinnable_mapped_file::segment_manager>;
using iterator = const char*;
using const_iterator = const char*;
explicit shared_cow_string(const allocator_type& alloc) : _data(nullptr), _alloc(alloc) {}
template<typename Iter>
explicit shared_cow_string(Iter begin, Iter end, const allocator_type& alloc) : shared_cow_string(alloc) {
std::size_t size = std::distance(begin, end);
impl* new_data = (impl*)&*_alloc.allocate(sizeof(impl) + size + 1);
new_data->reference_count = 1;
new_data->size = size;
std::copy(begin, end, new_data->data);
new_data->data[size] = '\0';
_data = new_data;
}
explicit shared_cow_string(const char* ptr, std::size_t size, const allocator_type& alloc) : shared_cow_string(alloc) {
impl* new_data = (impl*)&*_alloc.allocate(sizeof(impl) + size + 1);
new_data->reference_count = 1;
new_data->size = size;
std::memcpy(new_data->data, ptr, size);
new_data->data[size] = '\0';
_data = new_data;
}
explicit shared_cow_string(std::size_t size, boost::container::default_init_t, const allocator_type& alloc) : shared_cow_string(alloc) {
impl* new_data = (impl*)&*_alloc.allocate(sizeof(impl) + size + 1);
new_data->reference_count = 1;
new_data->size = size;
new_data->data[size] = '\0';
_data = new_data;
}
shared_cow_string(const shared_cow_string& other) : _data(other._data), _alloc(other._alloc) {
if(_data != nullptr) {
++_data->reference_count;
}
}
shared_cow_string(shared_cow_string&& other) : _data(other._data), _alloc(other._alloc) {
other._data = nullptr;
}
shared_cow_string& operator=(const shared_cow_string& other) {
*this = shared_cow_string{other};
return *this;
}
shared_cow_string& operator=(shared_cow_string&& other) {
if (this != &other) {
dec_refcount();
_data = other._data;
swatanabe-b1 marked this conversation as resolved.
Show resolved Hide resolved
other._data = nullptr;
}
return *this;
}
~shared_cow_string() {
dec_refcount();
}
void resize(std::size_t new_size, boost::container::default_init_t) {
impl* new_data = (impl*)&*_alloc.allocate(sizeof(impl) + new_size + 1);
new_data->reference_count = 1;
new_data->size = new_size;
new_data->data[new_size] = '\0';
dec_refcount();
_data = new_data;
}
template<typename F>
void resize_and_fill(std::size_t new_size, F&& f) {
resize(new_size, boost::container::default_init);
static_cast<F&&>(f)(_data->data, new_size);
}
void assign(const char* ptr, std::size_t size) {
impl* new_data = (impl*)&*_alloc.allocate(sizeof(impl) + size + 1);
new_data->reference_count = 1;
new_data->size = size;
if(size)
std::memcpy(new_data->data, ptr, size);
new_data->data[size] = '\0';
dec_refcount();
_data = new_data;
}
void assign(const unsigned char* ptr, std::size_t size) {
assign((char*)ptr, size);
}
const char * data() const {
if (_data) return _data->data;
else return nullptr;
}
std::size_t size() const {
if (_data) return _data->size;
else return 0;
}
const_iterator begin() const { return data(); }
const_iterator end() const {
if (_data) return _data->data + _data->size;
else return nullptr;
}
int compare(std::size_t start, std::size_t count, const char* other, std::size_t other_size) const {
std::size_t sz = size();
if(start > sz) BOOST_THROW_EXCEPTION(std::out_of_range{"shared_cow_string::compare"});
count = std::min(count, sz - start);
std::size_t cmp_len = std::min(count, other_size);
const char* start_ptr = data() + start;
int result = std::char_traits<char>::compare(start_ptr, other, cmp_len);
if (result != 0) return result;
else if (count < other_size) return -1;
else if(count > other_size) return 1;
else return 0;
}
bool operator==(const shared_cow_string& rhs) const {
return size() == rhs.size() && std::memcmp(data(), rhs.data(), size()) == 0;
}
bool operator!=(const shared_cow_string& rhs) const { return !(*this == rhs); }
const allocator_type& get_allocator() const { return _alloc; }
private:
void dec_refcount() {
if(_data && --_data->reference_count == 0) {
_alloc.deallocate((char*)&*_data, sizeof(shared_cow_string) + _data->size + 1);
}
}
bip::offset_ptr<impl> _data;
allocator_type _alloc;
};

} // namepsace chainbase
Loading