Discussion: upgrading to C++17

[dongcarl]

UPDATE: Release plan summary from April 2nd, 2020 meeting:

	C++11 Compat	C++17 Compat	Gitian/Guix Release
v0.20	✔️	if easy	C++11
v0.21	✔️	✔️	C++17
v0.22	❌	✔️	C++17

Meeting logs can be found here: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2020/bitcoin-core-dev.2020-04-02-19.01.log.html

---

There were some discussion around upgrading to C++17. It seems to be able to give us library and language features that will simplify future code. For example, for library features, our current optional.h is just a wrapper around boost::optional, and can be replaced with std::optional. Other library features that might be useful include (non-exhaustively): std::variant, the file system library, and std::shared_mutex. @elichai has kindly provided a table summarizing features useful for Bitcoin Core here: #16684 (comment).

I want to apologize for misspeaking about C++17 requiring GCC8, that is untrue, and I believe all of the features we would want will be covered by GCC7, and some by GCC6.

For some background on the cons: #13356 (comment)

---

Default GCC versions major distros (as of Aug 22nd):

	Operating System	release version	GCC version	Notes
✔️	Debian	Buster	8.3
✔️	Ubuntu	18.04 LTS	7.4
✔️	Fedora	30	9.1.1
✔️	RHEL	8	8.2
	CentOS	7	4.8.5	CentOS 8 (GCC 8.2) is being released as we speak

---

In-depth code examples of

C++17 features: https://github.com/AnthonyCalandra/modern-cpp-features/blob/master/CPP17.md
C++14 features: https://github.com/AnthonyCalandra/modern-cpp-features/blob/master/CPP14.md

---

Here are some compatibility tables from https://en.cppreference.com/w/cpp/compiler_support#cpp17

Library features:

Language features:

[practicalswift]

Some C++14 features worth mentioning as well: std::make_unique, more expressive constexpr functions allowed, return type deduction (auto foo() { … }), [[deprecated]], binary literals (0b10101110) and polymorphic lambdas ([](auto foo) { … }).

[elichai]

A table on features in C++14/17 which I think would be very useful (feel free to comment if you disagree or think other stuff should be added)

Name	Important	Nice to have	Boost Replacement
std::variant			✔️
std::make_unique		✅
Variable Templates		✅
Conditions in constexpr functions		✔️
auto in lambdas params		✅
Binary literals		✔️
std::filesystem	✔️		✔️
std::optional			✔️
[[nodiscard]]+[[fallthrough]]		✔️
de-structuring initialization		✅
shared_mutex + scoped_lock + shared_lock		✅	✔️

[elichai]

Another reason we might want to upgrade:
very powerful const expressions. both c++14 and 17 made constexpr way more powerfull.
such that we could start making a lot of functions constexpr which might maybe result in some performance increase (for functions that might be able to be evaluated at compile time)

[maflcko]

Is C++17 supported on Ubuntu Xenial, Debian 9, and the *BSD operating systems?

[ysangkok]

Debian 9 ships GCC 6.3 according to the development packages listing. GCC 6 is listed as supporting some features of C++17 experimentally.

Ubuntu Xenial ships with a default GCC 5.3, which is even older. A PPA offers newer GCC versions: ~ubuntu-toolchain-r

[laanwj]

Added 0.21 milestone optimistically

[sanjaykdragon]

I agree with moving to C++17, but when do we plan to move to the soon coming C++20?

[sipa]

@sanjaykdragon It's not that easy, as we need to remain compatible with various stable Linux distributions, which don't update their C++ compilers that frequently (or at all, after release of a particular distro version).

C++20 does not exist yet, so that seems entirely out of scope.

[sanjaykdragon]

@sanjaykdragon It's not that easy, as we need to remain compatible with various stable Linux distributions, which don't update their C++ compilers that frequently (or at all, after release of a particular distro version).

C++20 does not exist yet, so that seems entirely out of scope.

Are we just waiting for linux distributions to become compatible with C++17 for a migration?

[practicalswift]

Of the 13 build jobs we have in Travis it looks like three build jobs use compilers without C++17 support: CentOS 7, Ubuntu 16.04 (Xenial) and macOS Sierra (10.12). Sounds correct? :)

[laanwj]

Are we just waiting for linux distributions to become compatible with C++17 for a migration?

Yes, that's the main factor holding it up.

[sanjaykdragon]

Just a comment / improvement to make when we move to C++17:
use std::string_view more.

[practicalswift]

@sanjaykdragon std::string_view has really sharp edges with regards to life-time -- some would even claim it encourages use-after-free bugs.

I think our recommendation should be to try to avoid std::string_view in the general case, and use it only in places where it is both a.) needed for performance reasons (as demonstrated by benchmarks), and b.) guaranteed to be safe :)

[sanjaykdragon]

@sanjaykdragon std::string_view has really sharp edges -- some would even claim it encourages use-after-free bugs. I think our recommendation should be to avoid std::string_view in the general case :)

I'll leave this discussion until we actually start moving to C++17, but I think string_view would bring a lot of benefits if we do use it

[dongcarl]

Updated issue description with release plan summary from April 2nd, 2020 meeting.

[JeremyRubin]

On backporting:

I think the rough plan is to, as of 0.22, no longer worry about backporting features to 0.20 or below unless easy to do. Already we don't regularly backport 2 releases back, but end users may be doing their own back porting.

This means that modules like consensus can start using c++17 features more aggressively in 0.22, and mildly in 0.21 where there's a shim that can be implemented without much overhead.

An open question for 0.21 is if the backport shims can rely on other libraries; e.g., using boost option shim (which we already have in the code, but could cut with a c++17 build). For example, libconsensus 0.20 could be built with c++17 with no shims, and c++11 with shims, that might be sufficient for most users. Instead of dealing with this complex build issue, we can also wait until 0.22 to deploy boost in those contexts. The shim question is important because it points to needing a more rigorous definition of what c++11 compatibility means, but that seems to be an open question.

Personally, I think if 0.20 can be c++17 built, we can immediately start using c++17 APIs in new code for 0.21, and make 0.20 backports c++17 released without shims (by the time there is an important backport, c++17 support will only be better!), but this is probably a bit more aggressive than necessary.

[JeremyRubin]

To clarify; there is a difference between modules needed "compatibility level" without requiring linking in boost. A good rule of thumb is if in your module boost is already linked, then it would be OK to use a backport shim in it. This rules out libconsensus afaik; and is relevant to e.g. the work on Span.

I don't have a good understanding of who is using libconsensus to comment on how bad it would be to require a backport to either use ++17 or link in boost, so I'll demur on making a policy there.

[sipa]

FWIW, with #18468, it seems the codebase is entirely C++17 compilable.

[laanwj]

Moving this to the 0.22 milestone, I don't think there's anything left to do here for 0.21.

[theuni]

Noting this here so it doesn't get lost....

I'm afraid it'll be quite some time before we can safely use c++17's std::shared_mutex.

When working on another project which uses read/write locks, we ran into this nasty glibc bug: https://sourceware.org/bugzilla/show_bug.cgi?id=23861.

Specifically the problem is that totally valid uses of std::shared_mutex may result in a deadlock deep in glibc.

Unfortunately, the shared mutex path (which we can't hit currently) was broken in glibc for quite a while. It was fixed here: https://sourceware.org/git/?p=glibc.git;a=commit;h=f21e8f8ca466320fed38bdb71526c574dae98026 in 2018.

Ubuntu (for ex) only got the backported fix last week: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1864864

So until the world has moved on from this glibc bug, I'm afraid we'll need to pass on read/write locks :(

[maflcko]

We should probably determine the version of libc that has this bug fixed (2.29?) and put that with a comment in the source code?

[ysangkok]

@theuni The bugzilla link you posted is for bug 23861. But the Ubuntu bug you posted links 23844 (a different bug number). I am letting you know since I found it confusing. @MarcoFalke Bug 23844 was deferred from 2.29 according to release notes.

[ilyapopov]

@theuni This Ubuntu bug is about Ubuntu 18.04 Bionic.

Newer versions of Ubuntu, such as 20.04 LTS and 20.10 already have had the fixed version from the beginning.

[maflcko]

Closing this for now. If discussions about specific features are needed, they should go to #20419 or #19183 or whatever pull request proposes them.

[maflcko]

std::fs discussion here: #20460

[ysangkok]

@MarcoFalke How old do the patched glibc version need to be before it can be assumed present? Glibc 2.30 which contains fixes for bugs 23861 and 23844 was released in August 2019, and backports are available. The bugs only affect users running old distros without updating.

[fanquake]

The bugs only affect users running old distros without updating.

Ubuntu Bionic (18.04LTS) ships with glibc 2.27 and will remain supported until April 2023.

[ysangkok]

Bionic (18.04LTS) ships glibc 2.27

@fanquake Yes, but it received a backported fix as shown in the Launchpad link. All users have -updates enabled by default, so the only way you could miss it is if you didn't run apt upgrade since November. Does Core need to cater to that tiny minority that runs an LTS release but does not keep it updated? You can only use Bitcoin if you are internet-connected anyway, so what is the rationale for being internet-connected, but keeping your system buggy and vulnerable (other fixes could be more severe)?