Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

optimize compose by looking at previous commit and using rpmdb checksum #24

Closed
cgwalters opened this issue Aug 6, 2014 · 3 comments
Closed

optimize compose by looking at previous commit and using rpmdb checksum #24

cgwalters opened this issue Aug 6, 2014 · 3 comments

Comments

@cgwalters
Copy link
Member

We want the composes to be fast for many obvious reasons.

See related: https://bugzilla.gnome.org/show_bug.cgi?id=707827

James has an idea here for rpm-ostree.
@james-antill
Copy link
Contributor

the idea is basically to run a "hardlink" like thing over the new commit and the previous data and rely on --link-checkout-speedup ... but to actually run a "hardlink" like thing would require a lot of IO. But we can cheat when the new commit and the old commit have identical (bit for bit) rpms installed ... for a huge amount of rpm content we can just auto hardlink by comparing path names here.

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue May 20, 2015
@cgwalters
[WIP] Add a package object cache
180641c 
This commit implements coreos#24

Needs to be cleaned up, and isn't quite as much of a speedup as I
thought, because unfortunately the "lookup package owner of file" is
actually a non-indexed (i.e. loop over all pkgs) operation in libsolv
=( =( =(

We might be able to fix this by using librpm directly which from my
current reading of the source code has some caching for this.
@cgwalters cgwalters mentioned this issue May 20, 2015
Closed
@cgwalters
Copy link
Member Author

Depends on ostreedev/ostree#114

cgwalters added a commit to cgwalters/rpm-ostree that referenced this issue May 22, 2015
@cgwalters
[WIP] Add a package object cache
1c350fe 
This commit implements coreos#24

Now using librpm.  This could still be faster.
@cgwalters
Copy link
Member Author

This will be done by the unprivileged work.

@cgwalters cgwalters mentioned this issue Sep 21, 2017
Closed
cgwalters referenced this issue in cgwalters/rpm-ostree Feb 2, 2021
@cgwalters
scripts: Fix a stack use-after-free
ebb0a40 
I think this changed in a recent refactoring; basically since
we're passing this stack-allocated value to the child spawn
function we need to keep it alive.  This of course would
have been caught by Rust...

```
==672376==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffc290d9440 at pc 0x55c88c318946 bp 0x7ffc290d8b10 sp 0x7ffc290d8b08
    #0 0x55c88c318945 in script_child_setup src/libpriv/rpmostree-scripts.cxx:272
    #1 0x7f92089da902  (/lib64/libglib-2.0.so.0+0x9f902)
    #2 0x7f92089de20f  (/lib64/libglib-2.0.so.0+0xa320f)
    #3 0x7f92089de52e  (/lib64/libglib-2.0.so.0+0xa352e)
    #4 0x7f92089def02 in g_spawn_async_with_pipes (/lib64/libglib-2.0.so.0+0xa3f02)
    #5 0x7f9208b7445f  (/lib64/libgio-2.0.so.0+0xab45f)
    #6 0x7f9208b736d8 in g_subprocess_launcher_spawnv (/lib64/libgio-2.0.so.0+0xaa6d8)
    #7 0x55c88c3831b9 in rpmostree_bwrap_execute src/libpriv/rpmostree-bwrap.cxx:504
    #8 0x55c88c3836df in rpmostree_bwrap_run_captured src/libpriv/rpmostree-bwrap.cxx:450
    #9 0x55c88c31b5f1 in rpmostree_run_script_in_bwrap_container src/libpriv/rpmostree-scripts.cxx:469
    #10 0x55c88c31ca9d in impl_run_rpm_script src/libpriv/rpmostree-scripts.cxx:588
    #11 0x55c88c31d22b in run_script src/libpriv/rpmostree-scripts.cxx:637
    #12 0x55c88c31d22b in rpmostree_script_run_sync src/libpriv/rpmostree-scripts.cxx:778
    #13 0x55c88c2ef830 in run_script_sync src/libpriv/rpmostree-core.cxx:3661
    #14 0x55c88c30afa6 in rpmostree_context_assemble src/libpriv/rpmostree-core.cxx:4422
    #15 0x55c88c34a9af in install_packages src/app/rpmostree-compose-builtin-tree.cxx:451
    #16 0x55c88c34c174 in impl_install_tree src/app/rpmostree-compose-builtin-tree.cxx:925
    #17 0x55c88c350f84 in rpmostree_compose_builtin_tree src/app/rpmostree-compose-builtin-tree.cxx:1421
    #18 0x55c88c276ec8 in rpmostree_handle_subcommand src/app/libmain.cxx:405
    #19 0x55c88c27827c in rpmostree_main_inner src/app/libmain.cxx:521
    #20 0x55c88c27827c in rpmostreecxx::rpmostree_main(rust::cxxbridge1::Slice<rust::cxxbridge1::Str const>) src/app/libmain.cxx:546
    #21 0x55c88c271c25 in operator() /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:1257
    #22 0x55c88c271c25 in trycatch<rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>)::<lambda()>, rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>)::<lambda(char const*)> > /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:997
    #23 0x55c88c271c25 in rpmostreecxx$cxxbridge1$rpmostree_main /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:1255
    #24 0x55c88c0468f7 in rpmostree_rust::ffi::rpmostree_main::hfedda48c684245ce rust/src/lib.rs:25
    #25 0x55c88c0468f7 in rpm_ostree::inner_main::hf078b99ca4b270aa rust/src/main.rs:9
    #26 0x55c88c0468f7 in rpm_ostree::main::hc0ca527cfaa3f556 rust/src/main.rs:28
    #27 0x55c88c046b22 in core::ops::function::FnOnce::call_once::h8567110dac55274e /var/home/walters/.rustup/toolchains/1.48-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
    #28 0x55c88c046b22 in std::sys_common::backtrace::__rust_begin_short_backtrace::h1c67f2f52d05cfa0 /var/home/walters/.rustup/toolchains/1.48-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137
    #29 0x55c88c045fd7 in main (/usr/bin/rpm-ostree+0xc9fd7)
    #30 0x7f92076091e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    #31 0x55c88c045b9d in _start (/usr/bin/rpm-ostree+0xc9b9d)

Address 0x7ffc290d9440 is located in stack of thread T0 at offset 272 in frame
    #0 0x55c88c31a1af in rpmostree_run_script_in_bwrap_container src/libpriv/rpmostree-scripts.cxx:349
```
cgwalters referenced this issue in cgwalters/rpm-ostree Feb 3, 2021
@cgwalters
scripts: Fix a stack use-after-free
184d641 
I think this changed in a recent refactoring; basically since
we're passing this stack-allocated value to the child spawn
function we need to keep it alive.  This of course would
have been caught by Rust...

```
==672376==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffc290d9440 at pc 0x55c88c318946 bp 0x7ffc290d8b10 sp 0x7ffc290d8b08
    #0 0x55c88c318945 in script_child_setup src/libpriv/rpmostree-scripts.cxx:272
    #1 0x7f92089da902  (/lib64/libglib-2.0.so.0+0x9f902)
    #2 0x7f92089de20f  (/lib64/libglib-2.0.so.0+0xa320f)
    #3 0x7f92089de52e  (/lib64/libglib-2.0.so.0+0xa352e)
    #4 0x7f92089def02 in g_spawn_async_with_pipes (/lib64/libglib-2.0.so.0+0xa3f02)
    #5 0x7f9208b7445f  (/lib64/libgio-2.0.so.0+0xab45f)
    #6 0x7f9208b736d8 in g_subprocess_launcher_spawnv (/lib64/libgio-2.0.so.0+0xaa6d8)
    #7 0x55c88c3831b9 in rpmostree_bwrap_execute src/libpriv/rpmostree-bwrap.cxx:504
    #8 0x55c88c3836df in rpmostree_bwrap_run_captured src/libpriv/rpmostree-bwrap.cxx:450
    #9 0x55c88c31b5f1 in rpmostree_run_script_in_bwrap_container src/libpriv/rpmostree-scripts.cxx:469
    #10 0x55c88c31ca9d in impl_run_rpm_script src/libpriv/rpmostree-scripts.cxx:588
    #11 0x55c88c31d22b in run_script src/libpriv/rpmostree-scripts.cxx:637
    #12 0x55c88c31d22b in rpmostree_script_run_sync src/libpriv/rpmostree-scripts.cxx:778
    #13 0x55c88c2ef830 in run_script_sync src/libpriv/rpmostree-core.cxx:3661
    #14 0x55c88c30afa6 in rpmostree_context_assemble src/libpriv/rpmostree-core.cxx:4422
    #15 0x55c88c34a9af in install_packages src/app/rpmostree-compose-builtin-tree.cxx:451
    #16 0x55c88c34c174 in impl_install_tree src/app/rpmostree-compose-builtin-tree.cxx:925
    #17 0x55c88c350f84 in rpmostree_compose_builtin_tree src/app/rpmostree-compose-builtin-tree.cxx:1421
    #18 0x55c88c276ec8 in rpmostree_handle_subcommand src/app/libmain.cxx:405
    #19 0x55c88c27827c in rpmostree_main_inner src/app/libmain.cxx:521
    #20 0x55c88c27827c in rpmostreecxx::rpmostree_main(rust::cxxbridge1::Slice<rust::cxxbridge1::Str const>) src/app/libmain.cxx:546
    #21 0x55c88c271c25 in operator() /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:1257
    #22 0x55c88c271c25 in trycatch<rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>)::<lambda()>, rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>)::<lambda(char const*)> > /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:997
    #23 0x55c88c271c25 in rpmostreecxx$cxxbridge1$rpmostree_main /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:1255
    #24 0x55c88c0468f7 in rpmostree_rust::ffi::rpmostree_main::hfedda48c684245ce rust/src/lib.rs:25
    #25 0x55c88c0468f7 in rpm_ostree::inner_main::hf078b99ca4b270aa rust/src/main.rs:9
    #26 0x55c88c0468f7 in rpm_ostree::main::hc0ca527cfaa3f556 rust/src/main.rs:28
    #27 0x55c88c046b22 in core::ops::function::FnOnce::call_once::h8567110dac55274e /var/home/walters/.rustup/toolchains/1.48-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
    #28 0x55c88c046b22 in std::sys_common::backtrace::__rust_begin_short_backtrace::h1c67f2f52d05cfa0 /var/home/walters/.rustup/toolchains/1.48-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137
    #29 0x55c88c045fd7 in main (/usr/bin/rpm-ostree+0xc9fd7)
    #30 0x7f92076091e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    #31 0x55c88c045b9d in _start (/usr/bin/rpm-ostree+0xc9b9d)

Address 0x7ffc290d9440 is located in stack of thread T0 at offset 272 in frame
    #0 0x55c88c31a1af in rpmostree_run_script_in_bwrap_container src/libpriv/rpmostree-scripts.cxx:349
```
openshift-merge-robot pushed a commit that referenced this issue Feb 3, 2021
@cgwalters @openshift-merge-robot
scripts: Fix a stack use-after-free
d616f73 
I think this changed in a recent refactoring; basically since
we're passing this stack-allocated value to the child spawn
function we need to keep it alive.  This of course would
have been caught by Rust...

```
==672376==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffc290d9440 at pc 0x55c88c318946 bp 0x7ffc290d8b10 sp 0x7ffc290d8b08
    #0 0x55c88c318945 in script_child_setup src/libpriv/rpmostree-scripts.cxx:272
    #1 0x7f92089da902  (/lib64/libglib-2.0.so.0+0x9f902)
    #2 0x7f92089de20f  (/lib64/libglib-2.0.so.0+0xa320f)
    #3 0x7f92089de52e  (/lib64/libglib-2.0.so.0+0xa352e)
    #4 0x7f92089def02 in g_spawn_async_with_pipes (/lib64/libglib-2.0.so.0+0xa3f02)
    #5 0x7f9208b7445f  (/lib64/libgio-2.0.so.0+0xab45f)
    #6 0x7f9208b736d8 in g_subprocess_launcher_spawnv (/lib64/libgio-2.0.so.0+0xaa6d8)
    #7 0x55c88c3831b9 in rpmostree_bwrap_execute src/libpriv/rpmostree-bwrap.cxx:504
    #8 0x55c88c3836df in rpmostree_bwrap_run_captured src/libpriv/rpmostree-bwrap.cxx:450
    #9 0x55c88c31b5f1 in rpmostree_run_script_in_bwrap_container src/libpriv/rpmostree-scripts.cxx:469
    #10 0x55c88c31ca9d in impl_run_rpm_script src/libpriv/rpmostree-scripts.cxx:588
    #11 0x55c88c31d22b in run_script src/libpriv/rpmostree-scripts.cxx:637
    #12 0x55c88c31d22b in rpmostree_script_run_sync src/libpriv/rpmostree-scripts.cxx:778
    #13 0x55c88c2ef830 in run_script_sync src/libpriv/rpmostree-core.cxx:3661
    #14 0x55c88c30afa6 in rpmostree_context_assemble src/libpriv/rpmostree-core.cxx:4422
    #15 0x55c88c34a9af in install_packages src/app/rpmostree-compose-builtin-tree.cxx:451
    #16 0x55c88c34c174 in impl_install_tree src/app/rpmostree-compose-builtin-tree.cxx:925
    #17 0x55c88c350f84 in rpmostree_compose_builtin_tree src/app/rpmostree-compose-builtin-tree.cxx:1421
    #18 0x55c88c276ec8 in rpmostree_handle_subcommand src/app/libmain.cxx:405
    #19 0x55c88c27827c in rpmostree_main_inner src/app/libmain.cxx:521
    #20 0x55c88c27827c in rpmostreecxx::rpmostree_main(rust::cxxbridge1::Slice<rust::cxxbridge1::Str const>) src/app/libmain.cxx:546
    #21 0x55c88c271c25 in operator() /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:1257
    #22 0x55c88c271c25 in trycatch<rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>)::<lambda()>, rpmostreecxx::rpmostreecxx$cxxbridge1$rpmostree_main(rust::cxxbridge1::Slice<const rust::cxxbridge1::Str>)::<lambda(char const*)> > /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:997
    #23 0x55c88c271c25 in rpmostreecxx$cxxbridge1$rpmostree_main /var/srv/walters/src/github/coreos/rpm-ostree/rpmostree-cxxrs.cxx:1255
    #24 0x55c88c0468f7 in rpmostree_rust::ffi::rpmostree_main::hfedda48c684245ce rust/src/lib.rs:25
    #25 0x55c88c0468f7 in rpm_ostree::inner_main::hf078b99ca4b270aa rust/src/main.rs:9
    #26 0x55c88c0468f7 in rpm_ostree::main::hc0ca527cfaa3f556 rust/src/main.rs:28
    #27 0x55c88c046b22 in core::ops::function::FnOnce::call_once::h8567110dac55274e /var/home/walters/.rustup/toolchains/1.48-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227
    #28 0x55c88c046b22 in std::sys_common::backtrace::__rust_begin_short_backtrace::h1c67f2f52d05cfa0 /var/home/walters/.rustup/toolchains/1.48-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137
    #29 0x55c88c045fd7 in main (/usr/bin/rpm-ostree+0xc9fd7)
    #30 0x7f92076091e1 in __libc_start_main (/lib64/libc.so.6+0x281e1)
    #31 0x55c88c045b9d in _start (/usr/bin/rpm-ostree+0xc9b9d)

Address 0x7ffc290d9440 is located in stack of thread T0 at offset 272 in frame
    #0 0x55c88c31a1af in rpmostree_run_script_in_bwrap_container src/libpriv/rpmostree-scripts.cxx:349
```
cgwalters referenced this issue in cgwalters/rpm-ostree Feb 6, 2021
@cgwalters
WIP: Support for sanitizers
ef526d9 
This works fine when composing, but for some reason the client
side apparently infinite loops inside libsolv:

```
(lldb) thread backtrace
* thread #4, name = 'pool-/usr/bin/r'
  * frame #0: 0x00007fd61b97200f libc.so.6`__memcpy_sse2_unaligned_erms + 623
    frame #1: 0x00007fd61cbc88e6 libasan.so.6`__asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) + 214
    frame #2: 0x00007fd61cc4b725 libasan.so.6`__interceptor_realloc + 245
    frame #3: 0x00007fd61baec43e libsolv.so.1`solv_realloc + 30
    frame #4: 0x00007fd61baf0414 libsolv.so.1`repodata_add_dirstr + 276
    frame #5: 0x00007fd61bb6f755 libsolvext.so.1`end_element + 53
    frame #6: 0x00007fd61b05855d libxml2.so.2`xmlParseEndTag1.constprop.0 + 317
    frame #7: 0x00007fd61b063548 libxml2.so.2`xmlParseTryOrFinish.isra.0 + 888
    frame #8: 0x00007fd61af7ed20 libxml2.so.2`xmlParseChunk + 560
    frame #9: 0x00007fd61bb727e7 libsolvext.so.1`solv_xmlparser_parse + 183
    frame #10: 0x00007fd61bb5ea0e libsolvext.so.1`repo_add_rpmmd + 254
    frame #11: 0x000055a4fce7a5f5 rpm-ostree`::load_filelists_cb(repo=<unavailable>, fp=<unavailable>) at dnf-sack.cpp:444:23
    frame #12: 0x000055a4fce7cad6 rpm-ostree`load_ext(_DnfSack*, libdnf::Repo*, _hy_repo_repodata, char const*, char const*, int (*)(s_Repo*, _IO_FILE*), _GError**) at dnf-sack.cpp:430:13
    frame #13: 0x000055a4fce7df60 rpm-ostree`dnf_sack_load_repo at dnf-sack.cpp:1789:26
    frame #14: 0x000055a4fce7eee9 rpm-ostree`dnf_sack_add_repo at dnf-sack.cpp:2217:28
    frame #15: 0x000055a4fce7f0fb rpm-ostree`dnf_sack_add_repos at dnf-sack.cpp:2271:32
    frame #16: 0x000055a4fce870ee rpm-ostree`dnf_context_setup_sack_with_flags at dnf-context.cpp:1796:29
    frame #17: 0x000055a4fcdf757f rpm-ostree`rpmostree_context_download_metadata at rpmostree-core.cxx:1206:44
    frame #18: 0x000055a4fcdf95c3 rpm-ostree`rpmostree_context_prepare at rpmostree-core.cxx:2001:48
    frame #19: 0x000055a4fce54ab7 rpm-ostree`rpmostree_sysroot_upgrader_prep_layering at rpmostree-sysroot-upgrader.cxx:1018:38
    frame #20: 0x000055a4fcdcb143 rpm-ostree`deploy_transaction_execute(_RpmostreedTransaction*, _GCancellable*, _GError**) at rpmostreed-transaction-types.cxx:1445:49
    frame #21: 0x000055a4fcdba4cd rpm-ostree`transaction_execute_thread(_GTask*, void*, void*, _GCancellable*) at rpmostreed-transaction.cxx:340:34
    frame #22: 0x00007fd61c58f7e2 libgio-2.0.so.0`g_task_thread_pool_thread + 114
    frame #23: 0x00007fd61c3d7e54 libglib-2.0.so.0`g_thread_pool_thread_proxy.lto_priv.0 + 116
    frame #24: 0x00007fd61c3d52b2 libglib-2.0.so.0`g_thread_proxy + 82
    frame #25: 0x00007fd61b8af3f9 libpthread.so.0`start_thread + 233
    frame #26: 0x00007fd61b9c9903 libc.so.6`__clone + 67
(lldb)
```
cgwalters referenced this issue in cgwalters/rpm-ostree Feb 6, 2021
@cgwalters
WIP: Support for sanitizers
0216288 
This works fine when composing, but for some reason the client
side apparently infinite loops inside libsolv:

```
(lldb) thread backtrace
* thread #4, name = 'pool-/usr/bin/r'
  * frame #0: 0x00007fd61b97200f libc.so.6`__memcpy_sse2_unaligned_erms + 623
    frame #1: 0x00007fd61cbc88e6 libasan.so.6`__asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) + 214
    frame #2: 0x00007fd61cc4b725 libasan.so.6`__interceptor_realloc + 245
    frame #3: 0x00007fd61baec43e libsolv.so.1`solv_realloc + 30
    frame #4: 0x00007fd61baf0414 libsolv.so.1`repodata_add_dirstr + 276
    frame #5: 0x00007fd61bb6f755 libsolvext.so.1`end_element + 53
    frame #6: 0x00007fd61b05855d libxml2.so.2`xmlParseEndTag1.constprop.0 + 317
    frame #7: 0x00007fd61b063548 libxml2.so.2`xmlParseTryOrFinish.isra.0 + 888
    frame #8: 0x00007fd61af7ed20 libxml2.so.2`xmlParseChunk + 560
    frame #9: 0x00007fd61bb727e7 libsolvext.so.1`solv_xmlparser_parse + 183
    frame #10: 0x00007fd61bb5ea0e libsolvext.so.1`repo_add_rpmmd + 254
    frame #11: 0x000055a4fce7a5f5 rpm-ostree`::load_filelists_cb(repo=<unavailable>, fp=<unavailable>) at dnf-sack.cpp:444:23
    frame #12: 0x000055a4fce7cad6 rpm-ostree`load_ext(_DnfSack*, libdnf::Repo*, _hy_repo_repodata, char const*, char const*, int (*)(s_Repo*, _IO_FILE*), _GError**) at dnf-sack.cpp:430:13
    frame #13: 0x000055a4fce7df60 rpm-ostree`dnf_sack_load_repo at dnf-sack.cpp:1789:26
    frame #14: 0x000055a4fce7eee9 rpm-ostree`dnf_sack_add_repo at dnf-sack.cpp:2217:28
    frame #15: 0x000055a4fce7f0fb rpm-ostree`dnf_sack_add_repos at dnf-sack.cpp:2271:32
    frame #16: 0x000055a4fce870ee rpm-ostree`dnf_context_setup_sack_with_flags at dnf-context.cpp:1796:29
    frame #17: 0x000055a4fcdf757f rpm-ostree`rpmostree_context_download_metadata at rpmostree-core.cxx:1206:44
    frame #18: 0x000055a4fcdf95c3 rpm-ostree`rpmostree_context_prepare at rpmostree-core.cxx:2001:48
    frame #19: 0x000055a4fce54ab7 rpm-ostree`rpmostree_sysroot_upgrader_prep_layering at rpmostree-sysroot-upgrader.cxx:1018:38
    frame #20: 0x000055a4fcdcb143 rpm-ostree`deploy_transaction_execute(_RpmostreedTransaction*, _GCancellable*, _GError**) at rpmostreed-transaction-types.cxx:1445:49
    frame #21: 0x000055a4fcdba4cd rpm-ostree`transaction_execute_thread(_GTask*, void*, void*, _GCancellable*) at rpmostreed-transaction.cxx:340:34
    frame #22: 0x00007fd61c58f7e2 libgio-2.0.so.0`g_task_thread_pool_thread + 114
    frame #23: 0x00007fd61c3d7e54 libglib-2.0.so.0`g_thread_pool_thread_proxy.lto_priv.0 + 116
    frame #24: 0x00007fd61c3d52b2 libglib-2.0.so.0`g_thread_proxy + 82
    frame #25: 0x00007fd61b8af3f9 libpthread.so.0`start_thread + 233
    frame #26: 0x00007fd61b9c9903 libc.so.6`__clone + 67
(lldb)
```
cgwalters referenced this issue in cgwalters/rpm-ostree Feb 8, 2021
@cgwalters
Add --enable-sanitizers (not on by default yet)
06b2cd6 
This way we at least get unit test coverage (which...
our unit test coverage doesn't do much because our
main code paths require privileges or virt).

One main blocker to this is that rustc doesn't expose
first-class support for this yet:
rust-lang/rust#39699

At a practical level this works when building in release
mode but fails with `cargo test` for some reason; linker
arguments being pruned?  Not sure.

So I was able to use this when composing to find a bug,
but then for some other reason the client
side apparently infinite loops inside libsolv.

So we're not enabling this yet for those reasons, but
let's land the build infrastructure now.

```
(lldb) thread backtrace
* thread #4, name = 'pool-/usr/bin/r'
  * frame #0: 0x00007fd61b97200f libc.so.6`__memcpy_sse2_unaligned_erms + 623
    frame #1: 0x00007fd61cbc88e6 libasan.so.6`__asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) + 214
    frame #2: 0x00007fd61cc4b725 libasan.so.6`__interceptor_realloc + 245
    frame #3: 0x00007fd61baec43e libsolv.so.1`solv_realloc + 30
    frame #4: 0x00007fd61baf0414 libsolv.so.1`repodata_add_dirstr + 276
    frame #5: 0x00007fd61bb6f755 libsolvext.so.1`end_element + 53
    frame #6: 0x00007fd61b05855d libxml2.so.2`xmlParseEndTag1.constprop.0 + 317
    frame #7: 0x00007fd61b063548 libxml2.so.2`xmlParseTryOrFinish.isra.0 + 888
    frame #8: 0x00007fd61af7ed20 libxml2.so.2`xmlParseChunk + 560
    frame #9: 0x00007fd61bb727e7 libsolvext.so.1`solv_xmlparser_parse + 183
    frame #10: 0x00007fd61bb5ea0e libsolvext.so.1`repo_add_rpmmd + 254
    frame #11: 0x000055a4fce7a5f5 rpm-ostree`::load_filelists_cb(repo=<unavailable>, fp=<unavailable>) at dnf-sack.cpp:444:23
    frame #12: 0x000055a4fce7cad6 rpm-ostree`load_ext(_DnfSack*, libdnf::Repo*, _hy_repo_repodata, char const*, char const*, int (*)(s_Repo*, _IO_FILE*), _GError**) at dnf-sack.cpp:430:13
    frame #13: 0x000055a4fce7df60 rpm-ostree`dnf_sack_load_repo at dnf-sack.cpp:1789:26
    frame #14: 0x000055a4fce7eee9 rpm-ostree`dnf_sack_add_repo at dnf-sack.cpp:2217:28
    frame #15: 0x000055a4fce7f0fb rpm-ostree`dnf_sack_add_repos at dnf-sack.cpp:2271:32
    frame #16: 0x000055a4fce870ee rpm-ostree`dnf_context_setup_sack_with_flags at dnf-context.cpp:1796:29
    frame #17: 0x000055a4fcdf757f rpm-ostree`rpmostree_context_download_metadata at rpmostree-core.cxx:1206:44
    frame #18: 0x000055a4fcdf95c3 rpm-ostree`rpmostree_context_prepare at rpmostree-core.cxx:2001:48
    frame #19: 0x000055a4fce54ab7 rpm-ostree`rpmostree_sysroot_upgrader_prep_layering at rpmostree-sysroot-upgrader.cxx:1018:38
    frame #20: 0x000055a4fcdcb143 rpm-ostree`deploy_transaction_execute(_RpmostreedTransaction*, _GCancellable*, _GError**) at rpmostreed-transaction-types.cxx:1445:49
    frame #21: 0x000055a4fcdba4cd rpm-ostree`transaction_execute_thread(_GTask*, void*, void*, _GCancellable*) at rpmostreed-transaction.cxx:340:34
    frame #22: 0x00007fd61c58f7e2 libgio-2.0.so.0`g_task_thread_pool_thread + 114
    frame #23: 0x00007fd61c3d7e54 libglib-2.0.so.0`g_thread_pool_thread_proxy.lto_priv.0 + 116
    frame #24: 0x00007fd61c3d52b2 libglib-2.0.so.0`g_thread_proxy + 82
    frame #25: 0x00007fd61b8af3f9 libpthread.so.0`start_thread + 233
    frame #26: 0x00007fd61b9c9903 libc.so.6`__clone + 67
(lldb)
```
cgwalters referenced this issue in cgwalters/rpm-ostree Feb 10, 2021
@cgwalters
Add --enable-sanitizers (not on by default yet)
a3c09a7 
This way we at least get unit test coverage (which...
our unit test coverage doesn't do much because our
main code paths require privileges or virt).

One main blocker to this is that rustc doesn't expose
first-class support for this yet:
rust-lang/rust#39699

At a practical level this works when building in release
mode but fails with `cargo test` for some reason; linker
arguments being pruned?  Not sure.

So I was able to use this when composing to find a bug,
but then for some other reason the client
side apparently infinite loops inside libsolv.

So we're not enabling this yet for those reasons, but
let's land the build infrastructure now.

```
(lldb) thread backtrace
* thread #4, name = 'pool-/usr/bin/r'
  * frame #0: 0x00007fd61b97200f libc.so.6`__memcpy_sse2_unaligned_erms + 623
    frame #1: 0x00007fd61cbc88e6 libasan.so.6`__asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) + 214
    frame #2: 0x00007fd61cc4b725 libasan.so.6`__interceptor_realloc + 245
    frame #3: 0x00007fd61baec43e libsolv.so.1`solv_realloc + 30
    frame #4: 0x00007fd61baf0414 libsolv.so.1`repodata_add_dirstr + 276
    frame #5: 0x00007fd61bb6f755 libsolvext.so.1`end_element + 53
    frame #6: 0x00007fd61b05855d libxml2.so.2`xmlParseEndTag1.constprop.0 + 317
    frame #7: 0x00007fd61b063548 libxml2.so.2`xmlParseTryOrFinish.isra.0 + 888
    frame #8: 0x00007fd61af7ed20 libxml2.so.2`xmlParseChunk + 560
    frame #9: 0x00007fd61bb727e7 libsolvext.so.1`solv_xmlparser_parse + 183
    frame #10: 0x00007fd61bb5ea0e libsolvext.so.1`repo_add_rpmmd + 254
    frame #11: 0x000055a4fce7a5f5 rpm-ostree`::load_filelists_cb(repo=<unavailable>, fp=<unavailable>) at dnf-sack.cpp:444:23
    frame #12: 0x000055a4fce7cad6 rpm-ostree`load_ext(_DnfSack*, libdnf::Repo*, _hy_repo_repodata, char const*, char const*, int (*)(s_Repo*, _IO_FILE*), _GError**) at dnf-sack.cpp:430:13
    frame #13: 0x000055a4fce7df60 rpm-ostree`dnf_sack_load_repo at dnf-sack.cpp:1789:26
    frame #14: 0x000055a4fce7eee9 rpm-ostree`dnf_sack_add_repo at dnf-sack.cpp:2217:28
    frame #15: 0x000055a4fce7f0fb rpm-ostree`dnf_sack_add_repos at dnf-sack.cpp:2271:32
    frame #16: 0x000055a4fce870ee rpm-ostree`dnf_context_setup_sack_with_flags at dnf-context.cpp:1796:29
    frame #17: 0x000055a4fcdf757f rpm-ostree`rpmostree_context_download_metadata at rpmostree-core.cxx:1206:44
    frame #18: 0x000055a4fcdf95c3 rpm-ostree`rpmostree_context_prepare at rpmostree-core.cxx:2001:48
    frame #19: 0x000055a4fce54ab7 rpm-ostree`rpmostree_sysroot_upgrader_prep_layering at rpmostree-sysroot-upgrader.cxx:1018:38
    frame #20: 0x000055a4fcdcb143 rpm-ostree`deploy_transaction_execute(_RpmostreedTransaction*, _GCancellable*, _GError**) at rpmostreed-transaction-types.cxx:1445:49
    frame #21: 0x000055a4fcdba4cd rpm-ostree`transaction_execute_thread(_GTask*, void*, void*, _GCancellable*) at rpmostreed-transaction.cxx:340:34
    frame #22: 0x00007fd61c58f7e2 libgio-2.0.so.0`g_task_thread_pool_thread + 114
    frame #23: 0x00007fd61c3d7e54 libglib-2.0.so.0`g_thread_pool_thread_proxy.lto_priv.0 + 116
    frame #24: 0x00007fd61c3d52b2 libglib-2.0.so.0`g_thread_proxy + 82
    frame #25: 0x00007fd61b8af3f9 libpthread.so.0`start_thread + 233
    frame #26: 0x00007fd61b9c9903 libc.so.6`__clone + 67
(lldb)
```
openshift-merge-robot pushed a commit that referenced this issue Feb 10, 2021
@cgwalters @openshift-merge-robot
Add --enable-sanitizers (not on by default yet)
f065c0d 
This way we at least get unit test coverage (which...
our unit test coverage doesn't do much because our
main code paths require privileges or virt).

One main blocker to this is that rustc doesn't expose
first-class support for this yet:
rust-lang/rust#39699

At a practical level this works when building in release
mode but fails with `cargo test` for some reason; linker
arguments being pruned?  Not sure.

So I was able to use this when composing to find a bug,
but then for some other reason the client
side apparently infinite loops inside libsolv.

So we're not enabling this yet for those reasons, but
let's land the build infrastructure now.

```
(lldb) thread backtrace
* thread #4, name = 'pool-/usr/bin/r'
  * frame #0: 0x00007fd61b97200f libc.so.6`__memcpy_sse2_unaligned_erms + 623
    frame #1: 0x00007fd61cbc88e6 libasan.so.6`__asan::asan_realloc(void*, unsigned long, __sanitizer::BufferedStackTrace*) + 214
    frame #2: 0x00007fd61cc4b725 libasan.so.6`__interceptor_realloc + 245
    frame #3: 0x00007fd61baec43e libsolv.so.1`solv_realloc + 30
    frame #4: 0x00007fd61baf0414 libsolv.so.1`repodata_add_dirstr + 276
    frame #5: 0x00007fd61bb6f755 libsolvext.so.1`end_element + 53
    frame #6: 0x00007fd61b05855d libxml2.so.2`xmlParseEndTag1.constprop.0 + 317
    frame #7: 0x00007fd61b063548 libxml2.so.2`xmlParseTryOrFinish.isra.0 + 888
    frame #8: 0x00007fd61af7ed20 libxml2.so.2`xmlParseChunk + 560
    frame #9: 0x00007fd61bb727e7 libsolvext.so.1`solv_xmlparser_parse + 183
    frame #10: 0x00007fd61bb5ea0e libsolvext.so.1`repo_add_rpmmd + 254
    frame #11: 0x000055a4fce7a5f5 rpm-ostree`::load_filelists_cb(repo=<unavailable>, fp=<unavailable>) at dnf-sack.cpp:444:23
    frame #12: 0x000055a4fce7cad6 rpm-ostree`load_ext(_DnfSack*, libdnf::Repo*, _hy_repo_repodata, char const*, char const*, int (*)(s_Repo*, _IO_FILE*), _GError**) at dnf-sack.cpp:430:13
    frame #13: 0x000055a4fce7df60 rpm-ostree`dnf_sack_load_repo at dnf-sack.cpp:1789:26
    frame #14: 0x000055a4fce7eee9 rpm-ostree`dnf_sack_add_repo at dnf-sack.cpp:2217:28
    frame #15: 0x000055a4fce7f0fb rpm-ostree`dnf_sack_add_repos at dnf-sack.cpp:2271:32
    frame #16: 0x000055a4fce870ee rpm-ostree`dnf_context_setup_sack_with_flags at dnf-context.cpp:1796:29
    frame #17: 0x000055a4fcdf757f rpm-ostree`rpmostree_context_download_metadata at rpmostree-core.cxx:1206:44
    frame #18: 0x000055a4fcdf95c3 rpm-ostree`rpmostree_context_prepare at rpmostree-core.cxx:2001:48
    frame #19: 0x000055a4fce54ab7 rpm-ostree`rpmostree_sysroot_upgrader_prep_layering at rpmostree-sysroot-upgrader.cxx:1018:38
    frame #20: 0x000055a4fcdcb143 rpm-ostree`deploy_transaction_execute(_RpmostreedTransaction*, _GCancellable*, _GError**) at rpmostreed-transaction-types.cxx:1445:49
    frame #21: 0x000055a4fcdba4cd rpm-ostree`transaction_execute_thread(_GTask*, void*, void*, _GCancellable*) at rpmostreed-transaction.cxx:340:34
    frame #22: 0x00007fd61c58f7e2 libgio-2.0.so.0`g_task_thread_pool_thread + 114
    frame #23: 0x00007fd61c3d7e54 libglib-2.0.so.0`g_thread_pool_thread_proxy.lto_priv.0 + 116
    frame #24: 0x00007fd61c3d52b2 libglib-2.0.so.0`g_thread_proxy + 82
    frame #25: 0x00007fd61b8af3f9 libpthread.so.0`start_thread + 233
    frame #26: 0x00007fd61b9c9903 libc.so.6`__clone + 67
(lldb)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cgwalters @james-antill