Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

run ci tests on debian 10 #61

Merged
merged 18 commits into from
May 2, 2023
Merged

run ci tests on debian 10 #61

merged 18 commits into from
May 2, 2023

Conversation

leifwalsh
Copy link
Collaborator

No description provided.

@leifwalsh leifwalsh self-assigned this Apr 27, 2023
@codecov-commenter
Copy link

codecov-commenter commented Apr 27, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (2366e01) 41.60% compared to head (5c60475) 41.60%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #61   +/-   ##
=======================================
  Coverage   41.60%   41.60%           
=======================================
  Files           6        6           
  Lines         274      274           
=======================================
  Hits          114      114           
  Misses        160      160           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@leifwalsh leifwalsh requested a review from geofft April 29, 2023 23:33
@leifwalsh leifwalsh marked this pull request as ready for review April 29, 2023 23:33
@leifwalsh leifwalsh enabled auto-merge (squash) April 29, 2023 23:34
@leifwalsh
Copy link
Collaborator Author

ci/test.sh Outdated
cp ci/libnss_whatami.so.2 /lib
sed -i 's/\(passwd\|group\):/& whatami/' /etc/nsswitch.conf
dpkg -i nsncd*.deb
/usr/lib/nsncd &
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need to do this it means the dpkg -i isn't starting the daemon correctly (possibly because you're in a container without systemd? less likely, possibly because there's a /usr/sbin/policy-rc.d file in whatever chroot/environment you're in?). I think we should figure out how to run the test in a context where systemd is running, if possible, so we have test coverage for the systemd unit. Maybe that involves splitting this file into two separate tests, one for the systemd unit on the uncontained ubuntu-latest runner and one for the various platforms we support in containers.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was because there's no systemd in the container, I tried running systemctl status and the command wasn't there.

Yeah, I can add a test on Ubuntu with systemd

@leifwalsh
Copy link
Collaborator Author

@leifwalsh
Copy link
Collaborator Author

leifwalsh commented Apr 30, 2023

Ok @geofft I figured out why this is happening but not what to do about it.

 $ dpkg-shlibdeps -O target/debug/nsncd -v -v
dpkg-shlibdeps: debug: >> Scanning target/debug/nsncd (for Depends field)
dpkg-shlibdeps: debug: Library libgcc_s.so.1 found in /lib/x86_64-linux-gnu/libgcc_s.so.1
dpkg-shlibdeps: debug: Library librt.so.1 found in /lib/x86_64-linux-gnu/librt.so.1
dpkg-shlibdeps: debug: Library libpthread.so.0 found in /lib/x86_64-linux-gnu/libpthread.so.0
dpkg-shlibdeps: debug: Library libdl.so.2 found in /lib/x86_64-linux-gnu/libdl.so.2
dpkg-shlibdeps: debug: Library libc.so.6 found in /lib/x86_64-linux-gnu/libc.so.6
dpkg-shlibdeps: debug: Library ld-linux-x86-64.so.2 found in /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
dpkg-shlibdeps: debug: Library ld-linux-x86-64.so.2 found in /lib64/ld-linux-x86-64.so.2
dpkg-shlibdeps: debug: Using symbols file /var/lib/dpkg/info/libgcc-s1:amd64.symbols for libgcc_s.so.1
dpkg-shlibdeps: debug:  Minimal version of (libgcc-s1 #MINVER#) initialized with (3.0)
dpkg-shlibdeps: debug: Using symbols file /var/lib/dpkg/info/libc6:amd64.symbols for librt.so.1
dpkg-shlibdeps: debug:  Minimal version of (libc6 #MINVER#) initialized with (2.2.5)
dpkg-shlibdeps: debug: Using symbols file /var/lib/dpkg/info/libc6:amd64.symbols for libpthread.so.0
dpkg-shlibdeps: debug:  Minimal version of (libc6 #MINVER#) initialized with (2.2.5)
dpkg-shlibdeps: debug: Using symbols file /var/lib/dpkg/info/libc6:amd64.symbols for libdl.so.2
dpkg-shlibdeps: debug:  Minimal version of (libc6 #MINVER#) initialized with (2.2.5)
dpkg-shlibdeps: debug: Using symbols file /var/lib/dpkg/info/libc6:amd64.symbols for libc.so.6
dpkg-shlibdeps: debug:  Minimal version of (libc6 #MINVER#) initialized with (2.2.5)
dpkg-shlibdeps: debug: Using symbols file /var/lib/dpkg/info/libc6:amd64.symbols for ld-linux-x86-64.so.2
dpkg-shlibdeps: debug:  Minimal version of (libc6 #MINVER#) initialized with (2.2.5)
dpkg-shlibdeps: warning: binaries to analyze should already be installed in their package's directory
dpkg-shlibdeps: debug: Analyzing all undefined symbols
dpkg-shlibdeps: debug:  Looking up symbol getgrouplist@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 2.2.5, dep: libc6 #MINVER#)
dpkg-shlibdeps: debug:  Looking up symbol readlink@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 2.2.5, dep: libc6 #MINVER#)
...
dpkg-shlibdeps: debug:  Looking up symbol _Unwind_GetLanguageSpecificData@GCC_3.0
dpkg-shlibdeps: debug:  Found in symbols file of libgcc_s.so.1 (minver: 3.0, dep: libgcc-s1 #MINVER#)
dpkg-shlibdeps: debug:  Looking up symbol __environ@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 2.2.5, dep: libc6 #MINVER#)
dpkg-shlibdeps: debug:  Looking up symbol __gmon_start__@Base
dpkg-shlibdeps: debug:  Not found
dpkg-shlibdeps: debug:  Looking up symbol shutdown@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 2.2.5, dep: libc6 #MINVER#)
...
dpkg-shlibdeps: debug:  Looking up symbol memrchr@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 2.2.5, dep: libc6 #MINVER#)
dpkg-shlibdeps: debug:  Looking up symbol pthread_join@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libpthread.so.0 (minver: 2.2.5, dep: libc6 #MINVER#)
dpkg-shlibdeps: debug:  Looking up symbol pthread_setspecific@GLIBC_2.2.5
dpkg-shlibdeps: debug:  Found in symbols file of libpthread.so.0 (minver: 2.2.5, dep: libc6 #MINVER#)
dpkg-shlibdeps: debug:  Looking up symbol __nss_disable_nscd@GLIBC_PRIVATE
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 0, dep: libc6 (>> 2.31), libc6 (<< 2.32))
...
shlibs:Depends=libc6 (>= 2.28), libc6 (>> 2.31), libc6 (<< 2.32), libgcc-s1 (>= 4.2)

The annoying bit is right near the end:

dpkg-shlibdeps: debug:  Looking up symbol __nss_disable_nscd@GLIBC_PRIVATE
dpkg-shlibdeps: debug:  Found in symbols file of libc.so.6 (minver: 0, dep: libc6 (>> 2.31), libc6 (<< 2.32))

All the symbols that it sees except for __nss_disable_nscd are either found with some minimum version (in /var/lib/dpkg/info/libc6:amd64.symbols) or not found at all. But __nss_disable_nscd is GLIBC_PRIVATE, so it doesn't have any minimum version information. Because of that, it picks the "alternative dependency template" (after the |) from here:

 $ head -n2 /var/lib/dpkg/info/libc6:amd64.symbols
ld-linux-x86-64.so.2 libc6 #MINVER#
| libc6 (>> 2.31), libc6 (<< 2.32)

It's basically saying, we won't admit to there being some version range where __nss_disable_nscd is valid, so all we can say is the current one has it.

Is there some way you know of to tell dpkg-shlibdeps "no really just don't worry about that symbol"?

@leifwalsh
Copy link
Collaborator Author

Ok this works but it's gross d5dffe6

Copy link
Collaborator

@geofft geofft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm not sure of a better way to do that. I think you could use debian/shlibs.local to suppress the normal symbols+shlibs handling of libc? But you'd need to get the version bound right yourself, which seems worse than what you're doing.

The files generated by this PR work on our internal Debian 10 and 11 systems, so this seems like a good change.

As I mentioned to you the other day, I kind of want to just redo this entirely to build the Rust binaries with normal online cargo build --release inside a manylinux container, on the grounds that manylinux (of whatever version) is a more defensible libc baseline than "the oldest thing Two Sigma happens to run is Debian 10," and then stuff that binary into Debian packages as well as other packaging systems like RPM. Then we can drop the debian/rules vendor stuff. (But I'd still want to use the ordinary Debian packaging infrastructure as opposed to just tossing a .deb together ourselves, which means dh_shlibdeps will still get called on the binary, so the workaround remains applicable and having figured it out remains worthwhile.)

@@ -17,6 +17,7 @@ jobs:

steps:
- uses: actions/checkout@v3
- uses: actions-rust-lang/setup-rust-toolchain@v1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell, actions-rust-lang is owned by neither GitHub nor the Rust project. Do you know if either of these parties is working on an official action? If not, either

  • is v1 a tag we can audit or something, which will not change until we change this file?
  • how annoying is it to use rustup instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably(?) do rustup, I copied this from #58

sudo apt-get update
sudo apt-get install build-essential dpkg-dev
apt-get update
apt-get install -y build-essential dpkg-dev ca-certificates sudo curl
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you remember what you needed these for? Especially sudo is a little surprising in a container.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe actions-rust-lang/setup-rust-toolchain wanted them, but I can check in a bit.

@leifwalsh leifwalsh merged commit ef3e287 into main May 2, 2023
@leifwalsh leifwalsh deleted the debian-10 branch May 2, 2023 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants