Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI #1346

gnzlbg · 2019-05-16T12:59:13Z

No description provided.

rust-highfive · 2019-05-16T12:59:16Z

@gnzlbg: no appropriate reviewer found, use r? to override

gnzlbg · 2019-05-16T15:38:38Z

@asomers @myfreeweb it appears that EVFILT_EMPTY is not available in FreeBSD 11, but the comment in the build.rs says that the constant was added in that FreeBSD version. Is the comment incorrect?

asomers · 2019-05-16T15:49:41Z

Yep, that's wrong EVFILT_EMPTY was added by r312277 and made it into 12.0, but was never merged to stable/11. The erroneous comment was added by 2aeb382 .

https://svnweb.freebsd.org/base?view=revision&revision=312277

asomers · 2019-05-16T15:53:31Z

The title of this PR seems misleading. Tests were never disabled on FreeBSD, were they? It seems like what this PR actually does is enable nightly tests, --no-default-features tests, and --features extra_traits tests for FreeBSD. Perhaps the PR should be titled "enable full testing on FreeBSD"?

gnzlbg · 2019-05-16T15:55:06Z

There are many tests for many constants, functions, structs, etc. that were disabled at some point or another. Some of them were disabled in #1337 to unbreak CI. This PR goes through all disabled tests, and re-enables all that can be made to work.

gnzlbg · 2019-05-16T16:26:27Z

The latest logs show the failures that occur. Most of them appear upgradeable or easily "ignored".

asomers · 2019-05-16T16:29:57Z

Nonetheless, until #570 is fixed we should be targetting FreeBSD 11 (though we can update the image from 11.1 to 11.2). Upgrading the test image to 12.0 will cause a net reduction in test coverage.

asomers · 2019-05-16T16:50:02Z

Please revert those latest changes! They'll break all Rust programs on FreeBSD 11. Prior to those changes, Rust could run on both FreeBSD 11 and FreeBSD 12, it just couldn't use some of 12's newer features.

asomers · 2019-05-16T16:52:15Z

.cirrus.yml

+    - rustup target add i686-unknown-freebsd
+  test_script:
+    - . $HOME/.cargo/env
+    - sh ci/run.sh i686-unknown-freebsd


I suspect that this will fail. Because while FreeBSD amd64 can run i386 binaries, it doesn't include i386 headers, and libc's CI tests need to interrogate those headers. Testing FreeBSD i386 in CI will require an actual i386 environment, either a VM or a jail on an amd64 host.

Ideally we would use qemu-user for this, we need something like that to test aarch64-unknown-freebsd and friends as well.

qemu isn't necessary for running i386 binaries on an amd64 host; that capability is builtin. All you need is an i386 jail, or even a chroot. See https://github.com/nix-rust/nix/blob/master/.cirrus.yml for an example of testing i386 binaries for a project that doesn't need headers.

We do need headers, so that .cirrus.yml does not help much.

asomers

This PR can't be merged until the FreeBSD 12-specific changes are reverted, because they'll break Rust on FreeBSD 11.

gnzlbg · 2019-05-16T16:57:14Z

This PR can't be merged until the FreeBSD 12-specific changes are reverted, because they'll break Rust on FreeBSD 11.

Not all changes break Rust on FreeBSD 11 (e.g. mprotect pointer mutability), and some changes seem to imply that using Rust on FreeBSD 12 will invoke UB (e.g. the dirent changes).

It would be more useful if you could be more specific about which precise changes break what.

gnzlbg · 2019-05-16T16:57:52Z

FWIW "WIP" means work in progress. No need to panic about this getting merged.

asomers · 2019-05-16T17:04:56Z

This PR can't be merged until the FreeBSD 12-specific changes are reverted, because they'll break Rust on FreeBSD 11.

Not all changes break Rust on FreeBSD 11 (e.g. mprotect pointer mutability), and some changes seem to imply that using Rust on FreeBSD 12 will invoke UB (e.g. the dirent changes).

It would be more useful if you could be more specific about which precise changes break what.

There's no UB here; that's a compiler concept. Currently, using libc on FreeBSD 12 is safe thanks to ELF symbol versioning. For example, libc sets readdir's link_name to readdir@FBSD_1.0. That means that whether the OS is 11 or 12, the program will link to a version of readdir that returns FreeBSD 11's version of dirent. So using Rust's libc on FreeBSD 12 works, but it's limited to FreeBSD 11's capabilities. For example, if the running OS happens to have a file system mounted on a directory that has more than 88 characters, then the Rust program won't be able to see the full mountpoint path.

gnzlbg · 2019-05-16T17:46:59Z

Part of re-enabling all FreeBSD tests is figuring out what's broken. The code used to figure out what is broken does not necessarily have to be merged into libc, I don't know how you are jumping to the conclusion that the --cfg hack will be merged. Again: WIP means work-in-progress.

gnzlbg · 2019-05-16T17:50:55Z

There's no UB here;

freelocale in FreeBSD 12 returns void, but the FreeBSD 11 API returns int. We do not link freelocale to the FreeBSD 11 version AFAICT. That is, a user can call it, and inspect the int returned, which, if the function is called in FreeBSD 12, is uninitialized (and therefore it existing is already UB). If the FreeBSD 12 function were to actually return an int, but has a type-signature stating that it returns void, AFAICT, that would also be UB, because the int returned will overrite some part of the calling function stack.

How is the undefined behavior avoided here?

gnzlbg · 2019-05-16T17:51:52Z

The msgrcv function shows a similar issue, where the size of the return value changes between FreeBSD 11 and FreeBSD 12, and we do not link against the FreeBSD 11 version.

asomers · 2019-05-16T20:41:50Z

I don't think there's any UB issues here, since freelocale and msgrcv are external functions. There would only be UB if the compiler knew that freelocale's return value were uninitialized. But since these are FFI functions, the compiler believes that the value is initialied. It might return register garbage, but it won't do anything worse than that.

msgrcv's case is different. In that case the header in FreeBSD 11 is a mistake. The syscall was always returning the value as a ssize_t, but the prototype was wrong. In this case, we could probably change libc's definition to use ssize_t, but I might check with Ed Maste first.
https://svnweb.freebsd.org/base?view=revision&revision=303435

gnzlbg · 2019-05-16T20:44:19Z

From your comment in the other issue, for freelocale, we could probably just remove the int return type on FreeBSD 11 as well.

gnzlbg · 2019-05-24T06:44:27Z

So @asomers slowly this is taking a finished form, I'd like to hear your feedback. Basically, what I've done is split the freebsd module into two modules, one for freebsd11 and one for freebsd12. The FreeBSD11 is always used by default, and there are no direct options to compile the freebsd12 one.

In libc's CI, we define the LIBC_CI env variable, and if that's defined, and the host system is a FreeBSD 12 system, then the freebsd12 module is compiled and tested, which only triggers in the FreeBSD 12 cirrus images.

There are some cfg(freebsd12)s splittered around, mostly in the unix/mod.rs module, whether it makes sense to slowly move them to their appropriate modules or not.. can always be done later.

API wise, would it make sense to do something different for uname in FreeBSD 12 ? Also, we probably want to fix the return type of msgrcv to be ssize_t on FreeBSD 11 as well, but that can be done in a sub-sequent PR (this one should contain no breaking changes).

gnzlbg · 2019-05-24T09:17:37Z

So our API wrapper for lio_listio is AFAICT unsound. We do accept pointers that alias, but the C API uses restrict, which requires these pointers not to alias with anything. I'm going to leave it like this for now, but we probably want to use references & &mut ... here to imply noalias, but that would require teaching ctest to emit restrict when generating pointer types for references to fix this.

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

bors · 2019-05-26T09:56:38Z

💔 Test failed - checks-travis

gnzlbg · 2019-05-26T10:46:10Z

@bors: retry

bors · 2019-05-26T10:46:18Z

⌛ Testing commit 7437d0a with merge 9672c8d...

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

bors · 2019-05-26T11:59:02Z

💔 Test failed - checks-travis

gnzlbg · 2019-05-26T12:05:05Z

@bors: r+

bors · 2019-05-26T12:05:05Z

💡 This pull request was already approved, no need to approve it again.

This pull request previously failed. You should add more commits to fix the bug, or use retry to trigger a build again.

bors · 2019-05-26T12:05:06Z

📌 Commit 7437d0a has been approved by gnzlbg

bors · 2019-05-26T12:05:14Z

⌛ Testing commit 7437d0a with merge e148c9d...

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

bors · 2019-05-26T14:35:14Z

💥 Test timed out

gnzlbg · 2019-05-26T15:41:58Z

@bors: retry

bors · 2019-05-26T15:42:06Z

⌛ Testing commit 7437d0a with merge 819405b...

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

gnzlbg · 2019-05-26T15:43:04Z

For some reason bors does not like this PR. It runs green on all CI, but then bors fails to receive the notification that everything succeeds, and times out. Maybe closing and re-opening the PR helps, otherwise I'll merge it manually.

bors · 2019-05-26T16:55:29Z

💔 Test failed - checks-travis

asomers · 2019-05-26T19:01:35Z

The aarch64-android build is timing out. That's probably why homu isn't merging.

gnzlbg · 2019-05-26T19:10:01Z

That is happening very intermittently (e.g. master is green). The problem here is that even when the travis job was green homu was still not merging this PR. I don't know why, but in the queue it went from "pending" (as in CI is running) back to "approved" (as in, CI needs to run) every time the travis job finished green =/

gnzlbg · 2019-05-26T19:10:30Z

I thought maybe pulling it off the queue and re-adding it (by closing and re-opening the PR) would maybe fix that, but it didn't =/

asomers · 2019-05-26T19:53:26Z

You changed the name of the Cirrus-CI task. Do you need to make a matching change in homu's configuration, too?

gnzlbg · 2019-05-26T19:56:43Z

doesn't homu pick up all tasks automatically independently of their name?(I also added a new task here)

asomers · 2019-05-26T21:35:14Z

It can't, because it wouldn't know that a task existed until that task started. You wouldn't want homu to merge a PR before a CI task had started.

gnzlbg · 2019-05-26T21:36:57Z

There is a PR in rust-central-station to fix that

gnzlbg force-pushed the simplify_freebsd branch from 8fca383 to 1aa4f2f Compare May 16, 2019 15:17

asomers reviewed May 16, 2019

View reviewed changes

asomers suggested changes May 16, 2019

View reviewed changes

gnzlbg force-pushed the simplify_freebsd branch 3 times, most recently from d1e5373 to 0a953a1 Compare May 24, 2019 06:43

gnzlbg force-pushed the simplify_freebsd branch from 0a953a1 to 61b5df8 Compare May 24, 2019 07:06

gnzlbg force-pushed the simplify_freebsd branch 3 times, most recently from 67db2d9 to 0840ee1 Compare May 24, 2019 10:51

gnzlbg changed the title ~~WIP: Re-enable FreeBSD tests~~ Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI May 24, 2019

gnzlbg force-pushed the simplify_freebsd branch from 0840ee1 to eb373ad Compare May 24, 2019 10:53

bors added a commit that referenced this pull request May 26, 2019

Auto merge of #1346 - gnzlbg:simplify_freebsd, r=gnzlbg

d0959f9

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

bors added a commit that referenced this pull request May 26, 2019

Auto merge of #1346 - gnzlbg:simplify_freebsd, r=gnzlbg

9672c8d

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

bors added a commit that referenced this pull request May 26, 2019

Auto merge of #1346 - gnzlbg:simplify_freebsd, r=gnzlbg

e148c9d

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

gnzlbg closed this May 26, 2019

gnzlbg reopened this May 26, 2019

bors added a commit that referenced this pull request May 26, 2019

Auto merge of #1346 - gnzlbg:simplify_freebsd, r=gnzlbg

819405b

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI

gnzlbg merged commit 1059e0b into rust-lang:master May 26, 2019

This was referenced May 26, 2019

[breaking change] Cleanup linux and update MUSL #1365

Merged

[API-breaking] Expose si_addr on siginfo_t. Refs #716 #1316

Closed

gnzlbg mentioned this pull request Aug 23, 2019

Ref parameter incorrectly decorated with noalias attribute rust-lang/rust#63787

Closed

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI #1346

Clean up libc-test/build.rs for FreeBSD and enable testing FreeBSD12 on CI #1346

Conversation

gnzlbg commented May 16, 2019

rust-highfive commented May 16, 2019

gnzlbg commented May 16, 2019 • edited Loading

asomers commented May 16, 2019

asomers commented May 16, 2019

gnzlbg commented May 16, 2019

gnzlbg commented May 16, 2019

asomers commented May 16, 2019

asomers commented May 16, 2019

asomers May 16, 2019

Choose a reason for hiding this comment

gnzlbg May 16, 2019

Choose a reason for hiding this comment

asomers May 16, 2019

Choose a reason for hiding this comment

gnzlbg May 16, 2019

Choose a reason for hiding this comment

asomers left a comment

Choose a reason for hiding this comment

gnzlbg commented May 16, 2019

gnzlbg commented May 16, 2019

asomers commented May 16, 2019

gnzlbg commented May 16, 2019

gnzlbg commented May 16, 2019

gnzlbg commented May 16, 2019

asomers commented May 16, 2019

gnzlbg commented May 16, 2019 • edited Loading

gnzlbg commented May 24, 2019

gnzlbg commented May 24, 2019

bors commented May 26, 2019

gnzlbg commented May 26, 2019

bors commented May 26, 2019

bors commented May 26, 2019

gnzlbg commented May 26, 2019

bors commented May 26, 2019

bors commented May 26, 2019

bors commented May 26, 2019

bors commented May 26, 2019

gnzlbg commented May 26, 2019

bors commented May 26, 2019

gnzlbg commented May 26, 2019

bors commented May 26, 2019

asomers commented May 26, 2019

gnzlbg commented May 26, 2019

gnzlbg commented May 26, 2019

asomers commented May 26, 2019

gnzlbg commented May 26, 2019 • edited Loading

asomers commented May 26, 2019

gnzlbg commented May 26, 2019

gnzlbg commented May 16, 2019 •

edited

Loading

gnzlbg commented May 16, 2019 •

edited

Loading

gnzlbg commented May 26, 2019 •

edited

Loading