Linux system calls may return unreachable or unexpected error codes #10776
Comments
Mathnerd314
added
the
bug
|
Can you attach the used Zig code as well? https://github.com/trailofbits/ebpfault does not have Zig code and |
|
Not sure what you mean. I cooked up |
Can you run the program with strace and show the output behind |
|
It's a bit tricky to strace simultaneously with ebpfault - you have to start the process and then attach to it from both of them. But it's possible. Here is the panic-during-panic behavior with strace: $ cat hello.zig
const std = @import("std");
pub fn main() !void {
const stdin = std.io.getStdIn();
var line_buf: [20]u8 = undefined;
_ = try stdin.read(&line_buf);
const stdout = std.io.getStdOut().writer();
try stdout.print("Hello, {s}!\n", .{"world"});
}
$ zig build-exe hello.zig
$ ./hello &
[1] 83419
[1]+ Stopped ./hello
$ sudo ./ebpfault --config write.json -p 83419 &
[2] 83427
Generating fault injectors...
> write
Error list:
- 100% => -EINVAL
$ strace -p 83419 &
[3] 83443
strace: Process 83419 attached
--- stopped by SIGTTIN ---
$ fg 1
./hello
--- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=82295, si_uid=1000} ---
read(0, 1
"1\n", 20) = 2
write(1, "Hello, ", 7timestamp: 288916424653781 syscall: write process_id: 83419 thread_id: 83419 injected_error: -EINVAL
r15 0000000000000000 r14 0000000000000000 r13 0000000000000000
r12 ffffbf2fc294bf58 rbp ffffbf2fc294bf48 rbx 0000000000000000
r11 0000000000000000 r10 0000000000000000 r9 0000000000000000
r8 0000000000000001 rax ffffffffa98d8b70 rcx 0000000000000000
rdx ffffffffffffffff rsi ffffffffaa695bd9 rdi ffffbf2fc294bf58
orig_rax 0000000000000000 rip ffffffffa98d8b71 cs 0000000000000010
eflags 0000000000000202 rsp ffffbf2fc294bf38 ss 0000000000000018
) = -1 EINVAL (Invalid argument)
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
gettid() = 83419
write(2, "thread ", 7) = -1 EINVAL (Invalid argument)
timestamp: 288916425190934 syscall: write process_id: 83419 thread_id: 83419 injected_error: -EINVAL
r15 0000000000000000 r14 0000000000000000 r13 0000000000000000
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, r12 ffffbf2fc294bf58 rbp ffffbf2fc294bf48 rbx 0000000000000000
r11 0000000000000003 r10 0000000000000000 r9 0000000000000000
r8 0000000000000001 rax ffffffffa98d8b70 rcx 0000000000000000
rdx ffffffffffffffff rsi ffffffffaa695bd9 rdi ffffbf2fc294bf58
orig_rax 0000000000000000 rip ffffffffa98d8b71 cs 0000000000000010
eflags 0000000000000202 rsp ffffbf2fc294bf38 ss 0000000000000018
NULL, 8) = 0
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
write(2, "Panicked during a panic. Abortin"..., 35) = -1 EINVAL (Invalid argument)
timestamp: 288916425740521 syscall: write process_id: 83419 thread_id: 83419 injected_error: -EINVAL
r15 0000000000000000 r14 0000000000000000 r13 0000000000000000
r12 ffffbf2fc294bf58 rbp ffffbf2fc294bf48 rbx 0000000000000000
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, r11 0000000000000000 r10 0000000000000000 r9 0000000000000000
r8 0000000000000001 rax ffffffffa98d8b70 rcx 0000000000000000
rdx ffffffffffffffff rsi ffffffffaa695bd9 rdi ffffbf2fc294bf58
orig_rax 0000000000000000 rip ffffffffa98d8b71 cs 0000000000000010
eflags 0000000000000202 rsp ffffbf2fc294bf38 ss 0000000000000018
NULL, 8) = 0
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
rt_sigaction(SIGBUS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x205900}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[HUP INT RT_32], [], 8) = 0
gettid() = 83419
tkill(83419, SIGABRT) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=83419, si_uid=1000} ---
+++ killed by SIGABRT (core dumped) +++
Aborted (core dumped)
[3]+ Done strace -p 83419
All processes have been terminated
Exiting...
[2]+ Done sudo ./ebpfault --config write.json -p 83419So here with strace the behavior is consistent with the chdir one. But when you try to run hello directly under ebpfault with |
Vexu
added
standard library
|
#11178 is an example of how this happens "in the wild" as opposed to with fault injection. |
I had a similar experience with stage1, when it was unable to come up with a stack trace and panicked in between. I dont understand how the tool handles file stream redirects though (especially the standard streams stdin,stdout and stderr). |
Maybe call it use case? Ideally this would be fixed upstream by the Kernel having a proper machine readable interface. |
The interface is in fact really simple: any syscall can return any error code. This can be proven by modifying my steps-to-reproduce to the syscall and error code of your choice. Even if you ignore this case, Matthew Wilcox's message confirms that the kernel reserves the right to use whatever error codes they want, and they will never maintain such an interface. The only way to produce a more specific interface without playing error code whack-a-mole would be an automated analysis of the kernel code. But such an analysis would still give 50+ error codes for each syscall, so it is arguable if it would help. |
|
Most calls look something like this: switch (system.getErrno(res)) {
.SUCCESS => return,
.FAULT => unreachable,
.INVAL => unreachable,
.ACCES => return error.AccessDenied,
.NOMEM => return error.SystemResources,
.NOTDIR => return error.FileNotFound,
.PERM => return error.AccessDenied,
else => |err| return unexpectedErrno(err),
}The only two codes that are handled with
Unexpected error codes are handled by returning You could patch the Linux kernel to do anything, such as returning EINVAL for So, yes, the Linux kernel is assumed by the standard library to have certain kinds of behavior. If you want to reach into a lower level of abstraction, you can use
Generally, zig does allow writing robust programs, and robust programs should expect the kernel to never return EFAULT from chdir. Crashing in this case could be argued to be more robust, in fact, since proceeding with an unpredictable kernel may have unintended consequences. Or presumably, we have discovered that our pointer is dangling, which is dangerous and panicking is the right move here. If you have issues with how a specific syscall is wrapped, then you are welcome to file a bug report for that function and we can discuss it. |
andrewrk
removed
the
bug
With this method there is no way to get the original error code. Meanwhile C's
One practical case this would happen is where a new kernel is released but a stable branch of Zig is in use. The current std.os design does not allow handling a new error code in the application, so the standard library would have to be patched to add the error code. This ad-hoc patching is fragile and tedious. I'd like to be able to handle unexpected error codes specifically in the application. The behavior isn't undefined if you're expecting it to happen. But you can't handle the errors with std.os, because the error code is not accessible.
It seems that given the current design using std.os.linux is the only choice. But using std.os.linux you lose all the advantages of std.os (slices, cross-platform, retrying on EINTR, etc.), so I don't think it's really an option. Rather than a lower level of abstraction, I would prefer a high-level abstraction that provides decent error codes.
The reason I filed this issue was to get ahead of the game and avoid the next dozen issues about error codes. Also because enumerating the codes returned is a "Sisyphean task". If you really want some cases Zig is missing, how about EBADF, EDQUOT, ENXIO, EOPNOTSUPP, EROFS, ETXTBSY, EWOULDBLOCK in openZ() and EINTR in fsync(). |
Fair point. I will re-open this as a proposal. |
andrewrk
added
breaking
|
Boiling it down, I think all you need is something like this: threadlocal var lastErrNo: E = SUCCESS;
pub fn errno(r: usize) E {
const err = system.getErrno(r);
lastErrNo = err; // store errno during conversion to Error
return err;
}Then you can do: os.openZ("/dev/null", os.O.RDWR, 0) catch |err| switch (err) {
error.Unexpected => switch(std.os.lastErrNo) {
.EBADF => ...
}
}lastErrNo could even be OS-specific so you would have to write It doesn't cover unreachable, but as you say I think the kernel can be assumed to have some amount of sanity. |
|
Zig std lib will not be repeating the crime against humanity that is thread-local errno. |
|
Well, with #2647, the full error code can be added to the Unexpected error: pub const UnexpectedError = error{
/// The Operating System returned an error code not handled by Zig's standard library.
Unexpected: usize,
};
...
os.openZ("/dev/null", os.O.RDWR, 0) catch |err| switch (err) {
error.Unexpected => |errCode| switch(errCode) {
.EBADF => ...
}
}You said that you preferred passing the data as an out parameter. I guess that would work too: var errCode: usize = 0;
os.openZ("/dev/null", os.O.RDWR, 0, &errCode) catch |err| switch (err) {
error.Unexpected => switch(errCode) {
.EBADF => ...
}
}But it would mean every function in std.os would have to take an errCode parameter. |
I wanted a note on this issue that, even in the case that the only thing to do is crash, Ideally, |
|
There are a couple of issues open that track related things (e.g. #6925 and #6389). I'm not sure whether to open a new issue for specific examples, or add a comment to one of these. A blocking This code reproduces the issue in 0.14.0-dev.1694+3b465ebec. const std = @import("std");
const posix = std.posix;
var listener: ?posix.fd_t = null;
pub fn main() !void {
listener = try posix.socket(posix.AF.INET, posix.SOCK.STREAM, posix.IPPROTO.TCP);
try posix.setsockopt(listener.?, posix.SOL.SOCKET, posix.SO.REUSEADDR, &std.mem.toBytes(@as(c_int, 1)));
try posix.listen(listener.?, 128);
std.posix.sigaction(std.posix.SIG.INT, &.{
.handler = .{ .handler = shutdown },
.mask = std.posix.empty_sigset,
.flags = 0,
}, null);
_ = try posix.accept(listener.?, null, null, 0);
}
fn shutdown(_: c_int) callconv(.C) void {
if (listener) |l| {
posix.close(l);
}
} |
Zig Version
0.9.0
Steps to Reproduce
For this to work you must have a Linux kernel with
CONFIG_BPF_KPROBE_OVERRIDE=y. Apparently this is the default on Arch Linux, but for NixOS I had to add:and build a kernel.
Once that is done you can run:
Then in the shell:
This is only the easiest way to get rare kernel errors to show up. As Matthew Wilcox writes, there are 70+ filesystems and also hooks such as Linux Security Modules. These can all return their own error codes under the correct conditions. Simply do
git grep -ho '\-E[A-Z]*' fsorgit grep -ho '\-E[A-Z]*' securityin a kernel git repo to see the wide variety of error codes - nearly all error codes are in use. Furthermore a new release of the kernel may return new error codes. Figuring out the actual set of error codes each system call may return is a "Sisyphean" task that is best left unattempted.Expected Behavior
I expect Zig to allow writing robust programs, so crashing with a panic is definitely not what I expected. I should be able to see the exact error code such as EINVAL or EFAULT when it comes up in practice, in the stack trace - the message for Unexpected doesn't count because it only shows up in debug mode. I should be able to handle the error code in a way appropriate to the application, without modifying Zig's standard library.
Actual Behavior
On running
sudo ./ebpfault --config efault.json --exec ./chdir:On running
sudo ./ebpfault --config einval.json --exec ./chdir:The text was updated successfully, but these errors were encountered: