find a way to make psub --fifo safe from deadlock

[geoff-nixon]

There are actually a couple of bugs here.

The easy one is here — use_fifo is missing a sigil and is therefore a string comparison, causing psub to always act as psub -f.

Unfortunately, it's not as simple as fixing that typo as doing so will cause a non-interruptable hang under certain circumstances. I believe it occurs when the pipe buffer is exceeded? But I'm not sure how to actually determine the pipe buffer in fish. Maybe forking another process is needed? Or... something.

Anyway, here's a (hopefully) cross-platform test case (about 1.5MiB and requires Java, uses openssl to decode base64). It's a standalone wrapper for rhino. There's a big hunk of base64 in the middle of it, but the script is just:
java -jar (echo 'BIGHUNKOFBASE64' | openssl base64 -d | psub)

[ridiculousfish]

Nice diagnosis and test case!

[zanchey]

I've pushed a fix to a topic branch (19217f3 on psub_fix), but I don't want to merge to master until the buffering issue is worked out.

[faho]

Any progress on this?

[geoff-nixon]

@faho Yes. And no. But, yes.

But before we get to the good stuff, lets review:

1. Disregard everything in the psub man page.
   Virtually unchanged from when it was written 10 years ago for fish 1.0.0, literally (and by literally, I mean literally) every sentence in man psub is wrong (except for the example). With that out of the way...
2. What is 'process substitution'?
   Process substitution (which itself is something of a misnomer) was a concept introduced in the Korn shell (ksh88) as a shorthand for the often messy process of using file descriptors and named pipes with a command or program when the program expects a file as an argument. In other words, it is a way to emulate including these programs in a UNIX pipeline, where one otherwise could not, either simply by design (man dd), or for some other clever reason (man tee). The syntax used by shells that support this is:

command <(process) ...

The standard output of process is fed into a file descriptor or named pipe, which is passed as an argument to command.

and

process >(command) ...

The standard input of command is read from whatever process is doing with that argument (ideally, producing output).

This syntax is by no means standard, and it certainly is not POSIX. It is supported by ksh88 and ksh93, but not pdksh or mksh; bash supports it, but not in sh mode (bash --posix). It is supported in zsh when not in an emulation mode that proscribes it; and zsh also has another syntax, =(process), which we'll get to in a bit.

The fish psub function:

command (process | psub)

(purportedly) capitalizes on the fact that no special syntax is needed to perform "process substitution"; the ordinary syntax of command substitution (command or $(command) in Bourne shells, (command) is fish) can be used to accomplish the same thing pipeline within the command substitution whose standard output culminates in a file descriptor or named pipe.

1. So, why has this bug stayed in place since it was introduced seven years ago (c6ebb23), two years since I opened this issue, followed by about a dozen others?
   This is where it gets tricky, and there are two factors at play.

   a. Many utilities which take files as arguments (i.e., for input or output), do so for a reason. A buffered UNIX pipeline often simply is not acceptable ... blah blah blah I've been up all night hacking on this so I'll finish my blustering treatise later. 💀

Basically, I propose we use a ramdisk. It combines to the "durability", ability to handle large files and non-streamable data of "file substitution" (?) (zsh =(command), what we're doing with psub at present) with the speed and ephemeral nature of using a fd or fifo.

@ridiculousfish @zanchey @anyone @anyone @bueller... Initial thoughts?

function psub --description "Process substitution, revisited."

    set -l filename
    set -l funcname

    set -l halfmem
    set -l sectors
    set -l ramdisk
    set -l mountpoint
    set -l psubdir

    set -l use_file 0

    while set -q argv[1]
        switch $argv[1]
            case -h --help
                __fish_print_help psub
                return 0

            case -f --file
                set use_file 1
        end
        set -e argv[1]
    end

    if not status --is-command-substitution
        echo psub: Not inside of command substitution >&2
        return 1
    end

    if test -z "$TMPDIR"
        set TMPDIR /tmp
    end

    # Implemention for other systems is left as an exercise for the reader.
    if test (uname) != Darwin
        set use_file 1                            # ... mount -t tmpfs ...
    end

    if test $use_file -eq 1
        while not set psubdir (mktemp -d $TMPDIR/.psub.XXXXXXXXXX); end
        chmod 0300 $psubdir

        while not set filename (mktemp $psubdir/temp.XXXXXXXXXX); end
        chmod 0100 $psubdir
        chmod 0200 $filename
    else
        # Basically: detect memory and use 1/2 of it (the default with tmpfs
        # on other platforms) as a ramdisk. The memory is allocated as needed.
        # `hdid -nomount ram://SECTORS` (of 512 bytes) on Darwin.

        set halfmem (math (sysctl -n hw.memsize) / 2)
        set sectors (math $halfmem / 512)
        set ramdisk (hdid -nomount ram://$sectors | tr -d [:space:])
        chmod 0600 $ramdisk

        # $ramdisk is now something like '/dev/disk2'; it would be nice if we
        # could just use the raw device file as $filename, but if we do that
        # there's no EOF. So we format, mount, and use a tempfile on our
        # ramdisk. UDF or is probably as good as anything. We probably just
        # don't want any filesystem that's journaled to reduce overhead.


        while not set mountpoint (mktemp -d /$TMPDIR/.psub.XXXXXXXXXX); end
        chmod 0300 $mountpoint
        newfs_udf $ramdisk >/dev/null 2>&1
        mount_udf -o nobrowse $ramdisk $mountpoint

        while not set psubdir (mktemp -d $mountpoint/.psub.XXXXXXXXXX); end
        chmod 0300 $psubdir        

        while not set filename (mktemp $psubdir/temp.XXXXXXXXXX); end
        chmod 0100 $mountpoint
        chmod 0100 $psubdir
        chmod 0200 $filename
    end

    # Write stdin to tempfile
    cat > $filename
    chmod 0400 $filename


    # Write filename to stdout
    echo $filename

    # Find unique function name
    while true
        set funcname __fish_psub_(random)
        if not functions $funcname >/dev/null 2>&1
            break
        end
    end

    # Make sure we unmount and detatch when caller exits.
    function $funcname --on-job-exit caller --inherit-variable filename --inherit-variable funcname --inherit-variable use_file --inherit-variable ramdisk --inherit-variable mountpoint --inherit-variable psubdir
        chmod 0700 $psubdir $filename
        if test $use_file -eq 0
            umount -f $mountpoint >/dev/null 2>&1
            which hdiutil >/dev/null 2>&1
            and hdiutil detach $ramdisk >/dev/null 2>&1
            chmod 0700 $mountpoint
        end
        command rm -rf $mountpoint $psubdir $filename
        functions -e $funcname
    end
end

[faho]

Virtually unchanged from when it was written 10 years ago for fish 1.0.0, literally (and by literally, I mean literally) every sentence in man psub is wrong (except for the example).

I don't think so - it could be improved but I don't see much that is wrong. A bit imprecise and awkward maybe but assuming psub were bug-free it would mostly be sort-of correct. Anyway, this isn't really important to the matter at hand - we should improve psub documentation, but mostly we should just improve psub.

Many utilities which take files as arguments (i.e., for input or output), do so for a reason. A buffered UNIX pipeline often simply is not acceptable

Or try to comm or diff the output of two commands - how would you specify that? Piping syntax doesn't scale beyond a one-to-one relationship (at least I haven't ever seen how that'd work).

... mount -t tmpfs ...

Nope - "mount: only root can use "--types" option". (There is probably a way, but that one's not it)

which hdiutil >/dev/null 2>&1
and hdiutil detach $ramdisk >/dev/null 2>&1

This doesn't seem great - you're creating a ramdisk with one tool and then only detach it if another tool exists?

Anyway, I don't see the merits of this approach for my system - /tmp (where current psub stores its fifo or file) is already a tmpfs. As my cursory googling shows, it's the default on archlinux (my distro), Fedora, Debian (nope, they reverted) and maybe Ubuntu, so other linuxen also won't benefit from this - at all. Even for those that don't have /tmp on tmpfs, we could use /run instead, which is explicitly defined to be one.

Plus, fifos have an advantage that you don't have here - they can be filled in the background, which means the reading side can start earlier (which is presumably why zsh offers both fifos and files). Check the current psub source (try not to step on the bugs) - the fifo path does so, while the file path does not. Also, this increases the code complexity, especially the amount of OS-specific code.

Can't say I'm a fan.

[geoff-nixon]

I don't think so - it could be improved but I don't see much that is wrong. A bit imprecise and awkward maybe but assuming psub were bug-free it would mostly be sort-of correct. Anyway, this isn't really important to the matter at hand - we should improve psub documentation, but mostly we should just improve sub.

It is important if you one wants to know what we're trying to accomplish with this. Line-by-line:

Posix shells feature a syntax that is a mix between command substitution and piping, called process substitution.

• Posix shells do not feature process substitution.

It is used to send the output of a command into the calling command, much like command substitution, but with the difference that the output is not sent through commandline arguments but through a named pipe, with the filename of the named pipe sent as an argument to the calling program.

• Aside from the fact that bash is the only shell which can fall back to using a FIFO (on systems which lack numbered file descriptors), all other shells which implement process substitution use numbered file descriptors, not named pipes. The sentence is also self-contradictory, as it says output is not sent via command line arguments, then goes on to say the filename (the output of the final command in the pipeline) is sent as an argument.

psub combined with a regular command substitution provides the same functionality.

• It doesn't, since it exclusively uses named pipes (in theory). But its also wrong because:

If the -f or --file switch is given to psub, psub will use a regular file instead of a named pipe to communicate with the calling process.

• Nope, because of the type, this is what it always does, every since that switch was added.

This will cause psub to be significantly slower when large amounts of data are involved, but has the advantage that the reading process can seek in the stream.

• Nope, for the reason above, psub and psub -f have identical behavior. Its also not necessarily true in any case, depending on how the stream is buffered, disk throughput, etc.

Every sentence.
On to more relevant matters:

Many utilities which take files as arguments (i.e., for input or output), do so for a reason. A buffered UNIX pipeline often simply is not acceptable

Or try to comm or diff the output of two commands - how would you specify that? Piping syntax doesn't scale beyond a one-to-one relationship (at least I haven't ever seen how that'd work).

You're quoting right where I dropped off there, so I'm not sure we actually disagree here.

But my point here is threefold:

1. Often, a pipeline lacks the necessary complexity to handle all input and output (which we seem to agree on),
2. However, pipes of any type, and therefore process substitution by any means (FIFOs, /dev/fd/X, etc) may not (and frequently do not) function in a manner sufficient to produce the same behavior as a regular file, do to buffering, etc..
3. But notwithstanding the point above, there are a sufficient number of situations where one does not want large, intermediate temporary files on magnetic disks.

which hdiutil >/dev/null 2>&1
and hdiutil detach $ramdisk >/dev/null 2>&1

This doesn't seem great - you're creating a ramdisk with one tool and then only detach it if another tool exists?

No, while I probably don't need this guard line any more as I've since wrapped it in an if block, what I'm doing is handling an annoying fish "feature" which does not allow you to squash the output of attempting to use a command that does not exist.

~> asdfasdf >/dev/null 2>&1
fish: Unknown command 'asdfasdf'

... mount -t tmpfs ...
Nope - "mount: only root can use "--types" option". (There is probably a way, but that one's not it)
Anyway, I don't see the merits of this approach for my system - /tmp (where current psub stores its fifo or file) is already a tmpfs. As my cursory googling shows, it's the default on archlinux (my distro), Fedora, Debian (nope, they reverted) and maybe Ubuntu, so other linuxen also won't benefit from this - at all. Even for those that don't have /tmp on tmpfs...

Perhaps you missed my joke:

# Implemention for other systems is left as an exercise for the reader.

That is to say, it's likely already implemented. As in, when your init scripts ran mount -t tmpfs as root.

Plus, fifos have an advantage that you don't have here - they can be filled in the background, which means the reading side can start earlier (which is presumably why zsh offers both fifos and files).

There's nothing preventing one from reading a regular file while its still being written. tail -f?
The difference is a fifo is buffered, which is also why so many programs fail with fifos.

Also, this increases the code complexity, especially the amount of OS-specific code.

There is ample precedent for this. There is an immense ammount of OS-specific code in fish. ls.fish. open.fish. How man __fish_systemctl_SOMETHING.fish functions are there?

Can't say I'm a fan.

Well... sorry, I guess? We can't all run Arch Linux.

And I must say, pretty rude IMO, considering I only did any of this in light of the fact that you specifically asked for "progress" on this issue. I guess I interpreted that to mean more meaningful/fundamental improvements, since the forking problem is a much larger issue, well beyond the scope of this here..

If all you're looking for is a once-off workaround for 'THIS FISH DON'T FORK!', and you just want psub to work just like the <(kshisms), all you basically need is to fork the background process yourself.
This patch should do it.

[faho]

Posix shells do not feature process substitution.

Granted, but minor.

Aside from the fact that bash is the only shell which can fall back to using a FIFO (on systems which lack numbered file descriptors), all other shells which implement process substitution use numbered file descriptors, not named pipes.

Sure that zsh does it that way?

Nope, because of the type, this is what it always does, every since that switch was added.

"assuming psub were bug-free".

You're quoting right where I dropped off there, so I'm not sure we actually disagree here.

We don't - I was expanding on your point.

handling an annoying fish "feature" which does not allow you to squash the output of attempting to use a command that does not exist

Ah okay. In that case, shouldn't you do that with anything? Or isn't the error output here kinda important? You're leaking tmpfss (if I understand correctly).

Perhaps you missed my joke:

I was trying to say that it might be more complicated for other systems, though now I see that we probably could use /run on linux and just do the ramdisk setup on OSX/BSD.

There's nothing preventing one from reading a regular file while its still being written. tail -f?

Maybe we should consider running the cat > file in the background then, too?

There is ample precedent for this.

This seems a bit more complicated than most OS-specific paths.

How man __fish_systemctl_SOMETHING.fish functions are there?

For the record, I'm a bit annoyed by those, mostly since most of them are only used by the systemctl completion, AFAIK (I've thought about moving them into that, but I wanted to look into why they were moved out). Also, this is in completions, which are much less critical than psub.

And I must say, pretty rude IMO, considering I only did any of this in light of the fact that you specifically asked for "progress" on this issue. I guess I interpreted that to mean more meaningful/fundamental improvements, since the forking problem is a much larger issue, well beyond the scope of this here..

If I came of as rude, I'm sorry about that. It was never my intention. I was merely trying to express my technical opinion of your code. Maybe I was too blunt - might be my inherent german-ness (germanity?) or my mastery of the english language. Anyway, I appreciate your willingness to help here, I just don't agree with your proposal.

I only did any of this in light of the fact that you specifically asked for "progress" on this issue. I guess I interpreted that to mean more meaningful/fundamental improvements, since the forking problem is a much larger issue, well beyond the scope of this here..

I was more asking about @zanchey's topic branch and the work on the buffering issue. The buffering also bites us in other respects - look for bugs about functions running in the background, so it should be fixed anyway, which would also fix psub (well, that and the missing "$").

In that light, your ramdisk idea comes across as optimization work, and for that I didn't like the added complexity - the added forks (via e.g. math) might also cost more performance than they save, especially in short-lived psubs.

[geoff-nixon]

Apology accepted.

Sure that zsh does it that way?

mpb% echo <(echo) <(echo)
/dev/fd/11 /dev/fd/12

Or isn't the error output here kinda important? You're leaking tmpfss (if I understand correctly).

No, the error isn't important. And no, we're not leaking tmpfs's. Mac OS X (Darwin, technically) has a rather bizarre mechanism for userland disks. hdid technically creates an in-memory disk image; and this disk needs to be unmounted, then "ejected", then "detached" to actually remove entry in /dev and the inode. The "error" I'm suppressing is "disk2" unmounted. "disk2" ejected.

Maybe we should consider running the cat > file in the background then, too?

Well, here you run essentially the opposite risk of using a buffer; if the reading process consumes at a faster rate than the outputting process it will "starve" and terminate.

I was more asking about @zanchey's topic branch and the work on the buffering issue. The buffering also bites us in other respects - look for bugs about functions running in the background, so it should be fixed anyway, which would also fix psub (well, that and the missing "$").

So, I think you might be conflating issues with pipe buffer(s) with the issue of "to fork or not to fork" within a pipeline (ephemeral file descriptors).

Pipe buffers are handled in-kernel, and are particularly relevant to FIFOs and real file descriptors.
The pipe buffer typically has a hard limit set by the operating system. All shells suffer equally from the limits of the pipe buffer. In the image below, note the similar error messages, which are due to the pipe buffer being exceeded:

[ Note that this is with #2423; otherwise, the fish version would hang here. ]

There is a separate issue, of where, when, how, and with what, to fork and/or create a new thread, within a pipelined chain of commands.

See the lengthy discussion in #1228; in that vein, I still think fish needs to abstract and internalize the concept of file descriptors better; they needn't necessarily be tied to the actual file descriptors that exist outside the shell. In the following image, you can see bash actually creates and populates entries in /dev/fd that do not exist anywhere outside that instance of the shell; as far as I'm aware, fish does no such thing.

[faho]

The "error" I'm suppressing is "disk2" unmounted. "disk2" ejected.

Ummh...the error you'd be suppressing is "Unknown command 'hdiutil'" - this is about the which hdiutil; and part, not the hdiutil detach call. If detaching is important, you should show an error if it can't be done.

this disk needs to be unmounted, then "ejected", then "detached" to actually remove entry in /dev and the inode.

In that case are you leaking entries in /dev or are they reused?

Well, here you run essentially the opposite risk of using a buffer; if the reading process consumes at a faster rate than the outputting process it will "starve" and terminate.

So there is something stopping us from backgrounding writing to a regular file.

All shells suffer equally from the limits of the pipe buffer

Currently, fish suffers worse, because it actually hangs. (Which IIUC is because of #238 - we never get to the reading before finishing the writing so if we can't finish the writing because the buffer is full...)

---

Okay, let's look at the ramdisk stuff again: The setup on OSX is really rather complicated, while on linux we could use /run (with a fallback to /tmp). The advantage of this approach is that the data never hits the disk (unless of course it swaps) but is still seekable. The disadvantage is that even readers who can deal with waiting for data (like presumably tail -f) will be started only after the data is fully written - this also means the behavior for very large data with readers who can deal with a fifo is somewhat worse than the behavior with an actual fifo.

So it is just straight up better than using on-disk files (discounting the code complexity), but not strictly better than fifos - which is how we'd again end up with offering two solutions (and letting the user decide between them since we can't, like zsh does).

I still think fish needs to abstract and internalize the concept of file descriptors better; they needn't necessarily be tied to the actual file descriptors that exist outside the shell.

I'm afraid I don't completely understand - how would that help with psub here? Wouldn't that still be tied to the buffering limitations?

In the following image, you can see bash actually creates and populates entries in /dev/fd that do not exist anywhere outside that instance of the shell

Be careful what you wish for when it comes to bash and what things it does to /dev - or you might end up implementing /dev/tcp.

[zanchey]

As a systems administrator, I am terrified by the use of memory-backed filesystems as written.

[zanchey]

The topic branch has bitrotted, and wasn't a particularly novel fix so I've removed it while the rest of the issue is worked out.

[wlad-svennik]

Could we not have two commands for now? A psub which does process substitution using fifos; and an osub which runs the command, waits for it to finish, and puts its output in a file.

So adding the missing sigil gives us psub and without it we get osub.

[faho]

@jkabrg: See #2052 - the path that would be reached when adding that "$" is basically broken, so adding the sigil again would make it worse, not better.

[krader1961]

It's pretty clear that this is not going to be fixed as part of the 2.3.0 release milestone so I'm punting this back to next-2.x.

[krader1961]

I'm removing the "next-2.x" label because this has been open for three years. There is no reason to think this will become a priority to fix anytime soon.

[urxvtcd]

Hello there, I have encountered a situation where fish hangs upon process substitution, like in paste (tail -f foo | psub) (tail -f bar | psub). Bash analogue works fine. Is this related to this issue? And does some workaround exist?

[faho]

@urxvtcd: Yes, that's this issue. psub currently only returns when the file has been fully written, and with tail -f that never happens since it keeps writing.

The workaround is to use something that can follow multiple files at the same time (like multitail). Or use multiple terminal windows with one tail each, or use e.g. tmux.

Alternatively, and I recommend against this, you can make your own fifos (with mkfifo) and then redirect the tail outputs into those.

Note that, if you background the tails, this will pretty much by necessity create dangling jobs - tail -f won't ever quit, so whatever is reading it won't either. I expect this to be an issue with bash as well.

[krader1961]

While fixing #4222 I updated the psub documentation to clarify that --file is the default behavior. I also added a new --fifo flag to request the use of a named pipe and documented when and why you shouldn't use that flag. We still need to find a way to make using --fifo safe from deadlock but that's going to require fundamental changes to fish internals.

[zanchey]

Did 6c22c88 fix this?

[floam]

I don't think cat would be any better.

[floam]

Whoops, I think the title of that commit message was reversed.

[faho]

Did 6c22c88 fix this?

It did not - echo (cat BIGFILE | psub --fifo) still doesn't complete, without "--fifo" it works.

This sort of looks like fish is waiting for the cat, i.e. the process writing to the psub?

[sniperrifle2004]

It seems to me that there is a fundamental semantic difference between pipe-based process substitution (ie. psub --fifo <(..)) and command substitution: Command substitution must wait for the enclosed pipeline/job/whatever to complete (It has to gather all the output), while pipe-based process substitution must not due to the nature of a pipe (Whether it is named or an fd is irrelevant). I believe that this is why it is so hard to get psub --fifo to work beyond a single pipe buffer of output. No amount of backgrounding or changing utilities in psub will ever get the command substitution to let go of its enclosed job, before it is done. By pure coincidence this actually does manage to work:

echo (cat 'some_"large"_file' | psub --fifo &)

That looks very scary (to me, at least), but I think #238 might actually cause this to work consistently for me so far. Another thing that works is the following (This example is contrived and admittedly completely ridiculous):

echo (cat 'some_"large"_file' | psub --fifo | xargs cat)

Oof that hurts, but I think this illustrates quite clearly that the problem is not with the pipe, nor with psub, but with command substitution. While it may seem similar to pipe-based process substitution on the surface, their semantics are different. So different that trying to combine them together may be hard if not impossible (or a really really dirty hack).

psub --file (and zsh =(...)) on the other hand is semantically similar to command substitution. It will always faithfully capture all its input into its file (ie. wait for the preceding process to finish) and dutifully return the filename afterwards. By that point the earlier parts of the pipeline will also be finished so the command substitution can return. psub --file is also safer to use since it returns true files.

Given this I would prefer to have a separate construct for pipe-based process substitution, so that I am aware that what I am about to do will spawn a background job, that it will be taken care of for me by fish and that what I get back will be a (named) pipe file name that will also be taken care of for me by fish. And also that I will be properly aware of all the gotchas (Of which I am sure there are plenty). I love fish for its adherence to a simple clean syntax, but if a construct has complex semantics like pipe-based process substitution, it had better stand out. This is clearly not the case right now, given how common command substitution is.

[stephane-chazelas]

In case nobody has made that case yet

comsumer (producer | psub -F)

cannot work unless fish is modified so as not to wait for cmd1 in cmd1 | cmd2 any longer (like in the Bourne shell or AT&T ksh).

The (...) will not be substituted until producer | psub -F exits and that's not before producer exits. psub ultimately runs a tee some-fifo. tee will hand on the open(O_WRONLY) on that fifo until somebody (typically the consumer) opens it in read mode. That is not going to happen before (...) expands. So you'll get a deadlock unless consumer manages to write all its output in the pipe buffer and exits.

The only way to fix is to not wait for the producer. If there's a fifo, it's meant to be an IPC system, with commands running concurrently, not one after the other.

Another problem with the current approach is that it creates an extra process (cat or tee) that shoves data around for no good reason (instead of the producer writing directly to the pipe)

The interface would probably have to change, like in:

diff -u (psub cmd1) (psub cmd2)

With which you could do away with named pipe and use the far cleaner /dev/fd/x.

That psub would create a pipe, output /dev/fd/x (where x is the reading end of the pipe) for the command substitution and run cmd1 in background with its output going to the writing end of the pipe.

And you could have a psub -w to do the same as ksh's >(...).

You'd still need to implement a pipe builtin or operator though (like yash's x>|y). IMO, it would still be better to just implement <(...), >(...) properly a la ksh.

[stephane-chazelas]

2. What is 'process substitution'?
   Process substitution (which itself is something of a misnomer) was a concept introduced in the Korn shell (ksh88)

ksh had process substitution already before ksh86. ksh86 is when it was first documented. In that version, the reading ones <(...) could actually be written (...) like fish's command substitution as in:

diff (echo a) (echo b)

I don't know when that one was removed.

[stephane-chazelas]

* _Aside from the fact that bash is the only shell which can fall back to using a FIFO (on systems which lack numbered file descriptors), all other shells which implement process substitution use numbered file descriptors, not named pipes.

That's not true, when zsh and Byron Rakitzis's rc introduced theirs (in 1.0 in 1990 for zsh, 0.9 in 1991 for rc, a few years before bash, but after ksh's and Tom Duff's original rc), they were using named pipes only.

It's only later that they switched to unnamed pipes and /dev/fd on systems that supported them (but still fell back to named pipes on others). That was in 2.6beta17 (1996) for zsh and 1.2 for rc (later in 1991)

The ability to use named pipes when /dev/fd is not available was added to ksh93 in ksh93u+ in 2011.

(in any case, that's no about numbered file descriptors but about being able to refer to file descriptors with /dev/fd/x).

[krackers]

@stephane-chazelas Thanks for that, I was finding that things like cat (yes | psub) were blocking, and your post perfectly explains why it does so. Rewriting the psub function so that the syntax is cat (psub yes) by changing the line to $argv | command tee $filename >/dev/null & seems to fix it. The downside of this though is that if you do it this way there doesn't seem to be a clean way to support pipe constructions within there likecat <(yes | grep -v "n").

You could do cat (psub "yes | grep -v 'n'") and use eval $argv in the script, but that seems to deadlock, probably related to fish not supporting concurrent execution. Doing an explicit fork (fish -c "$argv ...) seems to work, but overall the syntax is not really convenient.

Edit: Although I guess it's similar to the syntax for moreutils pee (which implements input process substitution)

[tmccombs]

I see a few paths forward for this:

1. Add dedicated syntax for process substitution, so that the shell can handle passing the /dev/fd or fifo file to the process, and run the substituted pipeline in the background. I.e. Do what bash and zsh do. It sounds like there is opposition to that though.
2. Add special magic for the psub command, so that if a command substitution ends with a psub command, it will run the pipeline in the background. So basically, using psub is the special syntax for process substitutuion. I don't love the magic, but there is some precedent for it, like how string split behaves specially if it is at the end of a command substitution, and even the current psub will throw an error if it isn't used inside of a command substitution.
3. Bless the syntax (somecommand | psub --fifo &). At first appearance, it seems like this would have a race condition, where the substitution might happen before psub outputs the file name. But it seems like the way it actually works is it waits for stdout of the subcommand to complete, so maybe it just works fine?
4. Use an external command that basically does the equivalent of the following pseudocode:

   pipe = mkfifo() // or use /dev/fd
   if  fork() {
      exec(argv, stdin=pipe)
    } else {
      print(pipe.filename)
    }
   

   But this has the downside that you have to pass the command to run to this tool, which means it probably either needs to run fish to interpret the command, or be the fish executable itself.

I don't really love any of these though.

However, I think most of these could also offer a path forward for #1786 as well.

1. Just have similar syntax for output
2. psub, or a new command could have special behavior if it is at the beginning of a command substitution, although, that's a little weird, because psub needs to basically write to it's own stdout and the original stdout of the pipeline.
3. This one is tricky. One possibility could be (psub --fifo | cmd >/dev/null 2>&1 &) and have psub write the filename to stderr instead of stdout. But that is kind of gross.
4. Same command could have an option to forward stdout from the pipeline instead of stdin into it.