How to know if there is stdout/stderr beforehand

[isamu-isozaki]

Hi! This is a bit of a follow up question to #714 and I'm not sure if this is even possible, but in a process made with create_process, is there a way to know whether

1. The process outputs something in stdout/stderr or just goes for next stdin
2. Currently I'm doing

async def get_command_output(process):
    stdout_task = asyncio.create_task(process.stdout.read(MAX_BYTES))
    stderr_task = asyncio.create_task(process.stderr.read(MAX_BYTES))
    done, pending = await asyncio.wait(
        {stdout_task, stderr_task},
        return_when=asyncio.FIRST_COMPLETED
    )
    return done

to get the output, but is there a clever way to do this instead? I know there are sessions with callback for data_recieved but I was curious if there is a way through the process/if there is a known way to adapt a session to be like a process

Sorry for the basic qs and again thanks for the awesome library!

[ronf]

Without knowing something about the output of the command (or shell) you are running, there's no way to know when to stop reading from stdout or stderr and write more data to stdin. As for stdout vs. stderr, there's really no way to know which will get written to. However, if you don't care about keeping the stdout and stderr data separate, you can combine them into a single stream by passing an argument of stderr=asyncssh.STDOUT to create_process(). After that all stderr output received will be written to stdout instead, so you only have one output stream to read.

Your options here are basically one of the following:

1. Know something about the expected output (like what a shell prompt looks like) and read stdout until you see an indication that the output is done and that you should write more data to stdin.
2. Set up separate tasks to write data to stdin and read from stdout and stderr in parallel, if the command you are running doesn't mind you writing data before it is ready to consume it.
3. If the amount of input and output is small enough, you can use something like conn.run() and give it a command and data to write to stdin and let AsyncSSH take care of collecting the input in parallel on stdout and stderr, returning when the command finishes and stdin/stdout/stderr are closed. This will give you back all of the output on both stdout and stderr, as well as other things like the exit status of the command or the exit signal it sent back.

[isamu-isozaki]

@ronf Thanks for the quick reply! Love the way to combine outputs and thanks I think I'll try 2. I'm trying to make a way to interact with terminals from purely python so I can technically know the stdout/stdin/stderr responses of some commands, I don't think I can do all of them. Also, I want to be able to stay in the shell. For example, I don't think I can do su to change user and do subsequent commands for that user in conn.run() since if I understand right, each command is a new bash script.
But I think overall you are 100% right, the best choice might be to figure out the output pattern for all the possible commands and make a script for each of them maybe with an if statement or at least for each category.
I'll try this tomorrow and mark this as the answer.

[ronf]

One other thing I should mention: If you request a PTY or set the term_type for a process/session, it will automatically combine stdout and stderr, and you won't need the option I mentioned above. However, if you don't want a PTY, the stderr redirect will give you a similar result in terms of combining the streams.

Having a PTY generally helps if you are trying to run an interactive shell, especially around things like printing prompts, which you can then search for by using readuntil() to look for that shell prompt in the resulting output. That way, you don't have to know the exact output of specific commands, as long as those commands never output something matching the prompt you set on the shell.

You could also do things like sending a command of something like some_command; echo '*****some_unique_string*****' as a way to get a unique string you can match on at the end of the output, if you don't think the shell prompt is unique enough.

You're right about conn.run() running everything in a separate session, which means a fresh shell every time. As such, it can't be used when you want to do something like 'su' to get a root shell.

[isamu-isozaki]

@ronf sorry, one follow up q. How do I get output from term_type properly? I have the following script without term_type

import asyncio
import asyncssh
import sys
import time
import logging
# logging.basicConfig(level='DEBUG')
# asyncssh.set_debug_level(2)
async def run_client() -> None:
    async with asyncssh.connect('192.168.50.11', username='lowpriv', password='trustno1', known_hosts=None) as conn:
        try:
            result = await conn.run('echo "Hello!"', check=True)
        except asyncssh.ProcessError as exc:
            print(exc.stderr, end='')
            print(f'Process exited with status {exc.exit_status}',
                file=sys.stderr)
        else:
            print(result.stdout, end='')
        try:
            # Create a process with `su` command
            async with conn.create_process('su root', stderr=asyncssh.STDOUT) as process:
                print("Started process")

                result = await process.stdout.read(999)

                print("su result: ", result)
                process.stdin.write('aim8Du7h' + '\n')  # Send password


                # Wait for the prompt and output
                # Read the output of the `su` command
                process.stdin.write('id' + '\n')

                result = await process.stdout.read(999)
                print("id result: ", result)


            print("Ended")
        except asyncssh.ProcessError as exc:
            print(f"Error running su command: {exc.stderr}", file=sys.stderr)

# If you're in an environment where an event loop is already running (like Jupyter or IPython),
# you can simply use 'await' instead of 'run_until_complete'
try:
    # In Jupyter, use 'await' directly to run the coroutine
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SSH connection failed: ' + str(exc))

and without all the debugging code, I get output

Hello!
Started process
su result:  Password:
id result:  uid=0(root) gid=0(root) groups=0(root)

when I put in term_type, I noticed I needed another stdout in between and I got

Hello!
Started process
su result:  Password:
password result:

id result:  id

Ended

The code with term_type is

import asyncio
import asyncssh
import sys
import time
import logging
# logging.basicConfig(level='DEBUG')
# asyncssh.set_debug_level(2)
async def run_client() -> None:
    async with asyncssh.connect('192.168.50.11', username='lowpriv', password='trustno1', known_hosts=None) as conn:
        try:
            result = await conn.run('echo "Hello!"', check=True)
        except asyncssh.ProcessError as exc:
            print(exc.stderr, end='')
            print(f'Process exited with status {exc.exit_status}',
                file=sys.stderr)
        else:
            print(result.stdout, end='')
        try:
            # Create a process with `su` command
            async with conn.create_process('su root', stderr=asyncssh.STDOUT, term_type="ansi") as process:
                print("Started process")

                result = await process.stdout.read(999)

                print("su result: ", result)
                process.stdin.write('aim8Du7h' + '\n')  # Send password
                result = await process.stdout.read(999)
                print("password result: ", result)


                # Wait for the prompt and output
                # Read the output of the `su` command
                process.stdin.write('id' + '\n')

                result = await process.stdout.read(999)
                print("id result: ", result)


            print("Ended")
        except asyncssh.ProcessError as exc:
            print(f"Error running su command: {exc.stderr}", file=sys.stderr)

# If you're in an environment where an event loop is already running (like Jupyter or IPython),
# you can simply use 'await' instead of 'run_until_complete'
try:
    # In Jupyter, use 'await' directly to run the coroutine
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SSH connection failed: ' + str(exc))

Sorry for all the qs but I don't think I understand the output of the id part. Also I am assuming you meant

python -c 'import pty; pty.spawn("/bin/bash")'

for doing the pty and with testing this somewhat works but

1. I still wasn't able to figure out how to get the output for id
2. I didn't want to assume the target box has python installed(or assume we can install python on the system).
   For the above, below is the script I used with the output

import asyncio
import asyncssh
import sys
import time
import logging
# logging.basicConfig(level='DEBUG')
# asyncssh.set_debug_level(2)
async def run_client() -> None:
    async with asyncssh.connect('192.168.50.11', username='lowpriv', password='trustno1', known_hosts=None) as conn:
        try:
            result = await conn.run('echo "Hello!"', check=True)
        except asyncssh.ProcessError as exc:
            print(exc.stderr, end='')
            print(f'Process exited with status {exc.exit_status}',
                file=sys.stderr)
        else:
            print(result.stdout, end='')
        try:
            # Create a process with `su` command
            async with conn.create_process("""python3 -c 'import pty; pty.spawn("/bin/bash")'""", stderr=asyncssh.STDOUT) as process:
                print("Started process")
                result = await process.stdout.read(999)

                print("pty result: ", result)
                process.stdin.write("su root"+ "\n")
                result = await process.stdout.read(999)
                print("su result: ", result)
                result = await process.stdout.read(999)
                print("password result: ", result)



                process.stdin.write('aim8Du7h' + '\n')  # Send password


                # Wait for the prompt and output
                # Read the output of the `su` command
                process.stdin.write('id' + '\n')

                result = await process.stdout.read(999)
                print("id result: ", result)
                result = await process.stdout.read(999)
                print("id result: ", result)
                process.stdin.write('id' + '\n')
                print("id result: ", result)
                


            print("Ended")
        except asyncssh.ProcessError as exc:
            print(f"Error running su command: {exc.stderr}", file=sys.stderr)

# If you're in an environment where an event loop is already running (like Jupyter or IPython),
# you can simply use 'await' instead of 'run_until_complete'
try:
    # In Jupyter, use 'await' directly to run the coroutine
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SSH connection failed: ' + str(exc))

output

Hello!
Started process
pty result:  lowpriv@target:~$
su result:  su root

password result:  Password:
id result:

id result:  root@target:/home/lowpriv#
id result:  root@target:/home/lowpriv#
Ended

[ronf]

The way you are setting term_type on the create_process() call looks ok. Note that you should no longer need the stderr=asyncssh.STDOUTonce you've done that, as settingterm_type` automatically requests a PTY, and that causes stdout and stderr to automatically be merged.

Your sample code seems to be sending the "id" command twice, and the read() call seems to be in the wrong place relative to the second write() of the command. The real problem here, though, is that you need to keep reading data in a loop to make sure you get all of it. This is why it is important to have some kind prompt you can wait for. That way, instead of just calling read(999), you can do something like a readuntil() call which matches the prompt and the echoing of the command line back.

As a test, I did the following:

            proc.stdin.write('id\n')
            for i in range(1,10):
                print(i, repr(await proc.stdout.read(999)))
                await asyncio.sleep(0.01)

This actually broke the output into 3 separate reads. The first just echoed back 'id' and a CRLF. The second echoed a shell prompt and the "id" command again. Then, the output of the "id" command showed up in a third read, followed by a CRLF and another shell prompt.

If I change the code to wait for a shell prompt before sending "id", I see that first shell prompt in the first read() call (before writing "id"), followed by "id" and a CRLF in the second read and the command output in a third read along with a CRLF, and another shell prompt appears in a fourth read. This isn't completely predictable, though. It all depends on when the output comes out relative to the short sleeps I added. The sleeps aren't the right way to do this -- I just used that as a quick way to show that the output is not sent all at once, and so you can't expect to read it all at once.

A closer to proper example would be something like:

        PROMPT='sh-3.2$ '

        async with conn.create_process('sh', term_type='ansi') as proc:
            # Wait for shell prompt
            await proc.stdout.readuntil(PROMPT)

            # Write command
            proc.stdin.write('id\n')

            # Read back and throw away echo of command
            await proc.stdout.readline()

            # Read output of command and next shell prompt
            output = await proc.stdout.readuntil(PROMPT)

            # Print output of command with prompt stripped off the end
            print(output[:-len(PROMPT)], end='')

Note that I ran "sh" here explicitly, just to make the prompt more predictable and to avoid some noise introduced when I run my default shell, where it adds some extra output before and after the command output.

[isamu-isozaki]

@ronf Thank you so much! One weird bug/feature I found was for readuntil, for me, it just stopped reading until Prompt(ex, root@target instead of the entire line which is root@target:/home/lowpriv# id which might be an issue with where to divide) but I was able to get the output!

[ronf]

What were you passing to readuntil() as the prompt to look for? You need to pass the entire prompt, or at least a unique suffix of the prompt that won't appear anywhere else in the output. If you match on a prefix or substring of the prompt, it will match earlier, and that can cause a problem with the extra output becoming a part of the next read.

[isamu-isozaki]

@ronf Ah, then it might be just an issue with my usecase where I wanted the readuntil to read until the newline of that unique pattern. I noticed after I do su root the sh becomes root@target:/home/lowpriv# so I waited until # including the command. My understanding is that without the readuntil, where I just printed out all the lines, I got

one line id

one line root@target:/home/lowpriv# id

uid=0(root) gid=0(root) groups=0(root)

when I did

proc.stdin.write('id\n')

# # Read back and throw away echo of command
one_line = await proc.stdout.readline()
print("one line", one_line)
one_line = await proc.stdout.readline()
print("one line", one_line)
one_line = await proc.stdout.readline()
print("one line", one_line)

here with readuntil, what I get is

wait until  id
root@target:/home/lowpriv#
next line   id

final line
uid=0(root) gid=0(root) groups=0(root)

from

proc.stdin.write('id\n')

# # # Read output of command and next shell prompt
output = await proc.stdout.readuntil("#")
print("wait until ", output)
output = await proc.stdout.readline()
print("next line ", output)
output = await proc.stdout.readline()
print("final line")
print(output)

So you are absolutely right, this is not an issue with readuntil. I think just an issue with my usecase. I think for my case, I can just readline+divide each line by new character, and detect if # or $ is in there etc. And again thanks for all the support happy to hear suggestions etc

[ronf]

It's not a good idea to look for a single character, as output containing something like '$' or '#' will end up matching and incorrectly look like the end of a command's output.

If you want to process the data a line at a time, you can either get the entire output (with readuntil with a more unique prompt match) and then split the resulting string on CRLF bytes, or you could use the version of readuntil where you pass in multiple separators. For example:

        PROMPT='sh-3.2$ '
        CRLF='\r\n'

        async with conn.create_process('sh', term_type='ansi') as proc:
            # Wait for shell prompt
            await proc.stdout.readuntil(PROMPT)

            # Write command
            proc.stdin.write('env\n')

            # Read back echo of command
            await proc.stdout.readline()

            while True:
                # Read lines of output until the next shell prompt
                output = await proc.stdout.readuntil((PROMPT, CRLF))

                # Break out of loop when you see the shell prompt
                if output == PROMPT:
                    break

                # Print output of command with CRLF stripped off
                print(output[:-2])

You shouldn't need to worry about the initial prompt being different, as you are running the "su" command instead of a shell at the start. So, the only shell which prints a prompt should be the one after you become root.