BufferException: Underflow error

[nimantha-lakshan]

Version

2.12.1

Bug description

Caused by: org.apache.sshd.common.util.buffer.BufferException: Underflow: requested=1668243314, available=39
	at org.apache.sshd.common.util.buffer.Buffer.ensureAvailable(Buffer.java:661) ~[sshd-common-2.12.1.jar:2.12.1]
	at org.apache.sshd.common.util.buffer.ByteArrayBuffer.getString(ByteArrayBuffer.java:241) ~[sshd-common-2.12.1.jar:2.12.1]
	at org.apache.sshd.common.util.buffer.Buffer.getString(Buffer.java:332) ~[sshd-common-2.12.1.jar:2.12.1]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.readAttributes(AbstractSftpClient.java:439) ~[sshd-sftp-2.12.1.jar:2.12.1]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributesResponse(AbstractSftpClient.java:331) ~[sshd-sftp-2.12.1.jar:2.12.1]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.checkAttributes(AbstractSftpClient.java:325) ~[sshd-sftp-2.12.1.jar:2.12.1]
	at org.apache.sshd.sftp.client.impl.AbstractSftpClient.stat(AbstractSftpClient.java:990) ~[sshd-sftp-2.12.1.jar:2.12.1]
	at org.apache.sshd.sftp.client.fs.SftpClientDirectoryScanner.scan(SftpClientDirectoryScanner.java:125) ~[sshd-sftp-2.12.1.jar:2.12.1]
	at org.apache.sshd.sftp.client.fs.SftpClientDirectoryScanner.scan(SftpClientDirectoryScanner.java:115) ~[sshd-sftp-2.12.1.jar:2.12.1]

code example:

    private void process(String host, int port, String username, String password, String homePath) throws IOException {
        try (SshClient sshClient = SshClient.setUpDefaultClient()) {
            sshClient.setServerKeyVerifier(AcceptAllServerKeyVerifier.INSTANCE);
            sshClient.setHostConfigEntryResolver(HostConfigEntryResolver.EMPTY);
            sshClient.setKeyIdentityProvider(KeyIdentityProvider.EMPTY_KEYS_PROVIDER);
            sshClient.start();

            // Connect to the SFTP server
            try (ClientSession session = sshClient.connect(username, host, port).verify(Duration.ofSeconds(30)).getSession()) {
                session.addPasswordIdentity(password);

                log.info("Connected to {}:{}. Authenticating...", host, port);
                var verify = session.auth().verify(Duration.ofSeconds(30));
                log.info("Authentication {}", verify.isSuccess() ? "success" : "failure");

                try (SftpClient sftpClient = SftpClientFactory.instance().createSftpClient(session)) {

                    var directoryScanner = new SftpClientDirectoryScanner(homePath, "*.csv");
                    directoryScanner.setCaseSensitive(false);
                    var scan = directoryScanner.scan(sftpClient); // throws error from here
                }
            }
        }
    }

Actual behavior

SFTP client not working as expected.

I'm receiving this error org.apache.sshd.common.util.buffer.BufferException: Underflow: requested=1668243314, available=39
This happens only with some SFTP servers.

Expected behavior

To resolve this issue

Relevant log output

No response

Other information

No response

[tomaswolf]

Which servers? Which SFTP protocol version? Use debug or trace logging to capture the content of the response to the STAT request, then provide the log. (Or run the client with a debug breakpoint in AbstractSftpClient.readAttributes() and post the buffer contents as shown in the debugger.)

[nimantha-lakshan]

SFTP protocol version: 6

[tomaswolf]

Next time please provide the buffer content as copyable text. A screenshot is extremely cumbersome to work with.

What server is this? It appears to be broken.

If I parse this buffer content as an SFTP v6 SSH_FXP_ATTR packet, I cannot match it against the draft RFC.

• 0 0 0 69 Length of following data
• 105 SSH_FXP_ATTR
• 0 0 0 102 Request ID.
• 0 0 -1 -1 Flags. hex 0x00_00_FF_FF. Strange that all bits are set. In particular, 0x2 should not occur.
• 2 File type; 2 is a directory
• 0 0 0 0 0 0 16 0 File size, 0x1000 = 4096 bytes
• And now things go wrong: because bit 0x400 is set in the flags, we read the next 8 bytes as "allocation size", but its not present. Instead the buffer continues with
• 0 0 0 11 Length of the string following (11 bytes, apparently owner name)
• 105 116 101 115 99 111 95 114 111 111 116 = "itesco_root"
• 0 0 0 8 String length
• 102 116 112 103 114 111 117 112 = "ftpgroup"
• ...

Because we read an allocation size (due to the wrong flags), we skip 8 bytes and end up reading some of the characters (bytes 99 111 95 114, hex 0x63_6F_5F_72) as a string length, and then the exception occurs.

What server is this? It is broken; it sends invalid flags.

We cannot do anything about this in the client side. File a bug report against that server, and try to avoid using SFTP v6 for that server: force SFTP v3.

[nimantha-lakshan]

I'm sorry for just attaching a screenshot.

This is our internal SFTP server (ProFTPD), which is old. I'll let our team know about this. I managed to avoid the exception by using SFTP version 3 in the client.

Thank you very much for your help.

[nimantha-lakshan]

I'll close this issue

[tomaswolf]

BTW: this is proftpd/proftpd#578 . Was fixed in proftpd version 1.3.7rc1 in October 2019.

[tomaswolf]

There is a minor bug in Apache MINA sshd, though: when we request file attributes, we tell the server we're interested in all items:

mina-sshd/sshd-sftp/src/main/java/org/apache/sshd/sftp/client/impl/AbstractSftpClient.java

Lines 1027 to 1029 in 5a78e6d

	if (version >= SftpConstants.SFTP_V4) {
	buffer.putInt(SftpConstants.SSH_FILEXFER_ATTR_ALL);
	}

This sends 0x00_00_FF_FF. But this is not correct. In SFTP >= v4, bit 0x2 MUST NOT be set:

0x00000002 MUST NOT appear in the mask. draft RFC

Therefore re-opening to fix this.

[tomaswolf]

Done in commit 37d239e.