-
Notifications
You must be signed in to change notification settings - Fork 664
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fastGet Observations #1166
Comments
I can't reproduce the problem against an OpenSSH server. I've tried different node versions, I've tried forcing the SFTP connection to not assume OpenSSH so that it uses the different, lower max read size, and I've tried different file sizes. Every time the SHA1 hash matches on both ends. What is the ident of this server? Is it not OpenSSH, despite advertising OpenSSH SFTP extensions? |
I'd also be curious to know if the OpenSSH SFTP client exhibits the same corruption issue and if not, what the OpenSSH SFTP client debug output looks like for that transfer. |
Can you try the master branch of |
…ftp fastGet Related issue: mscdex/ssh2#1166
…e corrupted (#19) * fix: files which size is greater than 64KB are corrupted when using sftp fastGet Related issue: mscdex/ssh2#1166 * Update sftp.js
@mscdex I was able to replicate this issue on the latest version. 1gb file was consistently corrupted. I agree with @amitkirdatt that it is a chunk size negotiation issue but I am unsure how else to help. |
@Michael-Gibbons well, without any way for me to (reliably) reproduce the problem on my end, it's going to have to be up to others to debug the issue unfortunately. |
If it's any help I can confirm the problem does not exist on ssh2 v0.8.9 but it does exist on v1.13.0. Against the same remote server, of course. Also, here is the remote server software I think?
|
Had the exact same problem with both libraries. Thanks for sharing your solution! |
Hey guys, I had the same issue. However, having read the documentation again, I found that Here is the implementation: const _sftp_get_file = (sftp_conn, {remote_file_path, local_file_path}) => new Promise(resolve => {
let sftp_read_stream = sftp_conn.createReadStream(remote_file_path, {autoClose: true}),
write_stream = fs.createWriteStream(local_file_path),
downloaded_successfully = false,
_resolve = ({result = false, is_external_error = false, message = ''}) => resolve({
result,
is_external_error,
message,
})
;
console.log(`Download the file ${remote_file_path} => ${local_file_path} from SFTP...`);
sftp_read_stream.on('error', (err) => {
log_error(`Failed ReadStream: ${err.message}`, {err, remote_file_path});
_resolve({
result: false,
is_external_error: true,
message: `Failed to download a file from SFTP: ${err.message}`
});
});
write_stream.on('error', (err) => {
log_error(`Failed WriteStream: ${err.message}`, {err, local_file_path});
_resolve({ result: false });
});
write_stream.on('finish', () => {
console.log(`[!] File has been saved successfully`);
downloaded_successfully = true;
});
sftp_read_stream.on('close', () => {
downloaded_successfully
? _resolve({ result: true })
: 0 // means an Error has occurred
});
sftp_read_stream.pipe(write_stream);
}); |
Node version: v14.19.1
OS: MacOS 12.1
ssh2-sftp-client: 8.0.0
Issue: We were seeing files being corrupted when using
fastGet
.Resolution: We needed to adjust the chunkSize specific to the sftp server we are connecting to
We started out with ssh2-sftp-client but to narrow the problem down we tried
ssh2
as well. Originally posted here, but also posting in case it is useful to othersWe ended up solving the problem - Just sharing our experience:
Code using ssh2-sftp-client
The text file we are downloading is about ~86MB. We noticed that the file was consistently corrupted (data is missing).
To narrow down the issue we decided to whip up a quick test using
ssh2
We noticed the same corruption.
We then fired up the native sftp client in debug mode and noticed that the server rmax value was
32768
(same as the default chunkSize)We then added debug logging to ssh2
In the debug we noticed the following:
It looked liked the DATA was being broken up into 2 parts: 1 part was 31952 and the 2nd part was 816, which when added up is
32768
. We figured this was causing the issue.We then specified the chunkSize of 31952 for
fastGet
.We didn't see the DATA being broken up into two parts like before (31952 and 816).
The files are now downloading correctly.
I am sharing our experience in the hopes that somebody finds it useful.
ssh2-sftp-client updated code that worked for us:
ssh2 code:
The text was updated successfully, but these errors were encountered: