Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File "complete" #23

Open
crh3675 opened this issue Jan 30, 2016 · 5 comments
Open

File "complete" #23

crh3675 opened this issue Jan 30, 2016 · 5 comments

Comments

@crh3675
Copy link

crh3675 commented Jan 30, 2016

When we use this library (which is great), we still have to add some code to fs.stat the file once an inotify event occurs.

We must do this because larger files when copied come across the TCP pipe in smaller buffers until the file is completely written. We created some timeout events to make this work. Perhaps some attention can be put into addressing the idea of creating a custom EventEmitter or something of that nature when a file is completely written.

@yuanchuan
Copy link
Owner

Yes that would be quite useful, thanks for your suggestion :)

@victusfate
Copy link

for inspiration on polling, see here https://github.com/paulmillr/chokidar/blob/master/index.js#L265
ideally non-polling would be best, not sure how inotify delivers in_close events

@yuanchuan yuanchuan mentioned this issue Mar 16, 2016
@crh3675
Copy link
Author

crh3675 commented Jul 30, 2016

Just to note, we pass about 500,000 files through this library about 4 times a day (within 1 hour) with our custom timer check and it works flawlessly. Basically, stat the file when it comes in, set a timer for 500ms and stat again, if the size matches, presumedly the file is completely written. It hasn't failed yet. It may not be the best approach but it works for us

@crizstian
Copy link

@crh3675 hey could yo provide an example of you're aproach ?

@sandman45
Copy link

sandman45 commented Feb 15, 2019

I did something similar but just did a recursive function (used timeouts initially) but decided against it because I was able to check file.size and the previous size and its been working great since.
Having file complete event would be awesome then I dont need to do this code.

Here is my example

const checkFile = function (info, previousSize){
    fs.stat(info.path, (err, fileInfo) => {
        if (err === null) {
            if (fileInfo.size === previousSize && fileInfo.size > 0) {
                uploadFile(info, false);
            } else {
                checkFile(info, fileInfo.size);
            }
        } else {
            console.log(`File not found ${err}`);
        }
    });
};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants