I am looking at possible improvements to the NPM library called trash. As of v6.0.0,
npm test fails on my system. This is because my
/tmp is on a different mounted filesystem to my
/home directory. The failing test tries to trash a symlink file in a
/tmp folder, which requires copying it into a special subdirectory of my
/home directory. But the copy function used does not support symlinks.
For example, if
trash happened to be written in
python, it could probably have used shutils.move. That would be good enough, although not perfect. The documentation has a nice prominent warning that it does not copy ACL metadata. I would accept some limitations about copying metadata. As a bare minimum, I want to see execute permission being preserved.
I am very new to Node.js :-). Is there a good existing Node package, that would support
- moving files and directories
- moving symlinks
- "moving" between different filesystems, i.e. copy+delete
- copying the execute permission bit when necessary
What I have looked at so far
move-files, by the same author. It uses
fs.copyFile. This means it does not support copying directories or symlinks.
ncppackage (currently on version 2.0.0), and anything that relies on it for copying, seems very questionable to me. See issue #100 for references. If it was just that the code was frozen, that could be OK, but there seem to be core bug(s) and not enough effort to address them. (Either to analyze the bugs, or to review submitted patches; I haven't looked in detail).
cpr(and hence anything that uses it for copying) is not aware of symlinks. This means it will copy the symlink target. This is not what you want when you are moving a directory hierarchy.
fs-extrahas methods for moving files, and seems popular. It's just taking me a lot of time to get my head around its unconditional use of
graceful-fs. To make sure I'm phrasing specific questions about recommendation:
Would anyone have any reservations about the use of
fs-extrabeyond those explicitly mentioned in the documentation of the two packages?
- Do you have something other than
fs-extra, that you would suggest/recommend?
- Or on the contrary, would you suggest using
fs-extrain this case? If so, do you have a short analysis of why the use of
graceful-fsis not a significant concern here? E.g. something I might write as a comment, if I wanted to say that it works OK in this case, but that I would not unconditionally approve of these packages and their documentation?
everything beyond this point is the rant about graceful-fs
graceful-fs helps run high-concurrency file operations, particularly on MacOS where the default limit on maximum open files is on the low side. On the other hand, it means that a specific error -
EMFILE "too many open files" - will be replaced with delays. The documentation does not mention a timeout, so it implies it could deadlock in certain cases. I'm not aiming to implement such high-concurrency copies (and if something provided them by default, I wouldn't even want to accept it, without seeing exhaustive benchmarks and testing).
graceful-fs also suppresses a small number of other errors.
(A side question occurs to me - how much sense does it make to use
graceful-fs, if you don't opt-in to monkey-patch the main
fs module? If you have certain threads of operation that need to worry about hitting
EMFILE, surely you have to worry about them triggering
EMFILE in all the other threads of operations as well?
I looked briefly at the source code of
graceful-fs. It's not too large. I don't see it doing anything with timeouts. And it does not limit the number of open files that it uses - i.e. it does not deliberately reserve any open files for code which does not use
A case study for using graceful-fs: chokidar and readdirp
So I look beyond the official documentation of
graceful-fs. With a simple web search, I only really found one use of words like "hang" with respect to
graceful-fs. So I did not find evidence of complaints about this :-). On the other hand, the story I found fit very nicely into my prejudices :-).
that's kind of the point. (That is, the open will hang until you close something else.)
- the discussion eventually ended as follows -
I've started a fork of chokidar - graceful-chokidar - that uses graceful-fs instead.You may want to give chokidar 0.10 a try. Assuming you were using fsevents mode, your EMFILEs were probably coming from https://github.com/paulmillr/chokidar/blob/0.9.0/index.js#L411-L412. That part has been refactored now, replacing `recursive-readdir` with `readdirp`, which itself uses `graceful-fs`. I tried watching a huge directory (>250K files) on my system and, as the flood of add events were flowing, I observed the sorts of pauses one would expect from the graceful-fs behavior kicking in. In the meantime, changes were still detected properly, and performance was great - cpu utilization stayed low. I don't think it will make sense to switch all of chokidar to graceful-fs because it creates a risk of locking up the program in certain watch modes that hold open too many file descriptors, as described above. In this case it seems better to let the EMFILE happen, resulting in the user educating themselves about ulimit, etc.
It gets even better. The current version of chokidar uses the 3.x series of readdirp. This current series of
readdirp is mostly just 300 lines in
index.js. The dependency on
graceful-fs is gone (the only dependency is
picomatch). AFAICT, the revised
readdirp functions only perform one readdir() call at a time. They don't tend to hit the open file limit, because they do not open multiple directories at once. It appears that the whole issue was self-inflicted. It wasn't even the only problem with the approach they were using.
I've been using chokidar (shout out to @paulmillr and @es128) on a project and noticed that it causes my application to be unresponsive until the initial walk is complete. I traced the performance problem down to readdirp's extremely aggressive strategy in making massive numbers of filesystem requests in parallel.
The important insight that this issue is related to is that flooding the file system is not a good approach in general.
I might even say that uncontrolled parallelism is asking for trouble (and is the source of all of the work arounds I referenced above and and issues I've seen across libraries) so it should probably not be the default setting because it doesn't scale in general cases and doesn't play nice with other things that might be running.
I might even go as far as saying, it might not even need to be an option and if you remove this possibility, you can remove using graceful-fs, back pressure, etc.
Maybe I'm missing something...
(The record so far suggests he was not :-P)