0

I am looking at possible improvements to the NPM library called trash. As of v6.0.0, npm test fails on my system. This is because my /tmp is on a different mounted filesystem to my /home directory. The failing test tries to trash a symlink file in a /tmp folder, which requires copying it into a special subdirectory of my /home directory. But the copy function used does not support symlinks.

For example, if trash happened to be written in python, it could probably have used shutils.move. That would be good enough, although not perfect. The documentation has a nice prominent warning that it does not copy ACL metadata. I would accept some limitations about copying metadata. As a bare minimum, I want to see execute permission being preserved.

I am very new to Node.js :-). Is there a good existing Node package, that would support

  1. moving files and directories
  2. moving symlinks
  3. "moving" between different filesystems, i.e. copy+delete
  4. copying the execute permission bit when necessary

What I have looked at so far

  • trash currently uses move-files, by the same author. It uses fs.copyFile. This means it does not support copying directories or symlinks.

  • The ncp package (currently on version 2.0.0), and anything that relies on it for copying, seems very questionable to me. See issue #100 for references. If it was just that the code was frozen, that could be OK, but there seem to be core bug(s) and not enough effort to address them. (Either to analyze the bugs, or to review submitted patches; I haven't looked in detail).

  • cpr (and hence anything that uses it for copying) is not aware of symlinks. This means it will copy the symlink target. This is not what you want when you are moving a directory hierarchy.

  • fs-extra has methods for moving files, and seems popular. It's just taking me a lot of time to get my head around its unconditional use of graceful-fs. To make sure I'm phrasing specific questions about recommendation:

  • Would anyone have any reservations about the use of graceful-fs in fs-extra beyond those explicitly mentioned in the documentation of the two packages?

  • Do you have something other than fs-extra, that you would suggest/recommend?
  • Or on the contrary, would you suggest using fs-extra in this case? If so, do you have a short analysis of why the use of graceful-fs is not a significant concern here? E.g. something I might write as a comment, if I wanted to say that it works OK in this case, but that I would not unconditionally approve of these packages and their documentation?

everything beyond this point is the rant about graceful-fs

graceful-fs helps run high-concurrency file operations, particularly on MacOS where the default limit on maximum open files is on the low side. On the other hand, it means that a specific error - EMFILE "too many open files" - will be replaced with delays. The documentation does not mention a timeout, so it implies it could deadlock in certain cases. I'm not aiming to implement such high-concurrency copies (and if something provided them by default, I wouldn't even want to accept it, without seeing exhaustive benchmarks and testing). graceful-fs also suppresses a small number of other errors.

(A side question occurs to me - how much sense does it make to use graceful-fs, if you don't opt-in to monkey-patch the main fs module? If you have certain threads of operation that need to worry about hitting EMFILE, surely you have to worry about them triggering EMFILE in all the other threads of operations as well?

I looked briefly at the source code of graceful-fs. It's not too large. I don't see it doing anything with timeouts. And it does not limit the number of open files that it uses - i.e. it does not deliberately reserve any open files for code which does not use graceful-fs)

A case study for using graceful-fs: chokidar and readdirp

So I look beyond the official documentation of graceful-fs. With a simple web search, I only really found one use of words like "hang" with respect to graceful-fs. So I did not find evidence of complaints about this :-). On the other hand, the story I found fit very nicely into my prejudices :-).

that's kind of the point. (That is, the open will hang until you close something else.)

- the discussion eventually ended as follows -

I've started a fork of chokidar - graceful-chokidar - that uses graceful-fs instead.
You may want to give chokidar 0.10 a try. Assuming you were using fsevents mode, your EMFILEs were probably coming from https://github.com/paulmillr/chokidar/blob/0.9.0/index.js#L411-L412. That part has been refactored now, replacing `recursive-readdir` with `readdirp`, which itself uses `graceful-fs`. I tried watching a huge directory (>250K files) on my system and, as the flood of add events were flowing, I observed the sorts of pauses one would expect from the graceful-fs behavior kicking in. In the meantime, changes were still detected properly, and performance was great - cpu utilization stayed low. I don't think it will make sense to switch all of chokidar to graceful-fs because it creates a risk of locking up the program in certain watch modes that hold open too many file descriptors, as described above. In this case it seems better to let the EMFILE happen, resulting in the user educating themselves about ulimit, etc.

It gets even better. The current version of chokidar uses the 3.x series of readdirp. This current series of readdirp is mostly just 300 lines in index.js. The dependency on graceful-fs is gone (the only dependency is picomatch). AFAICT, the revised readdirp functions only perform one readdir() call at a time. They don't tend to hit the open file limit, because they do not open multiple directories at once. It appears that the whole issue was self-inflicted. It wasn't even the only problem with the approach they were using.

https://github.com/paulmillr/readdirp/issues/35

I've been using chokidar (shout out to @paulmillr and @es128) on a project and noticed that it causes my application to be unresponsive until the initial walk is complete. I traced the performance problem down to readdirp's extremely aggressive strategy in making massive numbers of filesystem requests in parallel.

[...]

The important insight that this issue is related to is that flooding the file system is not a good approach in general.

I might even say that uncontrolled parallelism is asking for trouble (and is the source of all of the work arounds I referenced above and and issues I've seen across libraries) so it should probably not be the default setting because it doesn't scale in general cases and doesn't play nice with other things that might be running.

I might even go as far as saying, it might not even need to be an option and if you remove this possibility, you can remove using graceful-fs, back pressure, etc.

Maybe I'm missing something...

(The record so far suggests he was not :-P)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.