1

I am wondering if anyone knows of a Java library that can take an input stream and spool it to disk, while allowing it to be read.

I am dealing with an issue where I need to stream very large files, and there is a server side timeout if nothing pulls from the stream in X amount of time.

Currently I read a chunk from the stream, process it, and then grab another chunk. This was working fine until I had an issue where the processing of a chunk of data from the stream took long enough for the stream to timeout.

I could fix this by downloading the stream fully before I process it, but that would mean I cannot start processing until the file is fully downloaded.

I am looking for a library that will download the input stream to disk, but allow the input stream to read while the download is happening. I would expect it to block if the input stream hits EOF but the download is still processing.

I could write this myself, but I was hoping someone had already gone through the trouble.

Improve this question
3
  • Would PipedInputStream/PipedOutputStream be a possibility? You can specify a quite large buffer (I have not tried). docs.oracle.com/javase/7/docs/api/java/io/PipedInputStream.html
    – Thorbjørn Ravn Andersen
    Oct 11, 2017 at 19:27
  • Your first line speaks more of the limitations of the OS than Java. Am I correct to suppose you are having this issue on Windows? As on Linux this is a non-issue. As recommended in the comment above you process the file in memory before writing it to disk then the whole disk issue can be avoided, unless you need to share between processes in which case there may be no simple solution (many 'clever' solutions of course are possible).
    – Quaternion
    Nov 3, 2017 at 2:02
  • It is not that I am having an issue, basically I am looking for something that allows me to read a file while it is downloading. The problem is that if my software takes too long before it calls read, the download connection will time out. I want something to be continuously downloading the file to disk, while providing a stream.
    – dspiegs
    Nov 7, 2017 at 15:33

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.