3

Assume you have files locally present (already downloaded, retrieval unrelated to problem) and you know or are very certain that they have something (some blocks of data) in common with a torrent you want to download. Is there a client which allows reading a (user selected) selection of files, comparing it against the list of block checksums of the torrent file and skipping blocks based on locally present data.

An example would be a live system ISO of an OS of which you know (or guess) that it didn't change 50 or 30 or 10 percent of its content in comparison to a newer version. Normally you'd use zsync if possible, but that depends on a central server. How would I save the traffic with the same idea zsync is based on in a bittorrent architecture?

The only prerequisite for server and client should be that they speak bittorrent and arbitrary bittorrent or magnetlinks should be usable with the solution.

I know about (solutions which don't comply with prerequisite):

  • zsync which does this job outside the bittorrent world. As there's no checksum transfer in HTTP, that has to be handled in zsync with .zsync files.
  • jigdo which is great for updating only relevant parts of (Debian) iso images, but it's not integrated into bittorrent
  • Deduplicating filesystems like btrfs or ZFS offer deduplicated streamed transmission of subvolumes/datasets (e.g. btrfs send and zfs send), but they're now really suitable for simple download tasks.
Improve this question
2
  • Don't quite get what you want. Any client can rehash local data. If full block (block length and start position are defined in the torrent file) is the same as original, then it will be marked as present and won't be redownloaded.
    – Smit Johnth
    Commented Nov 17, 2014 at 6:37
  • @SmitJohnth That applies when the locally present data belongs to the target file. I want to scan an arbitrary set of local data (especially files which don't have anything to do with the torrent except having some blocks of data in common) and reuse it in order to skip the transfer.
    – Kalle Richter
    Commented Dec 7, 2014 at 5:02

1 Answer 1

Reset to default
1

What you'll need to do is read through the torrent file and compare all the hashes with a hash table of files you have on your local system.

Then, rebuild a local directory tree that is identical to what is in the torrent file. Afterwards, when you access the torrent with a manager, it will see the existing files and on retrieve non-existing ones.

Improve this answer
2
  • That's for implementing my own solution (application or even better reusable library and application on top), right?
    – Kalle Richter
    Commented Apr 1, 2015 at 12:02
  • @Karl Richter - Yup. Essentially, that's what Bittorrent Sync does. Except, it uses an sqllite database to store all the relative information for maintaining synchronized changes across differing nodes.
    – naisanza
    Commented Apr 1, 2015 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.