• Version control system: VCS
  • Content-type-aware: the VCS is able to extract relevant structure from a file based on its file-type
  • API-driven: the VCS can be configured to apply updates through an API, rather than directly making changes to the filesystem.
  • 'Cloud content': content that is stored/read/updated on a third-party, eg, GDrive, GitHub, OnShape

The need for a content-type-aware VCS

Most VCS's I'm aware of, eg, git, svn, breezy, perform their diff-analysis on a line-by-line basis. (I assume because they're built primarily for versioning code?). This is a problem when one wants to version changes to minified JSON, images, or binary blobs (basically anything else) but where the exact text-encoding is irrelevant and the process of transforming the data into a line-aware format only adds computational and storage bloat.

For example, say I have a JSON object like so:


and I update it to:


Line-aware VCS's like the aforementioned simply store the entire line, including ["really","big","object"], for each snapshot. (Of course, with JSON in particular we can simply format entities with a newline in between, but this is not the case for every file type.) There is the option to store a series of <old_start_index>, <old_end_index>, <new_content> changes, but such a type-agnostic approach still penalizes trivial changes like 1 vs. "1".

The need for an API-driven VCS

This creates a problem when one want to update an in-memory object based on the contents of its corresponding updated file without having to completely reload the entire object into memory. For example,

  • loading 100k of fine-tuned parameters into a neural network's head instead of loading the entire 300M,
  • incrementally updating rectangular snapshots taken from satellite view of the earth whenever the user wishes to go forward or backward in time without reloading the globe every time, or
  • loading only the new data into a database without having to perform a complete dump and load every time an update is made.

The need to version 'cloud content'

Additionally, the filesystem metaphor breaks down when I'm trying to version content on a third-party service that offers the ability to version assets within their platform, but does not give me the option of downloading their content in a lossless format. This is a problem when my project stretches outside of the third-party service's platform and I want to keep all artifact versions in sync. For example:

  • Google docs/sheets/slides provide the ability to save and restore checkpoints, but their respective files are converted to a standard format when downloading, which destroys GSuite-specific features when re-uploading.
  • OnShape supports versioning and branching within a project, but unless you want to pay alot of money, it makes more sense to store and run your simulation code from a local git repository.
  • Entry-level CMS's often have to make a tradeoff between a synchronous git-backed architecture or an asynchronous DB-backed architecture. git-backed systems usually take longer to propagate, while DB-backed ones typically lack high-frequency version control.

I imagine one would integrate pointers to their 'cloud content' like so:

+ my_proj
|-- my_proj.gdrive # something like {"type":"gdrive", "id":"12345678901234567890"}
|-+ cad
  |-- part1.onshape # something like {"type":"onshape", "id":"09876543210987654321"}
  |-- part2.onshape # something like {"type":"onshape", "id":"12345678901234567890"}

Putting it all together

People version all kinds of file types on a daily basis, so I'm not asking for a VCS that can handle them all. Instead, this is what I'm asking: Are you aware of a version control system or wanna-be-in-progress that allows you to specify arbitrary diff and apply-changes algorithms, based on the type of content being versioned, including 'cloud content' (files that only point to content available through a third-party)?

  • ClearCase sort of does this Nov 15, 2022 at 2:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.