Terminology:
- Version control system: VCS
- Content-type-aware: the VCS is able to extract relevant structure from a file based on its file-type
- API-driven: the VCS can be configured to apply updates through an API, rather than directly making changes to the filesystem.
- 'Cloud content': content that is stored/read/updated on a third-party, eg, GDrive, GitHub, OnShape
The need for a content-type-aware VCS
Most VCS's I'm aware of, eg, git, svn, breezy, perform their diff-analysis on a line-by-line basis. (I assume because they're built primarily for versioning code?). This is a problem when one wants to version changes to minified JSON, images, or binary blobs (basically anything else) but where the exact text-encoding is irrelevant and the process of transforming the data into a line-aware format only adds computational and storage bloat.
For example, say I have a JSON object like so:
{"a":1,"b":["really","big","object"]}}
and I update it to:
{"a":2,"b":["really","big","object"]}}
Line-aware VCS's like the aforementioned simply store the entire line, including ["really","big","object"], for each snapshot. (Of course, with JSON in particular we can simply format entities with a newline in between, but this is not the case for every file type.) There is the option to store a series of <old_start_index>, <old_end_index>, <new_content> changes, but such a type-agnostic approach still penalizes trivial changes like 1 vs. "1".
The need for an API-driven VCS
This creates a problem when one want to update an in-memory object based on the contents of its corresponding updated file without having to completely reload the entire object into memory. For example,
- loading 100k of fine-tuned parameters into a neural network's head instead of loading the entire 300M,
- incrementally updating rectangular snapshots taken from satellite view of the earth whenever the user wishes to go forward or backward in time without reloading the globe every time, or
- loading only the new data into a database without having to perform a complete dump and load every time an update is made.
The need to version 'cloud content'
Additionally, the filesystem metaphor breaks down when I'm trying to version content on a third-party service that offers the ability to version assets within their platform, but does not give me the option of downloading their content in a lossless format. This is a problem when my project stretches outside of the third-party service's platform and I want to keep all artifact versions in sync. For example:
- Google docs/sheets/slides provide the ability to save and restore checkpoints, but their respective files are converted to a standard format when downloading, which destroys GSuite-specific features when re-uploading.
- OnShape supports versioning and branching within a project, but unless you want to pay alot of money, it makes more sense to store and run your simulation code from a local git repository.
- Entry-level CMS's often have to make a tradeoff between a synchronous git-backed architecture or an asynchronous DB-backed architecture. git-backed systems usually take longer to propagate, while DB-backed ones typically lack high-frequency version control.
I imagine one would integrate pointers to their 'cloud content' like so:
+ my_proj
|-- my_proj.gdrive # something like {"type":"gdrive", "id":"12345678901234567890"}
|-+ cad
|-- part1.onshape # something like {"type":"onshape", "id":"09876543210987654321"}
|-- part2.onshape # something like {"type":"onshape", "id":"12345678901234567890"}
Putting it all together
People version all kinds of file types on a daily basis, so I'm not asking for a VCS that can handle them all. Instead, this is what I'm asking: Are you aware of a version control system or wanna-be-in-progress that allows you to specify arbitrary diff and apply-changes algorithms, based on the type of content being versioned, including 'cloud content' (files that only point to content available through a third-party)?
