2

I am doing a Master's thesis and trying to decide the best way to organize my MATLAB data processing files. I want to ask for advice on the best versioning system to use. What would be the appropriate community? My work involves the following:

  1. I work alone (no one else is adding to the code ever)
  2. However, I use two different systems regularly (desktop at the office, laptop at home) and right now I'm just syncing with Google Drive.
  3. I have around 100 files for various functions related to data importing, processing, and producing plots.

I am starting to get tired of the uncertainty of Google Drive's backup timing, as I want to have the computer back up all the files when I leave school and have them synced with my laptop. (is there a way to force this to happen manually?)

I also know a bit about Git and think it would be helpful for me since I do a lot of code revamp to make things work better. Right now I have a ton of dead files and also have been doing really crude "versioning" by copying my working folder periodically to have a snapshot of it in some previous state. I believe Git would be a much better alternative.

One issue with GitHub seems to be the privacy aspect; I don't like the idea of all the code being public. Anyway, I'm trying to find the best Git or other software (or syncing system) that would be best for this type of workflow. Thank you.

Improve this question
2
  • I believe GitLab allows secured files even with the free version. Find it at: about.gitlab.com. It is Git too. – Eric S May 22 '18 at 20:55
  • github is not git. You can fully use git without the need of github. – Patrick Mevzek May 30 '18 at 3:28
2

First off - Start using git now - right now even if it is only on a local repository so that you can learn by doing - but don't put it on Google Drive or other file shares as it can become corrupted as a result. Many file sharing sites have in the small print a notice to not use them for hosting repositories as they will sometimes make changes that will invalidate the repository box is especially poor at this such as skipping files that they do not like or storing them differently.

The easiest way is to have a working repository on your hard drive at collage, another on your home machine and a transfer repository on a removable drive, (USB stick). Before you leave one location check in & push your changes then from your transfer drive pull the changes when you get to the other location pull from the drive. (It is a lot quicker than you might think).

Note that if you forget to sync between the locations git is very good at merging changes to plain text files but it is easier if you work on a different group of files, e.g. a different problem, until you get a chance to get synced - do not try to merge binary files.

The an easy you can get more sophisticated later workflow for using a USB drive to work between locations (assuming you already have some significant snapshots):

  1. Pick one location as your nominal master (e.g. an empty directory on your home computer)
  2. git init in your empty directory
  3. Put the earliest version of your files, in their directory structure from the snapshot directory down but without that directory - so as if it was your working directory when you had it.
  4. Create a gitignore file with the patterns for files that you do not wish to control, e.g.: *.bak - see git help ignore for more information.
  5. git status will let you see what git thinks might need to be checked in.
  6. Add the files using either one at a time with git add or all at once with git commit -a but see the next point
  7. Once files are added, or when doing the commit all, you need to do a commit with a descriptive message, e.g.: git commit -a -m"This is my old first snapshot from back in June" - if you forget the -m message an editor will open for you to type one.
  8. Copy your next snapshot over this directory replacing any existing files.
  9. git status will tell you which files actually changed.
  10. Add any new patterns that might be needed to the gitignore file.
  11. Repeat the add & commit steps above with different messages.
  12. Keep going until you are out of snapshots and then do the same with your current working copy.
  13. Once you are happy with it it might be a good idea to rename your existing working directory (for safety) and then rename your new git repository director to the previous working directory.
  14. On your USB drive use git clone with the path to where your repository is on that machine. This will create a new directory with the same name as the last part of the path.
  15. Back in your repository directory run git remote add usb d:/my_repo with the actual drive and path to the directory on the USB drive that you just created instead of d:/my_repo. This allows you to use git push usb to update the usb drive with your check-ins
  16. When you get to your other location you can insert the usb stick and then from a directory above your expected working location run git clone e:/my_repo note I am assuming that the drive letter might be different.
  17. Personally I would use the git remote add usb ... again in your new clone so that you can use the same push command in both locations.
  18. You can also add remotes in the USB copy for home & school if you would like to.

You could also consider having a private repository on one of:

You will have the get into the well worth developing habit(s) of:

  1. Branch for a new development
  2. Each time that you make a significant step commit the change(s)
  3. Push after committing any major steps or before changing location.
  4. After a change of location pull
  5. Merge when the branch is done.

Note that just about every version control system requires these steps but the names differ.

Improve this answer
3
  • Thanks Steve for that answer. I need to learn a bit more about how the repos work before I can do this. Do you mean to "clone" the repo each time I move locations to my hard drive? What would be the advantage of the USB stick over GitHub btw? I feel like I would be prone to forgetting it at either location – teepee May 22 '18 at 19:30
  • @teepee - The best and quickest ways to learn using a version control system is to learn by doing, (plus some reading), just remember you should be controlling source code (or equivalent). Expanding the USB bit above. – Steve Barnes May 23 '18 at 4:23
  • That is an excellent detailed answer you've written up. Thank you very much! I have been learning to work with the commands now and it makes sense thus far. So as far as USB vs. GitHub (free for students), would they both be OK to use for the same purpose then? Also, I'm trying to understand how things work with versioning. I am imagining that the state of ALL files is kept within the snapshot at each commit, correct? I.e. if I have f1.txt, f2.txt at commit 1, but delete f2.txt for commit 2, if I want to revert back to commit 1, would f2.txt re-appear? – teepee May 23 '18 at 21:59
1

Git seems like a good choice given that it provides tidy and sophisticated approaches to versioning. If you want to use GitHub but keep your repositories private, you can pay GitHub $7 a month and they'll let you make private repositories. Or avoid GitHub and use one of your two computers or a third machine as a Git server; all that takes is installing Git and setting up SSH access. Note that for the same or lower price as for GitHub private repositories, you can rent a virtual private server that you can use for Git as well as for anything else.

Improve this answer
9
  • 2
    Bitbucket offers free private repositories – Knerd May 22 '18 at 22:35
  • Thanks, I like this option a lot. One thing I'm trying to understand though is, let's say in one of my earliest commits, I have say 10 files each with different functions and such. A few commits down the road, I have only 2 files, but they cover the task. But, I remember there was some code written in a few of the older files that have since been deleted, which I'd like to copy in for some other purpose. Can I still find those long-since deleted files and their contents to look through it and see what was there? – teepee May 23 '18 at 22:01
  • @teepee Yes, you can check out the old commit of interest and see the full contents of every file as it was then. – Kodiologist May 23 '18 at 23:02
  • What is the procedure for doing that in a simple way? Everything I've read seems unintuitive. Like the git checkout -- (filename) is apparently the way you revert back to a file whose uodates haven't been committed, is that right? Can't one simply check out a given collection of files at a point in time shown in the log? – teepee May 23 '18 at 23:17
  • @teepee Just type the hash of the commit you want to check out instead of a filename. Git has a steep learning curve, so it's worth taking a good amount of time to read through tutorial material. – Kodiologist May 23 '18 at 23:56
1

Not exactly designed specifically for your use case, but Syncthing might be an option. It's designed for near continuous synchronization of data between nodes (with near-real-time syncing under some configurations in recent versions), and while it doesn't exactly have what you might consider traditional version control, it does have a somewhat simple but robust versioning system (it allows keeping just the last copy, keeping some number of copies which get tagged by date, or using a progressive thinning scheme or even an external script). It's also FOSS, uses minimal resources, and doesn't need any account on some central service (it may relay data through external servers if both nodes are behind NAT firewalls, but the relay nodes will only see encrypted data). If you just need versioning so that you can roll back changes you made to the files, thi sis what I would go with.

As far as more conventional VCS software, Git should be fine, with the rather important caveat that you should not be using it for binary files if at all possible (it's designed for line oriented textual data, so it's really bad at handling binary data efficiently). While your point about the public nature of Github is valid (mostly, there is a paid version that lets you have private repositories), it's important to understand that you don't actually need to use Github. Git is a specifically a distributed version control system, it works just fine in a traditional client-server arrangement like Github uses (and this is how most people use it), but it also works just as well in other configurations (say, a mesh topology for example).

In your case, as long as you can get a system that's accessible by both your work system and your home system and is on all the time (and just as importantly, uses an OS other than Windows so that you don't have to deal with trying to get Git running as a service on Windows), you can just store your master repository there, and push to and pull from it as needed, just like you would with Github (almost, since you don't get the nice web interface or the easy Windows integration).

Improve this answer
4
  • Thanks for your response; that is great! I wonder about why not having windows isn't a good method? So far I'm playing with the idea (mentioned below) of using GitHub (free for students) as my central repo and pushing to it whenever I switch computers. Both my school desktop and laptop are Windows, which is why I ask – teepee May 23 '18 at 21:56
  • Setting up git as a service on a WIndows system is a serious pain in the arse. It works fine on any UNIX-like system though. Either way, it doesn't really matter if you're going to just use GitHub, since you would only be using your laptop and desktop as client systems. – Austin Hemmelgarn May 24 '18 at 19:08
  • That's good to know. I think I'll stick with GitHub for now anyway, since it seems like an easy way to go. When working this way, i.e. With a central repo and GitHub and two computers that I'm working on back and forth, what is the cleanest way to work on a branch when I'm switching computers all the time? For example, say I make local changes at work on 'branch1', commit, push to GitHub, then go home and want to pick up where I left off, how do I "pull" that branch to continue working on It, without merging to the master? And then again the next day going back to the other computer? – teepee May 24 '18 at 21:27
  • @teepee If you don't have a local copy, you just git checkout the branch, and you'll get a local copy. If you do have a local copy, switch to the master branch with git checkout master, run git pull without any arguments (this will update all your local data to match the configured remote repo), and then switch back to the branch and run git pull again without any arguments to get the latest copy. – Austin Hemmelgarn May 25 '18 at 18:33
1

You could use the free tier of Visual Studio Team Services. It's Git based and in addition to source code management, you can also create and edit work items, including bugs, requirements, and tasks.

https://www.visualstudio.com/team-services/pricing/

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.