Git Annex is Great


UPDATE: With current versions of git, I no longer recommend git annex or git LFS unless you really need to store your large files on a separate server from your git repository. For the simple use I describe here, just add your large files to git like any other file and when you clone, you can avoid downloading the full repository history with git clone --filter=blob:none and use git as normal.

I'm developing the website for my business and I have a mix of code an images in my git repository. Since everyone seems to know that you shouldn't keep large binary files in your git repo, I decide to see what the current solutions to that problem are.

After doing a little bit of searching, I narrowed things down to git lfs and git annex. Git lfs looks so nice and simple, except I'm not using github. Sure, you can set up your own central git lfs server yourself, but that sounded suddenly not so nice and simple.

The website for git annex immediately hits you with all the power and flexibility that it has, and I was turned off by that complexity. I did like that it doesn't require any kind of central server for me to set up, so I didn't reject it outright. After some digging I found out that it does actually support a usage that looks a lot like git lfs, where you can configure it to automatically manage certain sets of files and then you just use git commands like normal. This is very nice. Here's how you set it up in your git repo (hopefully before you have committed any binary files to git, see my next blog post about fixing that):

git annex init
git annex config --set annex.largefiles 'mimeencoding=binary and largerthan=1b'

That's it! Now just use git commands like normal and annex will take care of binary files for you. The only tricky part to setting this up was figuring out that empty files, like those __init__.py files that Django creates, were considered binary files. That's why I had to add the and largerthan clause.

There are a couple of other things you might need to be aware of:

  • When you clone, none of the binary files will get copied to your clone until you run git annex get, and that will only copy over the files for the current commit. If you checkout an older commit or another branch, you might need to run git annex get again.
  • If you do start collaborating with others you'll have to make sure that their git annex get command can access the binary files. That's where you have many many options for setting up network communication that can work for your team. I have not delved into the details of that yet.
  • If you try to delete a clone, you'll discover that the annex files down under .git are read-only. Using sudo, or changing permissions with chmod will fix that.

Aside from those things I haven't noticed any trickiness with git annex. It seems like a great tool.

Comments

Post a Comment

Popular posts from this blog

SystemVerilog Fork Disable "Gotchas"

This is a long post with a lot of SystemVerilog code. The purpose of this entry is to hopefully save you from beating your head against the wall trying to figure out some of the subtleties of SystemVerilog processes (basically, threads). Subtleties like these are commonly referred to in the industry as "Gotchas" which makes them sound so playful and fun, but they really aren't either. I encourage you to run these examples with your simulator (if you have access to one) so that a) you can see the results first hand and better internalize what's going on, and b) you can tell me in the comments if this code works fine for you and I'll know I should go complain to my simulator vendor. OK, I'll start with a warm-up that everyone who writes any Verilog or SystemVerilog at all should be aware of, tasks are static by default. If you do this: module top; task do_stuff ( int wait_time); #wait_time $display ( "waited %0d, then did stuff" , wa...
Read more

'git revert' Is Not Equivalent To 'svn revert'

I just learned that if you have some changes in your working tree that you want to get rid of, you don't type 'git revert' like you might guess. No, that's what cvs, subversion, mercurial, and bazaar (to name a few) use revert to mean, but not git. With git, revert is used to undo actual commits. Thankfully, you can undo your revert with another 'git revert', I just learned. So let me repeat to be clear, if you have changes to your working files that you want to abandon, DO NOT do this: git revert HEAD That will undo your last commit. Do this instead: git reset --hard HEAD I'm glad I have that straightened out now. I'm wondering if /etc was really a good place for me to start out playing with git. UPDATE: Nearly two years later and I'm still getting comments on this. I'm glad I've been able to help people out this way. The discussion in the comments is good, and one thing I'd like to point out is that I now always use...
Read more

SystemVerilog Streaming Operator: Knowing Right from Left

SystemVerilog has this cool feature that is very handy for converting one type of collection of bits into another type of collection of bits.  It's the streaming operator.  Or the streaming concatenation operator.  Or maybe it's the concatenation of streaming expressions (it's also called pack/unpack parenthetically).  Whatever you want to call it, it's nice.  If you have an array of bytes and you want to turn it into an int, or an array of ints that you want to turn into an array of bytes, or if you have a class instance that you want to turn into a stream of bits, then streaming is amazing.  What used to require a mess of nested for-loops can now be done with a concise single line of code. As nice as it is, getting the hang of the streaming operator is tough.  The SystemVerilog 1800-2012 LRM isn't totally clear (at least to me) on the details of how they work.  The statement from the LRM that really got me was this, "The stream_operator <...
Read more