Environment Manager for More than Just Python

This whole virtualenv thing is pretty cool, but it has always seemed too specific to Python for me. Let me see if I can explain. At first I had no use for virtualenv whatsoever, I could just sudo apt-get install python-whatever and get what I needed. As my distro got older and I didn't want to update my whole system, I started using virtualenv and pip to install and manage python packages instead of apt. It's great, except that the postgresql packages that came with my distro were getting crusty too. Because postgresql is not a python package, I get no help from virtualenv. Does anyone else have this problem? How do you deal with it?


This reminds me of a very old problem from the chip design (EDA, ASIC design, whatever you want to call it) world. To design chips you buy licenses for expensive simulators and synthesis tools, and those tools are constantly being revved, fixing old bugs, introducing new features, and unfortunately, introducing new bugs. Because of that you never ever upgrade from one version to the next, you always install a whole new copy of each version, and then you manipulate your PATH and other environment variables that the tools use according to which version you want to experiment with today. I once worte a tool that's not too different from virtualenv if you squint a little, but it was more generic for managing your unix environment in general. It was great for dealing with all the versions of chip design tools we used. Unfortunately it's locked inside the company I wrote it for. I have since discovered that an open source project very similar to the one I wrote exists, but it uses far too much tcl ;-)


I started an open source project to try and tackle this again (before I had used virtualenv, actually), and I have wondered if it could solve this general problem for not just python. It's pretty raw, but you can see it here.


If you browsed around that project, you can see I haven't gone very far with it. It works for me, but it needs some love to make it easier for others to get up an running with it. I'm more than open to questions on how to install and use it, feedback on the design and usage model, and to contributions from others.

Comments

Anonymous said…
I may be on the wrong path but this sounds alot like homebrew for Mac. you can install many versions of the same package and because it just symlinks to a directory in your path it can rebuild the links to point at the version you need at that moment. I have run into some dependency issues on occasion, but for the most part it has been really helpful and makes running multiple versions of things like mysql and imagemagick fairly painless.
May 29, 2012 at 3:32 PM
Bryan said…
I didn't know homebrew did that, that's very interesting. Using symlinks can get you a lot of the way there, but some application (especially EDA tools) expect certain environment variables to be set certain ways depending on the version you are using. It's also nice to type 'which command-name' and see right in the path the version that you are getting. It can also be nice to be able to set arbitrary aliases, shell functions, and environment variables according to which project you are working on, and so having a little shell config file for each project is the way Environment Modules and the tools I've written do it. I believe that's essentially what virtualenv does too.
May 29, 2012 at 4:30 PM
Anonymous said…
The Nix package manager applies the same principle to the extreme: you can not only have different versions of each package, but even the same version but with different versions of the dependencies or the compiler. Nix uses a hash of the complete dependency graph of each package for unique identification.
May 29, 2012 at 11:08 PM
Alex said…
I don't use anything more than a bit of shell hackage to manipulate PATH and LD_LIBRARY_PATH. Typically when I'm looking at something from source I'll create a directory with a "install" and whatever version control in it. I build with prefix pointing to the install dir and execute switch_world to add it to my path.

At home I use Gentoo so if I want the more recent goodness in the system permanently I just bump the ebuild in my overlay. At work where I run and develop for Ubuntu we just backport the package, add it to our autobuild infrastructure and install from overlay apt repos.
May 30, 2012 at 3:40 AM
Bryan said…
nix is also very interesting. I played with it a few years back, but I can't find my notes. It seems like the downside for me was that it wouldn't handle third-party software (such as EDA tools) without somehow packaging them up as nix packages? I could be wrong. Seems like it didn't have a way to set up arbitrary aliases, shell functions, and environment variables on a per-project basis like I mentioned in my previous comment. Please correct me if I'm wrong.
May 30, 2012 at 8:49 AM
dim said…
Have a look at http://pgapt.debian.net/ (soon to be http://apt.postgresql.org) for picking the right PostgreSQL version in your debian installation.
May 31, 2012 at 5:09 AM
rerx said…
On high performance computing clusters one usually uses environment modules to make it easy to switch between various versions of libraries and software packages.
January 4, 2013 at 8:15 AM
Post a Comment

Popular posts from this blog

SystemVerilog Fork Disable "Gotchas"

This is a long post with a lot of SystemVerilog code. The purpose of this entry is to hopefully save you from beating your head against the wall trying to figure out some of the subtleties of SystemVerilog processes (basically, threads). Subtleties like these are commonly referred to in the industry as "Gotchas" which makes them sound so playful and fun, but they really aren't either. I encourage you to run these examples with your simulator (if you have access to one) so that a) you can see the results first hand and better internalize what's going on, and b) you can tell me in the comments if this code works fine for you and I'll know I should go complain to my simulator vendor. OK, I'll start with a warm-up that everyone who writes any Verilog or SystemVerilog at all should be aware of, tasks are static by default. If you do this: module top; task do_stuff ( int wait_time); #wait_time $display ( "waited %0d, then did stuff" , wa...
Read more

'git revert' Is Not Equivalent To 'svn revert'

I just learned that if you have some changes in your working tree that you want to get rid of, you don't type 'git revert' like you might guess. No, that's what cvs, subversion, mercurial, and bazaar (to name a few) use revert to mean, but not git. With git, revert is used to undo actual commits. Thankfully, you can undo your revert with another 'git revert', I just learned. So let me repeat to be clear, if you have changes to your working files that you want to abandon, DO NOT do this: git revert HEAD That will undo your last commit. Do this instead: git reset --hard HEAD I'm glad I have that straightened out now. I'm wondering if /etc was really a good place for me to start out playing with git. UPDATE: Nearly two years later and I'm still getting comments on this. I'm glad I've been able to help people out this way. The discussion in the comments is good, and one thing I'd like to point out is that I now always use...
Read more

SystemVerilog Streaming Operator: Knowing Right from Left

SystemVerilog has this cool feature that is very handy for converting one type of collection of bits into another type of collection of bits.  It's the streaming operator.  Or the streaming concatenation operator.  Or maybe it's the concatenation of streaming expressions (it's also called pack/unpack parenthetically).  Whatever you want to call it, it's nice.  If you have an array of bytes and you want to turn it into an int, or an array of ints that you want to turn into an array of bytes, or if you have a class instance that you want to turn into a stream of bits, then streaming is amazing.  What used to require a mess of nested for-loops can now be done with a concise single line of code. As nice as it is, getting the hang of the streaming operator is tough.  The SystemVerilog 1800-2012 LRM isn't totally clear (at least to me) on the details of how they work.  The statement from the LRM that really got me was this, "The stream_operator <...
Read more