Tuesday, May 29, 2012

Environment Manager for More than Just Python

This whole virtualenv thing is pretty cool, but it has always seemed too specific to Python for me. Let me see if I can explain. At first I had no use for virtualenv whatsoever, I could just sudo apt-get install python-whatever and get what I needed. As my distro got older and I didn't want to update my whole system, I started using virtualenv and pip to install and manage python packages instead of apt. It's great, except that the postgresql packages that came with my distro were getting crusty too. Because postgresql is not a python package, I get no help from virtualenv. Does anyone else have this problem? How do you deal with it?


This reminds me of a very old problem from the chip design (EDA, ASIC design, whatever you want to call it) world. To design chips you buy licenses for expensive simulators and synthesis tools, and those tools are constantly being revved, fixing old bugs, introducing new features, and unfortunately, introducing new bugs. Because of that you never ever upgrade from one version to the next, you always install a whole new copy of each version, and then you manipulate your PATH and other environment variables that the tools use according to which version you want to experiment with today. I once worte a tool that's not too different from virtualenv if you squint a little, but it was more generic for managing your unix environment in general. It was great for dealing with all the versions of chip design tools we used. Unfortunately it's locked inside the company I wrote it for. I have since discovered that an open source project very similar to the one I wrote exists, but it uses far too much tcl ;-)


I started an open source project to try and tackle this again (before I had used virtualenv, actually), and I have wondered if it could solve this general problem for not just python. It's pretty raw, but you can see it here.


If you browsed around that project, you can see I haven't gone very far with it. It works for me, but it needs some love to make it easier for others to get up an running with it. I'm more than open to questions on how to install and use it, feedback on the design and usage model, and to contributions from others.

7 comments:

Anonymous said...

I may be on the wrong path but this sounds alot like homebrew for Mac. you can install many versions of the same package and because it just symlinks to a directory in your path it can rebuild the links to point at the version you need at that moment. I have run into some dependency issues on occasion, but for the most part it has been really helpful and makes running multiple versions of things like mysql and imagemagick fairly painless.

Bryan said...

I didn't know homebrew did that, that's very interesting. Using symlinks can get you a lot of the way there, but some application (especially EDA tools) expect certain environment variables to be set certain ways depending on the version you are using. It's also nice to type 'which command-name' and see right in the path the version that you are getting. It can also be nice to be able to set arbitrary aliases, shell functions, and environment variables according to which project you are working on, and so having a little shell config file for each project is the way Environment Modules and the tools I've written do it. I believe that's essentially what virtualenv does too.

khinsen said...

The Nix package manager applies the same principle to the extreme: you can not only have different versions of each package, but even the same version but with different versions of the dependencies or the compiler. Nix uses a hash of the complete dependency graph of each package for unique identification.

Alex said...

I don't use anything more than a bit of shell hackage to manipulate PATH and LD_LIBRARY_PATH. Typically when I'm looking at something from source I'll create a directory with a "install" and whatever version control in it. I build with prefix pointing to the install dir and execute switch_world to add it to my path.

At home I use Gentoo so if I want the more recent goodness in the system permanently I just bump the ebuild in my overlay. At work where I run and develop for Ubuntu we just backport the package, add it to our autobuild infrastructure and install from overlay apt repos.

Bryan said...

nix is also very interesting. I played with it a few years back, but I can't find my notes. It seems like the downside for me was that it wouldn't handle third-party software (such as EDA tools) without somehow packaging them up as nix packages? I could be wrong. Seems like it didn't have a way to set up arbitrary aliases, shell functions, and environment variables on a per-project basis like I mentioned in my previous comment. Please correct me if I'm wrong.

dim said...

Have a look at http://pgapt.debian.net/ (soon to be http://apt.postgresql.org) for picking the right PostgreSQL version in your debian installation.

rerx said...

On high performance computing clusters one usually uses environment modules to make it easy to switch between various versions of libraries and software packages.