Neuroconda at MRC CBU
Neuroconda is a Conda environment that bundles Python, R and some system binaries and libraries for neuroimaging analysis. For general Neuroconda documentation, please see the project Github page. Here we describe the details of the existing Neuroconda deployment at CBU.
What are environments and why is this useful?
It's worth familiarising yourself with the Conda documentation to get some idea of how environments work. At a high level, when you activate Neuroconda, your system path is modified to add various libraries and packages to the path. So for any analysis code to work, the environment must first be activated. The neat thing about this extra step is that you now have control over exactly which package versions your code is running, instead of implicitly depending on whatever happens to be on your path at the moment.
Each time a new Neuroconda release is made, we create a new environment on the imaging system. You can se all the available Neuroconda versions by doing this in a login node shell session:
conda env list | grep neuroconda
As long as you always activate the same Neuroconda release, code you write should run reproducibly for yourself and even people at other institutions.
What is included in Neuroconda?
Python with a fairly exhaustive set of scientific computing and neuroimaging packages, including the Jupyter Lab developer environment (recommended starting point for beginners)
R with all the usual expected packages, including the R-Studio developer environment
various unix command-line tools including git (with git-annex), pandoc, dcm2niix, and perl
- FSL including the fsleyes viewer
- SPM12 (added to your matlab path)
- Freesurfer
- AFNI
- ANTS
As you can see, Neuroconda is much more than a Python environment. It provides basically all the scientific computing tools you need. Instead of having to keep track of how to add each of the above packages to your path, you can just activate Neuroconda and get on with it.
Using Neuroconda at CBU
The easiest way to get started is to do something like this on a login node
source /imaging/local/software/neuroconda/v2_0/neuroconda.csh
This works assuming you've kept your shell to the CBU default TCSH shell. If you've wisely switched to bash, zsh or another modern shell, the line you need is
source /imaging/local/software/neuroconda/v2_0/neuroconda.sh
This will add all the neuroconda packages to your path, but also some non-conda neuroimaging packages (listed in plain font above). You can consult the script to see exactly what packages get added. To test that everything worked, try doing
which python; which fsl
You should get something like
/imaging/local/software/miniconda/envs/neuroconda_2_0/bin/python /imaging/local/software/centos7/fsl/bin/fsl
Just keep in mind that everything is set up specifically in this shell session. If you start another shell or log out and in again, you'll have to run the source command again.
For code without non-Conda dependencies
If you aren't planning to use those non-Conda neuro packages (the ones not in bold font in the list above), you can just use Neuroconda like any Conda environment by doing
conda activate neuroconda_2_0
For batch / job submission
If you want to use Neuroconda non-interactively, for instance when running a batch script or submitting a job to our SLURM cluster, things get a little bit more complicated. The environment you just activated is specific to the shell you're currently running, as we just saw. And conda environments can only be activated in interactive login shells out of the box, so simply putting the above lines in your shell scripts may not work.
The best way around this is to add the following line to the top of your bash shell scripts:
source $(conda info --base)/etc/profile.d/conda.sh
After which the commands above should work normally inside the script.
A more hacky route is to put the source or conda activate line above in your shell login script, so that every shell you start is in neuroconda. You can then put the -l flag in your shell script she-bang to force scripts to run as login shell. But that may have unintended effects (for example, it may break vncserver). I don't recommend this approach.
Adding and updating packages
The central neuroconda environments we provide are read-only. This is to preserve reproducibility. We're all using the same environments, so if you were able to change things it would have unpredictable effects for everyone else on the imaging system.
If you need something, please get in touch with Johan who will be happy to add it to the next Neuroconda release. But if you need it sooner than that, you can try to clone the environment. Once you have your personal version of the Neuroconda environment (which lives in your home directory rather than on the shared imaging drive) you can modify it as you see fit.
If cloning doesn't work (usually because conda-forge is aggressive about marking builds as 'broken' which causes them to get dropped from the main channel), you can rebuild the environment from the YAML instead, like so:
conda env create -f https://github.com/jooh/neuroconda/raw/v2.0/neuroconda.yml --name my_neuroconda_2_0
This works because unlike cloning, neuroconda.yml specifies package versions only, not specific builds. So if the main neuroconda_2_0 has a given package version with a build that is now 'broken' on conda-forge, you will create an environment where the package is the same version but with an updated, working build. Crystal clear right?
But every so often that fails too, because the package dependencies have changed and now the environment in neuroconda.yml contains conflicts (ie, packages that depend on different versions of the same supporting package). When this happens, it's time to ping Johan because Neuroconda will have to be updated.