ParallelNotes - Methods

Upload page content

You can upload content for the page named below. If you change the page name, you can also upload content for another page. If the page name is empty, we derive the page name from the file name.

File to load page content from
Page name
Comment
Type the odd letters out: scieNce GATHeRS knowledge fAster tHAN SOCIeTY GATHErS wisdom

location: ParallelNotes

Alex and Matthew playing with python parallel programming

Ipython1 - http://ipython.scipy.org/moin/IPython1 Docs therefore - http://ipython.scipy.org/doc/ipython1/html

Then we copied the sshx script from the ipython1 distro into /imaging/local/linux/bin

Shell setup

The engines use the ~/.bashrc to initialize environment variables for the shell (this does not happen by default because sh is used, which does not read startup scripts). The problem being that Alex is set up with the usual default tcsh, which uses ~/.cshrc.

Accordingly, we added these lines to ~/bashrc: 

# FSL stuff - if we are using FSL
export FSLDIR=/imaging/local/linux/bin/fsl-4.0.1
. ${FSLDIR}/etc/fslconf/fsl.sh
export PATH=${FSLDIR}/bin:$PATH

cbu_img_sys=/imaging/local/linux
# If on linux, set stuff for libraries, and python
if [ "`/bin/uname`" = "Linux" ]; then
    export PATH=${cbu_img_sys}/bin:$PATH
    export LD_LIBRARY_PATH=${cbu_img_sys}/lib
fi

Cluster setup

We made a tiny script to test with: 

   1 controller = dict(host='l42',
   2                   engine_port=None, # default is 10105
   3                   control_port=None,
   4                   )
   5 
   6 # keys are hostnames, values are the number of engines on that host
   7 engines = dict(l31=2,
   8                l32=2,
   9                l33=2,
  10                l34=2,
  11                )

Then we ran ipcluster -f small_cluster.py

and got this: 

Starting controller:
Starting engines:   
Sarting 2 engines on l34
Sarting 2 engines on l32
Sarting 2 engines on l33
Sarting 2 engines on l31

Your cluster is up and running.

For interactive use, you can make a MultiEngineClient with:

from ipython1.kernel import client
mec = client.MultiEngineClient(('l42',10105))

You can then cleanly stop the cluster from IPython using:

mec.kill(controller=True)