PYTHON-CBRG

Section: CCB Python (7)
Last updated: Wed 10 Jan 16:53:42 GMT 2024
Index Return to Main Contents

DEPRECATION NOTICE

This CCB documentation has been superseded by our new website: https://lumin.imm.ox.ac.uk

OVERVIEW

As well as the standard Python versions that you'd normally expect, we've additionally preinstalled hundreds of additional packages for you to use python-installs(7). These are available via the python-cbrg module.

BASIC USAGE

If you just want to get up and running with our curated set of commonly used bioinformatics packages, you can do so with a single command:

$ module load python-cbrg

If you want to see the full list of installed packages, load the module as above and then run the command:

$ pip list | less

Please note, however, that this command will take some time to run and the output will be rather long.

REQUESTING ADDITIONAL PACKAGES

If you need to use a package which isn't already installed, please contact us via the address below before attempting to install a local copy. In many cases we can easily add it to the central installation.

INSTALLING LOCALLY

There are a range of package management tools for Python - PIP, venv, Poetry, Conda. The simplest is PIP, and has the advantage of being compatible with our interactive Jupyter Notebook. Install a package locally with:

$ module load python-cbrg  # (or python-base)
$ PYTHONUSERBASE=/project/example/user/my-python python -m pip install --user PKG

If PIP is missing then run:

$ PYTHONUSERBASE=/project/example/user/my-python python -m ensurepip --user

Then to import in Jupyter, add to top of notebook:

import sys; sys.path.insert(0, '/project/example/user/my-python/lib/python3.11/site-packages')

CONDA

While we do not offically support Conda, we can provide a basic set of good practices for using it on our system. Essentially this is being aware that when installing Conda on our network-based filesystem, then Conda can be very slow depending on how busy the cluster is.

During Conda install you are asked whether to "update your shell profile to automatically initialize conda". You should decline this, preventing your SSH logins being delayed by Conda. You can manually activate Conda with:

$ . path/to/your/Conda/install/bin/activate

Also, do not run

$ conda init

as this will add automatic initialization to your .bashrc. If you have already added automatic initialization to your .bashrc file, then disable simply by deleting or relocating the clearly-marked Conda block from your .bashrc.

Cleaning old packages or regenerating the index cache can speedup activation:

$ conda clean --packages
$ conda clean --index-cache

Disabling automatic update can speedup environment creation and new package installs. To disable, put the following in file ~/.condarc:

auto_update_conda: false

You can manually update with:

$ conda update --all && conda update conda

ADVANCED USAGE

The setup of the python-cbrg module uses the following system.

python-base contains fixed, unchanging installations of the base languages. This is for safety; they cannot be accidentally overwritten causing unexpected changes of behaviour. python-cbrg contains separate package and library repositories for each version of Python. Because packages and library versions also change over time, we take a snapshot of the state every 3 months and then lock this to prevent changes causing unexpected behaviour. A single current version for each provides a continual rolling 'head' where changes are applied. Loading the python-cbrg module will automatically pull in the latest stable base and all packages or libraries:

$ module load python-cbrg
Loading python-cbrg/current
  Loading requirement: python-base/3.8.3
$ module list
Currently Loaded Modulefiles:
  1) python-base/3.8.3(default) 2) python-cbrg/current(default)

However, if you want to use a different version of the base, you can do that by loading it manually first:

$ module load python-base/3.6.10
$ module load python-cbrg
$ module list
Currently Loaded Modulefiles:
   1) python-base/3.6.10 2) python-cbrg/current(default)

Simmilarly, if you want to use a different version of the libraries, for example because a recent update broke something you relied on, you can do that by loading it manually:

$ module load python-cbrg/202104
Loading python-cbrg/202104
  Loading requirement: python-base/3.8.3
$ module list
Currently Loaded Modulefiles:
   1) python-base/3.8.3(default) 2) python-cbrg/202104

GETTING HELP

You can email the CCB team using the email address help@imm.ox.ac.uk. Using this address ensures your email is logged and assigned a tracking number, and will go to all the core team, which means the appropriate person or people will be able to pick it up.

COPYRIGHT

This text is copyright University of Oxford and MRC and may not be reproduced or redistributed without permission.

AUTHOR

Duncan Tooke <duncan.tooke@imm.ox.ac.uk>