SSH-TUNNEL

Section: SSH Tunnelling and Port Forwarding (7)
Last updated: Wed 10 Jan 16:53:42 GMT 2024
Index Return to Main Contents

DEPRECATION NOTICE

This CCB documentation has been superseded by our new website: https://lumin.imm.ox.ac.uk

OVERVIEW

Sometimes, you may wish to use software on JADE which is accessed via a locally hosted web link, the most common example being Jupyter Notebooks. Whilst we provide an out-of-the-box method of running Jupyter Notebooks using our software collection - see welcome(7) - there may be times you wish to use a different version of Jypyter or other software we don't have an easy launcher for.

The solution to this problem is to create an SSH tunnel and forward the necessary port to your local computer. This guide explains how to to this.

BEFORE YOU START

Creating an SSH tunnel is a moderately advanced topic and is recommened only if you're comfortable with Linux and SSH. If you're a relative beginner, we strongly recommend sticking with our pre-provided services until you gain confidence. We are unable to offer signficant help for SSH port forwarding and you are expected to perform your own troubleshooting before asing for help.

Additionally, if you need to use SSH tunnelling you should consider setting up key-based SSH access and an SSH agent with agent forwarding. Attempting to use tunnelling without these is perfectly possible, but you will be repeatedly asked for your CCB password. A quick internet search should provide any number of step-by-step guides on how to do this.

HOW TUNNELLING WORKS

SSH tunnels create a secure connection between two computers. By adding port forwarding to a tunnel, you can force traffic that would normally go over the internet to go through the tunnel.

The key trick is that you can chain multiple SSH tunnels together in several hops, and send traffic through them to places that aren't accessible on the internet in the first place. This allows you to connect your local web browser directly to a web service running on one of our compute nodes.

The key flag for setting up an SSH tunnel is -L. There are many variations on this flag, however the one we use is in the form:

-L LOCAL_PORT:localhost:REMOTE_PORT

This tells SSH that we want it to connect together the traffic trying to pass between our local and remote ports.

CHOOSING A PORT NUMBER

A network port can only be used by one program on each of the computers in question, and there's relatively few rules on which ports to use. The program you're running may or may not give you the option of which port it will run on, but for our SSH tunnels we get to choose.

Ideally, we want to keep things as simple as possible, so where possible we recommend that you only attempt to use a single tunnel at once, and that when you do you forward the port equal to your CCB login ID. You can easily find this with the following command:

$ id -u

For this example, we'll use the port 20001 belonging to user dtooke. Please note that all login IDs on the CCB are in the range 20000 to 40000 and that forwarding the port associated with another user is considered to be against our acceptable-use(7). If you need to forward additional ports, please use a number in the range 40001 to 60000.

EXAMPLE

This is an example of running the command-line program cellxgene over an SSH tunnel with port forwarding. To do this we require two separate SSH sessions.

SSH SESSION 1

Here, I log into JADE, start an interactive session as described in slurm-basics(7) and run my program:

$ ssh dtooke@login1.molbiol.ox.ac.uk
dtooke@imm-login1:~$ srun --partition=short --cpus-per-task=4 --mem=32G --pty bash -i
srun: job 17804 queued and waiting for resources
srun: job 17804 has been allocated resources
dtooke@imm-wn2:~$ module load cellxgene
Loading cellxgene/20230703
  Loading requirement: python-base/3.11.3
dtooke@imm-wn2:~$ cellxgene launch https://github.com/chanzuckerberg/cellxgene/raw/main/example-dataset/pbmc3k.h5ad
[cellxgene] Starting the CLI...
[cellxgene] Loading data from pbmc3k.h5ad.
[cellxgene] Warning: Moving element from .uns['neighbors']['distances'] to .obsp['distances'].

This is where adjacency matrices should go now. 
[cellxgene] Warning: Moving element from .uns['neighbors']['connectivities'] to .obsp['connectivities'].

This is where adjacency matrices should go now. 
[cellxgene] Launching! Please go to http://localhost:5005 in your browser.
[cellxgene] Type CTRL-C at any time to exit.

This gives me two key pieces of information - the program is running on imm-wn2 and using port 5005.

SSH SESSION 2

Now I start my SSH tunnel:

$ ssh -A -L 5005:localhost:20001 dtooke@login1.molbiol.ox.ac.uk
dtooke@imm-login1:~$ ssh -N -L 20001:localhost:5005 dtooke@imm-wn2
[This command produces no output]

My first SSH forwards my agent so that the second one doesn't need a password and sets up a port forward between 5005 on my local computer and 20001 on the login node. The second is a non-login session (logins are not permitted on worker nodes) that sets up a port forward between 20001 on the login node and port 5005 on the worker node.

The end result of this is that port 5005 on the login node is forwarded to port 5005 on my local computer and I can directly open the link in my browser. All of the forwarding is done over port 20001.

CLOSING THE SESSION

When I'm finished I need to do the following in order:

- Close my browser window

- Control-C in session 2 to end the second tunnel

- exit in session 2 to end the first tunnel

- Control-C in session 1 to end the program

- exit in session 1 to exit my interactive session

TROUBLESHOOTING

Using an SSH tunnel with port forwarding is not trivial. If you run into problems, here are some things to check:

- Make absolutely certain that you have the correct worker node and port for the program that you're running; they may both change and you can't rely on past commands to work the same way twice

- The second session with the SSH tunnel must be from your local computer to the login node, and from there to the worker node. Accidentally starting the second session directly from the login node will not work.

- Double check that you have the port numbers in the two tunnel hops the correct way around, and note that they are opposite and that the order matters

- If you get a message in the form:

bind [127.0.0.1]:20001: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 20001
Could not request local forwarding.

it means that there is already another SSH session using that port. The most likely explanation is that you did not cleanly shut down your session and that one of your own SSH tunnels is still running somewhere. The easist solution is to pick a different port in the range 40000 to 60000 and try again if you are unable to find the process in question.

GETTING HELP

You can email the CCB team using the email address help@imm.ox.ac.uk. Using this address ensures your email is logged and assigned a tracking number, and will go to all the core team, which means the appropriate person or people will be able to pick it up. However, please note that, as described above, SSH tunnels and port forwarding are an advanced topic and that you are expected to have performed your own troubleshooting before asking for help.

COPYRIGHT

This text is copyright University of Oxford and MRC and may not be reproduced or redistributed without permission.

AUTHOR

Duncan Tooke <duncan.tooke@imm.ox.ac.uk>