Introduction

The SSH (Secure Shell) utility is used to establish remote connections using the command-line. Simply connect to one of the interactive Linux machines using the name "lx-pool.gsi.de". Replace "jdoe" with your user account name. First time around the system will ask you to accept the remote computer into the list of known hosts. You should compare the fingerprint presented to you with the fingerprint shown on our InteractiveMachines page, before you accept it as a trustworthy computer from GSI by typing "yes" followed by pressing enter (return, here represented by the "↵" symbol):

» ssh jdoe@lx-pool.gsi.de↵
The authenticity of host 'lx-pool.gsi.de (140.181.70.153)' can't be established.
RSA key fingerprint is 96:15:0e:e7:70:09:60:9a:c4:f6:89:05:be:ed:be:c6.
Are you sure you want to continue connecting (yes/no)? yes↵
Warning: Permanently added 'lx-pool.gsi.de' (RSA) to the list of known hosts.
jdoe@lx-pool.gsi.de's password:↵
jdoe@lxi050: exit↵

You will be prompted to type in your password before you can login. Next time you login, ssh will check if you connect is authentic (and alerts you if the remote host cannot be trusted anymore). Type exit to logout from the remote computer and end the ssh session. Enable support for tunneling of graphical user interface applications via ssh with the -X option:

» ssh -X jdoe@lx-pool.gsi.de↵

Very often this is called enabling X11 forwarding. Continue reading to learn about more sophisticated use-cases of ssh.

Security Advice

A fully functional ssh-account is a very valuable resource for malicious activities. It is therefore very important to pay attention to security. Unfortunately, most of the security-settings cannot be technically enforced, or come at a great cost to usability. General advice is:
  1. Use a strong password for your account
  2. Use a strong password for your private key
  3. Never use unencrypted private keys
  4. Do not use rsa-Keys. Use ed25519 instead.
  5. Use -o with ssh-keygen. On older versions of ssh-keygen, this provides much better resistance to brute force.
  6. Do not use agent-forwarding, use -J on the command line or ProxyJump in your config file instead
  7. Only use X11-forwarding when necessary

Default Account Name for GSI

Many people don't use the same account name on their local computer as they have received from GSI. In this case we recommend to create a general definition in the ssh client configuration file $HOME/.ssh/config for all hosts in GSI:

### Default user for GSI
Host *.gsi.de
  User jdoe

It is possible to enable X11 forwarding by default with "ForwardX11 yes".

Create a Key Pair for Authentication

It is required by the security-policy of GSI/FAIR, that you protect your private key with a non-trivial password!

Generate an ssh public/private key pair on your local computer (unless you have one) with the ssh-keygen utility. The public portion of this key pair will reside on the servers you want to connect to, while the private key needs to remain in $HOME/.ssh/id_ed25519 on your local machine. It is very important that the private key is never disclosed to anyone. Think of automatic backups, lost laptops, old hard drives and so on. The only way to keep this under control is to choose a strong password for your private key.

ssh-keygen -o -q -f ~/.ssh/id_ed25519 -t rsa

The public key, which is the counterpart of your private key, will be located in $HOME/.ssh/id_25519.pub.

The password of your private key can be changed with:

ssh-keygen -o -f ~/.ssh/id_ed25519 -p
It is required by the security-policy of GSI/FAIR, that you protect your private key with a non-trivial password!

Use a Public Key for Login

Install your public key manually to a remote machine using the ssh-copy-id program:

ssh-copy-id jdoe@lx-pool.gsi.de:

If you work on a computer missing this program you can alternatively add your public key to the file $HOME/.ssh/authorized_keyson the remote computer. Upload the file to the remote computer:
scp ~/.ssh/id_ed25519.pub jdoe@lx-pool.gsi.de

Login to the remote computer and append the public key to the authorized keys file:
$ cat ~/id_ed25519.pub >> ~/.ssh/authorized_keys
$ chmod 700 ~/.ssh
$ chmod 600 ~/.ssh/authorized_keys

In case the authorized key file has not been present before, you may need to make sure that the file system permission are set correctly. Your key should now be used automatically when logging in:
$ ssh jdoe@lx-pool.gsi.de↵
Enter passphrase for key '.ssh/id_ed25519': 
jdoe@lxi050: exit↵
Notice how the prompt no asks for the passphrase of the private key, and not the password of the user-account jdoe. This passphrase must be entered everytime you want to use this key. Since typing strong passphrases quickly becomes very boring, it is recommended to use an ssh-agent.

Use ssh-agent

The ssh-agent is a small daemon, that caches your unencrypted private key in memory. Usually an ssh-agent is started at login time (for GUI sessions eg. KDE logins). You can mange the your ssh-agent with the command ssh-add, e.g. to check if an ssh-agent is running:
» ssh-add -l
The agent has no identities.       <-- ssh-agent is running, but no keys are loaded

Private keys are added with ssh-add:
» ssh-add ~/.ssh/id_ed25519
Enter passphrase for /home/jdoe/.ssh/id_ed25519:
Identity added: /home/jdoe/.ssh/id_ed25519 (/home/jdoe/.ssh/id_ed25519)
» ssh-add -l 
2048 2b:c5:77:23:c1:34:ab:23:79:e6:34:71:7a:65:70:ce .ssh/id_ed25519 (RSA)
Now you should not be prompted for a password anymore when using ssh:
$ ssh jdoe@lx-pool.gsi.de↵
jdoe@lxi050: exit↵

Proxy Jumping

It is often necessary to login to one machine, and connect from there to another machine. Think working on kronos from home, which is not reachable from the internet. Since lxpool.gsi.de is reachable from the internet, you can use this as a jump host:
ssh -J jdoe@lx-pool.gsi.de jdoe@kronos.hpc.gsi.de
This is a long command, and you need to type your password twice. You can fix this by using an ssh-agent and a config-file. To proxy an ssh-connection to kronos.hpc.gsi.de over lxpool.gsi.de can put the following in ~/.ssh/config
Host lxpool                                         # <-- lxpool is an arbitrary name, choose what suits you
   User $USER_AT_GSI                                # <-- must be your account at gsi
   ForwardAgent no
   HostName lxpool.gsi.de                           # <-- must be the real name of lxpool
Host kronos                                         # <-- again, arbitrary
   User $USER_AT_GSI
   Hostname kronos.hpc.gsi.de
   ProxyJump lxpool                                 # <-- lxpool here must be the arbirtary name chosen above
                                                    #     for the connection to lxpool.

If setup like that, you should be able to connect to kronos by just typing ssh kronos, using the arbitrary for the kronos-connection.

Agent Forwarding

ALERT! Agent forwarding is considered bad practice and should not be used. Avoid it by using jump proxies, which do not suffer from the insecurity of agent forwarding.

SSH agent forwarding forwards the connection to remote machines you logon to. Your private keys are not transferred to the remote machine, but it is possible to use your keys on the remote machine through the forwarded connection. If an attacker has the remote machine under her control, she can use your credentials to login to other machines.

To enable agent forwarding, add the -A option when you run ssh.
»  ssh -A -J jdeo@lx-pool.gsi.de kronos.hpc.gsi.de
Try to avoid forwarding your ssh-agent whenever possible. If you absolutely need to, at least add your keys using -c as option to ssh-add. This way, every time you (or someone else) tries to use you the key loaded with ssh-add -c you will be asked to confirm the usage.

Managing Fingerprints

When connecting to remote computer using SSH for the first time you'll be prompted to accept its host fingerprint. Once accepted, this fingerprint will be stored in $HOME/.ssh/known_hosts. A List of all known host fingerprints can be displayed with.

ssh-keygen -lv -f $HOME/.ssh/known_hosts

Removing all fingerprints belonging to a host:
ssh-keygen -R hostname.gsi.de

Show the fingerprint of a host (and optionally append it to the list of known hosts) with ssh-keyscan:

» ssh-keyscan -t rsa,dsa hostname.gsi.de
[…]
» ssh-keyscan hostname.gsi.de >> $HOME/.ssh/known_hosts

Search for a host in your known host file file:

ssh-keygen -H -F hostname.gsi.de

Avoid Host IP Checking on Login Pools

Connecting to the InteractiveMachines at GSI using one of the alias names like lx-pool.gsi.de will send you to a different host at each reconnect. All hosts part of the same pool of login machines have the same host key for authentication with your client connection.

You may see a warning that the host IP address has changed, but login will be possible nevertheless. It is possible to disable this warning message by adding to your personal SSH configuration (in the file $HOME/.ssh/config) the following to lines:

Host lx-pool.gsi.de
    CheckHostIP no

Replace the (DNS) name lx-pool.gsi.de with any other pool of login machine like ica.hpc.gsi.de for example.

Moving Data with Secure Copy

SCP stands for secure copy, and is used to copy files across ssh connections. Following example copies an entire directory /local/path/ to the users home-directory on one of GSIs InteractiveMachines:

scp -r /local/path/ jdoe@lx-pool.gsi.de:remote/path/

The option -r ensures that all file in the directory tree are copied recursively. The next example copies file from the InteractiveMachine to the local computer:

scp -r jdoe@lx-pool.gsi.de:remote/file/*.pdf /local/path

In both cases your define first what to copy, and second where to copy. Source and/or destination can be located on a remote computer. In case you plan to transfer a signification amount of data you can increase the speed by using an alternative encryption method. Define the a faster cipher algorithm like Blowfish (default is AES 128) with the option =-c=:

scp -c blowfish -r jdoe@lx-pool.gsi.de:/path/to/data/*.root /local/path/to/data

Private Network Tunnel over SSH

Connections to single network ports can be forwarded vie SSH tunnel with command line option -L

ssh -L :: 

Example:

Forward requests to internal GSI webserver:

ssh -L 54321:www-internal.gsi.de:80 lx-pool.gsi.de

Now you will be able to connect to www-internal.gsi.de via http://localhost:54321.

sshuttle

Shuttle implements VPN access via SSH. It helps to comfortably access GSI network resources from home transparently. Install the shuttle package on Debian (>=7) and Ubuntu (>= 11.10):

» sudo apt-get install sshuttle

Connect to the GSI network (first the Sudo password is prompted, next the SSH login password):

» sshuttle --dns --remote $USER@lxpool.gsi.de --daemon --pidfile=/tmp/sshuttle.pid 0/0

This will route all your traffic (including DNS requests) transparently through SSH. Disconnect with:

» kill $(cat /tmp/sshuttle.pid)

The script ssh-tunnel simplifies this process:

» ssh-tunnel help
[…]
» ssh-tunnel connect jdoe@lxpool.gsi.de
[local sudo] Password: 
jdoe@gsi.de's password: 
Connected.
[…]
» ssh-tunnel status
Sshuttle running with PID 15083.
» ssh-tunnel disconnect
» ssh-tunnel status
Sshuttle not connected.

Limit tunneling to a range of IP networks:

» ssh-tunnel connect lxpool.gsi.de 140.181.64.0/18 10.12.0.0/16 

Mount Directories over SSH

Use SSHFS to mount a directory on a remote system to a local folder, so you will be able to do any operation on the mounted files with local tools (e.g. to edit a text file).

Install sshfs from the following sources:

  • sshfs package on Debian, Ubuntu, RedHat, CentOS, Fedora, ArchLinux
  • SSHFS bundle on Mac OS X
  • win-sshfs on Windows XP/Vista/7

The command sshfs mounts a remote path with:

» sshfs [user@]host[:port] /mnt/path -C -o reconnect,auto_cache,follow_symlinks
» fusermount -u /mnt/path

The command fusermount is used to unmount the remote system.

The script ssh-fs simplifies this process:

» ssh-fs mount kronos.hpc.gsi.de:/lustre/nyx ~/nyx
» ssh-fs list 
kronos.hpc.gsi.de:/lustre/nyx on /my_home_dir/nyx
» ssh-fs umount ~/nyx

Mount Nyx directories when outside the GSI network and sshuttle is not available

If you are located outside the perimeter of the GSI network, a direct connection to the frontend nodes of our cluster (kronos.hpc.gsi.de) is not possible. If a proxy like sshuttle is not available on your local machine, there are two possible ways to mount a remote filesystem locally:

1) ssh -f user@lx-pool.gsi.de -L 3000:kronos.hpc.gsi.de:22 -N
2) sshfs -p 3000 user@127.0.0.1:/lustre/nyx /my_local_dir

Get a Mathematica License Over Ssh

Ssh has the capability to tunnel arbitrary network traffic through the SSH connection. This way internal services that are normally blocked by the firewall from remote can be made accessible.

Example: To obtain a GSI Mathematica license from remote (this implies you already have Mathematica installed) use the following command:

ssh -L 16286:licenseserver.gsi.de:16286 lx-pool.gsi.de

This will connect to one of the computers of the central interactive Linux farm while forwarding the Mathematica license server to your local computer. You have to put !localhost in your local mathpass file to tell Mathematica where to find its licenseserver. Then startup Mathematica and it will obtain a GSI floating license.

Trouble Shooting

Getting help from the IT

If something does not work, you can always try to open a ticket. If possible include the debugging output of your ssh-command. You can get the debugging output by specifiying -vvv:
$ ssh -vvv jdoe@lxpool.gsi.de

Password Forgotten

Please contact the UserHelpDesk (SB3 1.247, ☎ 2515).

ssh-agent

The following error message when trying to add keys to your ssh-agent means, that your ssh-agent is not running, or the environment variables are not pointing to the correct location.
Error connecting to agent: No such file or directory
A quick-and-dirty-fix is to start ssh-agent manually: eval $(ssh-agent) This will likely result in many ssh-agents running on your computer. A better solution is to use systemd user-sessions to start a session (google is your friend). If you don't like this, have a look ssh-agent-session: The script ssh-agent-session helps to use a single ssh-agent session in multiple shells. It will start a new agent and store the connection information to ~/.ssh/agent-session.

» source ssh-agent-session
ssh-agent started, session in /home/jdoe/.ssh/agent-session
» ssh-add ~/.ssh/id_ed25519
[…]

Another shell can bind to the same agent:

» source ssh-agent-session
ssh-agent running with process ID 19264
» ssh-add -l 
[…]

It is convenient to source this script within the shell profile, in order to bind all shells to a single ssh-agent instance.

» echo "source /path/to/ssh-agent-session"  >> ~/.bashrc

Session Hangs

In case the computer loses connectivity with the remote host your ssh session could hang, and is not reacting anymore on keyboard input. For these cases use the escape sequence ~. to end the connection even when you have no command prompt. Type return twice (for good measure) and then the tilde followed by a period. This will terminate the ssh connection from the client end instead of the server end.

Note that besides the terminate connection escape sequence you can use ~? to show a list of all available escapes:

Supported escape sequences:
  ~.  - terminate connection (and any multiplexed sessions)
  ~B  - send a BREAK to the remote system
  ~C  - open a command line
  ~R  - Request rekey (SSH protocol 2 only)
  ~^Z - suspend ssh
  ~#  - list forwarded connections
  ~&  - background ssh (when waiting for connections to terminate)
  ~?  - this message
  ~~  - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

Permission Denied

The permission denied error indicates some failure with your credentials. Some common mistakes are:

  1. Your ssh-agent has no private keys loaded.
  2. Your ssh-agent has too many private keys loaded. The default ssh-server-config allows 6 authentication tries, so if you have 7 or more private keys loaded, this can lead to problems.
  3. None of your private keys matches an entry in ~/.ssh/authorized_keys on the target node.
  4. The login account doesn't exist on the remote node.
  5. Your password was wrong
Topic attachments
I Attachment Action Size Date Who Comment
ssh-agent-sessionEXT ssh-agent-session manage 2 K 2014-04-10 - 14:48 VictorPenso Script to share a single ssh-agent across multiple shells.
ssh-fsEXT ssh-fs manage 1 K 2014-12-10 - 13:09 VictorPenso Mount a remote directory over SSH
ssh-tunnelEXT ssh-tunnel manage 2 K 2014-10-16 - 11:09 VictorPenso Manage SSH network tunnels.
Topic revision: r44 - 2020-06-09, MatthiasPausch - This page was cached on 2020-12-02 - 23:04.

This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding GSI Wiki? Send feedback | Imprint (german) | Privacy Policy (german)