Always connected reverse SSH port forwarding with systemd

I was recently working on a RaspberryPi powered wall information display for our office in California. After the folks in office got the RPi on their local WiFi network and manually setup a SSH tunnel that forwards local SSH port to a host that I have SSH access, I took over. This device was in a network that wasn’t managed by us so port forwarding from a public IP to the device was not an option. The first thing I needed to ensure was reachability. If this device were to be restarted or the existing connection closed for some reason it needed to automatically reestablish the connection. Autossh was the de-facto standard in the past but now that systems is default in most modern distributions, its a good candidate to manage and monitor our reverse SSH tunnel

Terms used

  • remote host refers to a device running in a third-party managed network i.e you have no control over any networking equipment. Its public IP may or mayn’t change
  • managed host refers to a server/device whose SSH port is reachable

Assumptions

  • remote host runs a distribution that uses systemd as an init system
  • A user named callhome on the remote host is able to SSH using public key authentication as incoming@managed host

systemd service configuration

Create a systemd service unit by adding the below mentioned config to a file called /etc/systemd/system/call-home.service.

[Unit]
Description=Forward local SSH port to remote host
After=network-online.target
Before=multi-user.target
DefaultDependencies=no

[Service]
# SSH connection uses the private key stored in this
# users home dir (~/.ssh/)
User=callhome

# SSH connection with port forwarding
# Forwards local port 22 to port 5000
ExecStart=/usr/bin/ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ServerAliveInterval=20 -o ServerAliveCountMax=1 -o ExitOnForwardFailure=yes -N -T -R5000:localhost:22 incoming@managedhost.example.com

# wait 60 seconds before trying to restart the connection
# if it disconnects 
RestartSec=60

# keep retrying no matter what
Restart=always

[Install]
WantedBy=multi-user.target

Ensure that this service starts at boot

root@raspberry-pi:~# systemctl  enable  call-home
Created symlink from /etc/systemd/system/multi-user.target.wants/call-home.service to /etc/systemd/system/call-home.service.
root@display1:~#

Start the service and test if port forwarding works

root@raspberry-pi:~# systemctl  start  call-home
# check to see if the connection was established 
root@raspberry-pi:~# sudo journalctl  -u call-home
Jun 25 18:03:00 raspberry-pi systemd[1]: Starting SSH reverse tunnelling...
Jun 25 18:03:00 raspberry-pi systemd[1]: Started SSH reverse tunnelling.
Jun 25 18:03:01 raspberry-pi ssh[23582]: Warning: Permanently added '1.2.3.4' (ECDSA) to the list of known hosts.

If everything worked, you should be able to connect to port 5000 on the managed host, authenticate and reach remote host, like so:

root@ip-172-31-20-1 :~# ssh -p 5000  pi@127.0.0.1
pi@127.0.0.1's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun 25 20:17:28 2016 from localhost
pi@raspberry-pi:~ $

Troubleshooting

When attempting to connect from managed host to remote host you get error ssh: connect to host 127.0.0.1 port 5000: Connection refused

  • On the remote host check if forwarding worked, like so:
pi@raspberry-pi:~ $ sudo journalctl  -u call-home