SSH Security Made Easy: An Introduction to ssh-audit

ssh-audit is a powerful tool designed to help you assess the security of your SSH servers (and clients!). It provides detailed information about the server’s configuration, supported algorithms, and potential vulnerabilities. In this guide, I’ll walk you through the steps to install ssh-audit and run your first security tests. Secure SSH configuration made easy.

Installation on Linux

  1. Clone the Repository: Open your terminal and clone the ssh-audit repository from GitHub:
    git clone https://github.com/jtesta/ssh-audit.git
  2. Navigate to the Directory: Change to the ssh-audit directory:
    cd ssh-audit
  3. Install Dependencies: Ensure you have Python installed on your system. If not, install it using your package manager. For example, on Ubuntu:
    sudo apt-get install python3

Installation on macOS

To install ssh-audit , run:
brew install ssh-audit
(You have already Brew installed, right ?)

Please check the ssh-audit url for many other setup options (Docker,Windows,etc.)

Test the SSH-Server against vulnerabilities

execute ssh-audit <hostname>
Replace <hostname> with the IP address or domain name of the SSH server you want to audit.

Example of Ubuntu’s 24.04 LTS default SSHD setup:

(if you add the -l warn switch you just get the vulnerabilities presented)

Interpreting the Results: ssh-audit will provide a detailed report of the server’s configuration, including supported key exchange algorithms, encryption ciphers, and MAC algorithms. Look for any warnings or recommendations to improve your server’s security.

Remediation

After running ssh-audit and identifying potential vulnerabilities or weak configurations in your SSH server, it’s important to take steps to remediate these issues. Below are examples of how to apply them:

Example for Ubuntu 24.04.1 LTS:

(Note: This is just an example. The example eliminates vulnerabilities for the SSH-daemon, but it can well be that this snippet does not fit for your setup. Handle with care)

This snippet creates a configuration file (51-ssh-harden_202412.conf) in directory /etc/ssh/sshd_config.d/ with the specified settings to enhance the security of your SSH server.

(SSHD restart required)



Example for RHEL 9.4

(Note: This is just an example. This example eliminates vulnerabilities for the SSH-daemon, but it can well be that this snippet does not fit for your setup. Handle with care)

(SSHD restart required)

Proof the remediation

run ssh-audit again!

Example-output after remediation:

How can I test if my SSH-Client is not vulnerable ?

If you run ssh-audit with the switch -c it creates an ssh-service on port 2222 and audits every connection attempt:

output after the login-attempt (ssh 127.0.0.1 -p 2222)


Make your SSH-communication more secure, if not the SSH-Service opens an attack surface for uninvited visitors.
Secure SSH configuration is Key!

Consider other additional security-steps like:
Secure your SSH communication with certificates
Lab setup: Secure your SSH communication with certificates
Fail2Ban: ban hosts that cause multiple authentication errors

..
.


Do you use NFTables or IPTables (or both) ?

Most major Linux distributions have adopted nftables as their default firewall framework, often using it under the hood for iptables commands. Here are some of the key distributions that support nftables:

  1. Debian: Starting with Debian Buster, nftables is the default backend for iptables.
  2. Ubuntu: From Ubuntu 20.10 (Groovy Gorilla) onwards, nftables is included and can be used as the default firewall framework.
  3. Fedora: Fedora has integrated nftables and uses it as the default firewall framework.
  4. Arch Linux: Arch Linux includes nftables and provides packages for easy installation and configuration.
  5. Red Hat Enterprise Linux (RHEL): RHEL 8 and later versions use nftables as the default packet filtering framework.

Let’s examine a fresh installed Ubuntu 24.04 LTS on an RPI:

What is iptables -V telling me ?

The system does not use the legacy iptables framework, instead it uses the nf_tables version of iptables which provides a bridge to the nftables infrastructure/framework.

to complete the knowledge we check the symbolic link of iptables:

Iptables-nft ruleset appears in the rule listing of nftables.

Is iptables-nft and nftables then the same ? No, but they share the infrastructure of nftables.

Here’s how they work together:

Compatibility Layer
iptables-nft: This is a variant of iptables that uses the nftables kernel API. When you use iptables commands, they are translated into nftables rules by this compatibility layer. This allows you to continue using familiar iptables commands while benefiting from the advanced features of nftables.
iptables-legacy: This is the traditional iptables that directly interacts with the kernel’s iptables API. If you use iptables-legacy, it operates independently of nftables and does not translate rules into nftables format.
Interaction
Rule Management: When you use iptables-nft, the rules you create are managed by nftables under the hood. This means that nftables takes precedence, and the rules are stored in the nftables ruleset.
Kernel API: Both iptables-nft and nftables use the same kernel API for packet filtering. This ensures that the packet matching and filtering behavior is consistent, regardless of which tool you use to create the rules.
Coexistence: If you use both iptables-legacy and nftables, they can coexist, but it’s generally recommended to stick with one framework to avoid conflicts and ensure consistency.

Best Practices

Transition to nftables: If you’re starting fresh or looking to modernize your firewall management, transitioning to nftables is recommended. It offers better performance, more features, and a simpler syntax.
Use iptables-nft: If you prefer using iptables commands, use the iptables-nft variant to take advantage of nftables’ capabilities while maintaining familiarity with iptables syntax.
By understanding how iptables and nftables interact, you can make informed decisions about managing your firewall rules and ensure a smooth transition to nftables.

Check out the official nftables wiki: http://wiki.nftables.org/

Secure your SSH communication with certificates

How about securing your SSH-Server to only support login-attempts including a valid signed certificate from a trusted CA ?


This sounds pretty cool, but there are a couple of pitfalls which should be outlined first:

  • OpenSSH supports cert-based authentication since version 5.4 (in 2010)
  • OpenSSH does not support x.509-certificates !
  • OpenSSH has implemented its own certificate format

OpenSSH’s decision to use its own proprietary SSH certificates instead of X.509 certificates, as outlined in RFC 6187 (no draft, proposed standard!), is rooted in several practical and technical reasons. Let’s dive into the details:

Simplicity and Efficiency

  1. Simplicity: OpenSSH certificates are designed to be simple and efficient. They contain only the necessary information for SSH authentication, such as the public key, name, expiration date, and associated permissionsThis simplicity makes them easier to implement and manage compared to the more complex X.509 certificates, which include a broader range of attributes and extensions.
  2. Efficiency: The lightweight nature of OpenSSH certificates means they are faster to process and verify. This efficiency is particularly important in environments with a large number of SSH connections, where performance can be a critical factor.

Security and Flexibility

  1. Security: OpenSSH certificates offer several security advantages. They are digitally signed, which means they cannot be altered without invalidating the signatureAdditionally, they support short-lived certificates, which automatically expire after a set period, reducing the risk of unauthorized access if a certificate is compromised.
  2. Flexibility: OpenSSH certificates provide flexibility in terms of configuration and usage. They allow for custom validity periods, source restrictions, command restrictions, and option enforcementThis level of customization is not as easily achievable with X.509 certificates, which are designed for a broader range of applications beyond SSH.

Management and Usability

  1. Centralized Management: OpenSSH certificates simplify the management of SSH access. Instead of managing individual public keys for each user and server, administrators can use a single Certificate Authority (CA) to issue and revoke certificatesThis centralized approach makes it easier to onboard and offboard users, as well as manage access permissions.
  2. Usability: The proprietary SSH certificate format is tailored specifically for SSH use cases, making it more user-friendly for administrators and developers who work primarily with SSH. The familiarity and ease of use of OpenSSH certificates can lead to better adoption and fewer implementation issues.

Is there any way to still use X.509-certificates with SSH ? Sure!
There are various products on the market available supporting X.509-based certificates like:
PKIX-SSH secure shell with X.509 v3 certificate support (OpenSSH patch for X.509-support)
Tectia® SSH Client/Server
wolfSSL
-and so on and so forth. This is no complete list 🙂

Keep in mind that big players like RedHat rely on the proprietary certificate-solution of OpenSSH


My (personal) Summary:
while X.509 certificates are widely used and supported for various applications, OpenSSH’s proprietary certificates offer a more streamlined, secure, and manageable solution for SSH authentication. The decision to use a proprietary format is driven by the need for simplicity, efficiency, security, flexibility, and ease of management. A patch of the OpenSSH-libraries is not needed.

When you lock down your SSH-daemon to only allow logins with valid certificates of your SSH-CA you start creating an additional security-layer for your SSH-Service.
Just think of securing the SSH-service on an internet-facing (Bastion-)hosts:
Without ssh-certs you need tools like Crowdsec, SshGuard, Fail2ban to e.g. jail hacking attempts, but you get still a lot of noise in your logs.
Fail2ban for example creates time-based filters based on the Source-IP of the hacking-attempt:

dynamic FW-entries:
To                         Action      From
--                         ------      ----
Anywhere                   REJECT      1.234.58.136               # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      51.161.153.48              # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      189.241.227.175            # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      193.32.162.79              # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.238             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.235             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.237             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.236             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      1.234.58.142               # by Fail2Ban after 2 attempts against sshd

logs:
2024-08-28 07:48:00,596 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 07:48:00
2024-08-28 08:01:05,385 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:01:05
2024-08-28 08:04:25,692 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 08:04:25
2024-08-28 08:13:23,523 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 08:13:23
2024-08-28 08:16:29,521 fail2ban.actions        [773]: NOTICE  [apache-noscript] Unban 64.227.153.228
2024-08-28 08:20:49,352 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:20:49
2024-08-28 08:27:37,117 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.128.142.238
2024-08-28 08:27:38,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 112.163.28.218
2024-08-28 08:27:54,621 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.134.110.112
2024-08-28 08:28:22,790 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.97.177.217
2024-08-28 08:31:20,214 fail2ban.actions        [773]: NOTICE  [sshd] Unban 117.83.178.140
2024-08-28 08:36:06,460 fail2ban.actions        [773]: NOTICE  [sshd] Unban 207.172.160.36
2024-08-28 08:36:20,119 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 08:36:19
2024-08-28 08:36:31,386 fail2ban.filter         [773]: INFO    [apache-noscript] Found 167.172.208.130 - 2024-08-28 08:36:31
2024-08-28 08:38:00,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.73.131
2024-08-28 08:38:15,008 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.221.80.92
2024-08-28 08:38:57,178 fail2ban.actions        [773]: NOTICE  [sshd] Unban 177.53.215.134
2024-08-28 08:39:12,549 fail2ban.actions        [773]: NOTICE  [sshd] Unban 115.73.209.212
2024-08-28 08:39:25,317 fail2ban.actions        [773]: NOTICE  [sshd] Unban 78.153.149.132
2024-08-28 08:39:54,106 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.239.254
2024-08-28 08:40:03,800 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:40:03
2024-08-28 08:40:29,317 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:40:29
2024-08-28 08:41:04,291 fail2ban.actions        [773]: NOTICE  [sshd] Unban 180.131.108.240
2024-08-28 08:43:08,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 36.26.76.62
2024-08-28 08:44:26,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 27.254.207.91
2024-08-28 08:59:30,066 fail2ban.filter         [773]: INFO    [sshd] Found 146.59.228.24 - 2024-08-28 08:59:30
2024-08-28 08:59:52,008 fail2ban.filter         [773]: INFO    [sshd] Found 1.218.138.131 - 2024-08-28 08:59:51
2024-08-28 09:01:00,569 fail2ban.filter         [773]: INFO    [sshd] Found 47.245.28.86 - 2024-08-28 09:01:00
2024-08-28 09:11:10,660 fail2ban.filter         [773]: INFO    [sshd] Found 140.143.171.137 - 2024-08-28 09:11:10
2024-08-28 09:15:49,726 fail2ban.filter         [773]: INFO    [sshd] Found 103.200.20.12 - 2024-08-28 09:15:49
2024-08-28 09:17:24,908 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 09:17:24
2024-08-28 09:26:50,534 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 09:26:50
2024-08-28 09:27:17,395 fail2ban.actions        [773]: NOTICE  [sshd] Unban 14.40.8.125
2024-08-28 09:27:46,027 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 09:27:46
2024-08-28 09:28:14,571 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:28:14
2024-08-28 09:30:05,778 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:30:05
2024-08-28 09:36:10,170 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:10
2024-08-28 09:36:33,524 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:36:33
2024-08-28 09:36:57,262 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:57
2024-08-28 09:36:57,711 fail2ban.actions        [773]: NOTICE  [sshd] Ban 189.241.227.175
2024-08-28 09:37:34,771 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:37:34
2024-08-28 09:37:35,064 fail2ban.actions        [773]: NOTICE  [sshd] Ban 51.161.153.48
2024-08-28 09:42:44,848 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:42:44
2024-08-28 09:44:11,137 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:44:10
2024-08-28 09:45:23,912 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:45:23
2024-08-28 09:45:23,929 fail2ban.actions        [773]: NOTICE  [sshd] Ban 1.234.58.136
2024-08-28 09:57:28,398 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 09:57:28
2024-08-28 10:10:03,236 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:304:200::6a05 - 2024-08-28 10:10:03
2024-08-28 10:12:18,780 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 10:12:18
2024-08-28 10:17:56,698 fail2ban.filter         [773]: INFO    [sshd] Found 202.190.50.129 - 2024-08-28 10:17:56
2024-08-28 10:28:07,991 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 10:28:07
2024-08-28 10:44:49,312 fail2ban.filter         [773]: INFO    [sshd] Found 51.68.143.159 - 2024-08-28 10:44:48
2024-08-28 10:48:23,088 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 10:48:23


Status for the jail: sshd
|- Filter
|  |- Currently failed:	4
|  |- Total failed:	2595
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	8
   |- Total banned:	611
   `- Banned IP list:	83.81.169.235 83.81.169.236 83.81.169.237 183.81.169.238 193.32.162.79 1.234.58.136 1.234.58.137 1.234.58.142


With cert-based ssh-authentication these tools are no more needed as a valid client-ssh-cert is mandatory.

I will create in the next days a howto based on the smallstep-solution to show the pros (and cons) for cert-based ssh from an DevOps perspective.


Duplicati RPi setup on 64-bit Ubuntu OS Jammy (22.04)

Setting up Duplicati on Ubuntu Jammy (22.04) for Raspberry Pi (RPI) is a great way to ensure your data is securely backed up. Duplicati is a free, open-source backup solution that allows you to store encrypted, incremental, and compressed backups on various cloud storage services and remote file servers. It supports a wide range of storage options, including Amazon S3, Google Drive, Dropbox, Onedrive and many more.

Duplicati uses AES-256 encryption to secure your data before it is uploaded, ensuring your privacy is protected from unauthorized access. The software is designed to handle network issues and interrupted backups efficiently, making it a reliable choice for online backups.

This guide will walk you through the process of installing and configuring Duplicati on your RPI running the 64-bit ARM version of Ubuntu Jammy.

Steps to setup Duplicati:
-Download the latest Duplicati-package (2.0.8.1 as of writing this page)

wget https://updates.duplicati.com/beta/duplicati_2.0.8.1-1_all.deb


Output:
--2024-08-26 11:12:53--  https://updates.duplicati.com/beta/duplicati_2.0.8.1-1_all.deb
Resolving updates.duplicati.com (updates.duplicati.com)... 2606:4700:7::60, 2a06:98c1:58::60, 172.66.0.96, ...
Connecting to updates.duplicati.com (updates.duplicati.com)|2606:4700:7::60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32505580 (31M) [application/x-debian-package]
Saving to: ‘duplicati_2.0.8.1-1_all.deb.1’

duplicati_2.0.8.1-1_all.deb.1                   100%[====================================================================================================>]  31.00M  11.3MB/s    in 2.7s

2024-08-26 11:12:56 (11.3 MB/s) - ‘duplicati_2.0.8.1-1_all.deb.1’ saved [32505580/32505580]

-Install the package

apt install ./duplicati_2.0.8.1-1_all.deb

Output:
.....
snip
.....
137 new root certificates were added to your trust store.
Import process completed.
Done
done.
Setting up duplicati (2.0.8.1-1) ...
Setting up mono-devel (6.8.0.105+dfsg-3.2) ...
update-alternatives: using /usr/bin/mono-csc to provide /usr/bin/cli-csc (c-sharp-compiler) in auto mode
update-alternatives: using /usr/bin/resgen to provide /usr/bin/cli-resgen (resource-file-generator) in auto mode
update-alternatives: using /usr/bin/al to provide /usr/bin/cli-al (assembly-linker) in auto mode
update-alternatives: using /usr/bin/sn to provide /usr/bin/cli-sn (strong-name-tool) in auto mode
Setting up mono-xsp4-base (4.2-2.3) ...
Setting up mono-xsp4 (4.2-2.3) ...
Using Mono XSP 4 port: 8084
Binding Mono XSP 4 address: 0.0.0.0
Use of uninitialized value $libs in concatenation (.) or string at /usr/sbin/mono-xsp4-update line 216.
Setting up monodoc-http (4.2-3.1) ...
Use of uninitialized value $libs in concatenation (.) or string at /usr/sbin/mono-xsp4-update line 216.
Processing triggers for libgdk-pixbuf-2.0-0:arm64 (2.42.8+dfsg-1ubuntu0.3) ...

As next we start the Duplicati-service and check if the service is running:

systemctl start duplicati
systemctl status duplicati

Output:
● duplicati.service - Duplicati web-server
     Loaded: loaded (/lib/systemd/system/duplicati.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-08-26 11:24:22 CEST; 4s ago
   Main PID: 1129 (mono)
      Tasks: 16 (limit: 4422)
     Memory: 55.7M
        CPU: 2.516s
     CGroup: /system.slice/duplicati.service
             ├─1129 DuplicatiServer /usr/lib/duplicati/Duplicati.Server.exe
             └─1133 /usr/bin/mono-sgen /usr/lib/duplicati/Duplicati.Server.exe

Aug 26 11:24:22 raspi24n systemd[1]: Started Duplicati web-server.

Duplicati uses port 8200 and binds the service per default to the loopback-address.

-To get access from remote to the Duplicati Gui edit /etc/default/duplicati and add “–webservice-interface=any” to the DAEMON_OPTS

vi /etc/default/duplicati

# Defaults for duplicati initscript
# sourced by /etc/init.d/duplicati
# installed at /etc/default/duplicati by the maintainer scripts

#
# This is a POSIX shell fragment
#

# Additional options that are passed to the Daemon.
DAEMON_OPTS="--webservice-interface=any"

-Restart the service

systemctl restart duplicati

Now you are able to access the Duplicati-GUI from remote on port 8200:
http://[IP-RPI]:8200

Make sure to set a password , close the page with <OK>:

In case you want to have the service activated at boot-time enable the service at startup:

systemctl enable duplicate

Output:
Created symlink /etc/systemd/system/multi-user.target.wants/duplicati.service → /lib/systemd/system/duplicati.service.

et voilà ! Duplicati is now up and running in a very basic setup.

To increase security make the Duplicati-page secure with adding a PKCS12-Container to have HTTPS-security enabled.

The Daemon-opts statement needs to be enhanced with:

--webservice-sslcertificatefile=[path]
--webservice-sslcertificatepassword=[password (empty is ok]
(in case the PKCS12-Container is unprotected(=no password set)), please keep in mind to add the second statement with an empty password. Additional instructions can be found on the Duplicati-docs.


Example-section:


ADD-ON: example to backup a folder to Onedrive
Duplicati’s robust features, including AES-256 encryption, ensure that your data is securely backed up and protected from unauthorized access. 

This small guide shows you how to create a daily, incremental Backup of a local RPI-folder to Onedrive:

Local folder to Backup: /daten/important_data
Onedrive Backup folder: /Backup/rpi_backup

Access the Duplicati-Gui and add a Backup: