Secure your SSH communication with certificates

How about securing your SSH-Server to only support login-attempts including a valid signed certificate from a trusted CA ?


This sounds pretty cool, but there are a couple of pitfalls which should be outlined first:

  • OpenSSH supports cert-based authentication since version 5.4 (in 2010)
  • OpenSSH does not support x.509-certificates !
  • OpenSSH has implemented its own certificate format

OpenSSH’s decision to use its own proprietary SSH certificates instead of X.509 certificates, as outlined in RFC 6187 (no draft, proposed standard!), is rooted in several practical and technical reasons. Let’s dive into the details:

Simplicity and Efficiency

  1. Simplicity: OpenSSH certificates are designed to be simple and efficient. They contain only the necessary information for SSH authentication, such as the public key, name, expiration date, and associated permissions. This simplicity makes them easier to implement and manage compared to the more complex X.509 certificates, which include a broader range of attributes and extensions.
  2. Efficiency: The lightweight nature of OpenSSH certificates means they are faster to process and verify. This efficiency is particularly important in environments with a large number of SSH connections, where performance can be a critical factor.

Security and Flexibility

  1. Security: OpenSSH certificates offer several security advantages. They are digitally signed, which means they cannot be altered without invalidating the signature. Additionally, they support short-lived certificates, which automatically expire after a set period, reducing the risk of unauthorized access if a certificate is compromised.
  2. Flexibility: OpenSSH certificates provide flexibility in terms of configuration and usage. They allow for custom validity periods, source restrictions, command restrictions, and option enforcement. This level of customization is not as easily achievable with X.509 certificates, which are designed for a broader range of applications beyond SSH.

Management and Usability

  1. Centralized Management: OpenSSH certificates simplify the management of SSH access. Instead of managing individual public keys for each user and server, administrators can use a single Certificate Authority (CA) to issue and revoke certificates. This centralized approach makes it easier to onboard and offboard users, as well as manage access permissions.
  2. Usability: The proprietary SSH certificate format is tailored specifically for SSH use cases, making it more user-friendly for administrators and developers who work primarily with SSH. The familiarity and ease of use of OpenSSH certificates can lead to better adoption and fewer implementation issues.

Is there any way to still use X.509-certificates with SSH ? Sure!
There are various products on the market available supporting X.509-based certificates like:
–PKIX-SSH secure shell with X.509 v3 certificate support (OpenSSH patch for X.509-support)
–Tectia® SSH Client/Server
–wolfSSL
-and so on and so forth. This is no complete list 🙂

Keep in mind that big players like RedHat rely on the proprietary certificate-solution of OpenSSH

My (personal) Summary:
while X.509 certificates are widely used and supported for various applications, OpenSSH’s proprietary certificates offer a more streamlined, secure, and manageable solution for SSH authentication. The decision to use a proprietary format is driven by the need for simplicity, efficiency, security, flexibility, and ease of management. A patch of the OpenSSH-libraries is not needed.

When you lock down your SSH-daemon to only allow logins with valid certificates of your SSH-CA you start creating an additional security-layer for your SSH-Service.
Just think of securing the SSH-service on an internet-facing (Bastion-)hosts:
Without ssh-certs you need tools like Crowdsec, SshGuard, Fail2ban to e.g. jail hacking attempts, but you get still a lot of noise in your logs.
Fail2ban for example creates time-based filters based on the Source-IP of the hacking-attempt:

dynamic FW-entries:
To                         Action      From
--                         ------      ----
Anywhere                   REJECT      1.234.58.136               # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      51.161.153.48              # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      189.241.227.175            # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      193.32.162.79              # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.238             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.235             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.237             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.236             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      1.234.58.142               # by Fail2Ban after 2 attempts against sshd

logs:
2024-08-28 07:48:00,596 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 07:48:00
2024-08-28 08:01:05,385 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:01:05
2024-08-28 08:04:25,692 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 08:04:25
2024-08-28 08:13:23,523 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 08:13:23
2024-08-28 08:16:29,521 fail2ban.actions        [773]: NOTICE  [apache-noscript] Unban 64.227.153.228
2024-08-28 08:20:49,352 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:20:49
2024-08-28 08:27:37,117 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.128.142.238
2024-08-28 08:27:38,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 112.163.28.218
2024-08-28 08:27:54,621 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.134.110.112
2024-08-28 08:28:22,790 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.97.177.217
2024-08-28 08:31:20,214 fail2ban.actions        [773]: NOTICE  [sshd] Unban 117.83.178.140
2024-08-28 08:36:06,460 fail2ban.actions        [773]: NOTICE  [sshd] Unban 207.172.160.36
2024-08-28 08:36:20,119 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 08:36:19
2024-08-28 08:36:31,386 fail2ban.filter         [773]: INFO    [apache-noscript] Found 167.172.208.130 - 2024-08-28 08:36:31
2024-08-28 08:38:00,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.73.131
2024-08-28 08:38:15,008 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.221.80.92
2024-08-28 08:38:57,178 fail2ban.actions        [773]: NOTICE  [sshd] Unban 177.53.215.134
2024-08-28 08:39:12,549 fail2ban.actions        [773]: NOTICE  [sshd] Unban 115.73.209.212
2024-08-28 08:39:25,317 fail2ban.actions        [773]: NOTICE  [sshd] Unban 78.153.149.132
2024-08-28 08:39:54,106 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.239.254
2024-08-28 08:40:03,800 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:40:03
2024-08-28 08:40:29,317 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:40:29
2024-08-28 08:41:04,291 fail2ban.actions        [773]: NOTICE  [sshd] Unban 180.131.108.240
2024-08-28 08:43:08,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 36.26.76.62
2024-08-28 08:44:26,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 27.254.207.91
2024-08-28 08:59:30,066 fail2ban.filter         [773]: INFO    [sshd] Found 146.59.228.24 - 2024-08-28 08:59:30
2024-08-28 08:59:52,008 fail2ban.filter         [773]: INFO    [sshd] Found 1.218.138.131 - 2024-08-28 08:59:51
2024-08-28 09:01:00,569 fail2ban.filter         [773]: INFO    [sshd] Found 47.245.28.86 - 2024-08-28 09:01:00
2024-08-28 09:11:10,660 fail2ban.filter         [773]: INFO    [sshd] Found 140.143.171.137 - 2024-08-28 09:11:10
2024-08-28 09:15:49,726 fail2ban.filter         [773]: INFO    [sshd] Found 103.200.20.12 - 2024-08-28 09:15:49
2024-08-28 09:17:24,908 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 09:17:24
2024-08-28 09:26:50,534 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 09:26:50
2024-08-28 09:27:17,395 fail2ban.actions        [773]: NOTICE  [sshd] Unban 14.40.8.125
2024-08-28 09:27:46,027 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 09:27:46
2024-08-28 09:28:14,571 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:28:14
2024-08-28 09:30:05,778 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:30:05
2024-08-28 09:36:10,170 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:10
2024-08-28 09:36:33,524 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:36:33
2024-08-28 09:36:57,262 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:57
2024-08-28 09:36:57,711 fail2ban.actions        [773]: NOTICE  [sshd] Ban 189.241.227.175
2024-08-28 09:37:34,771 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:37:34
2024-08-28 09:37:35,064 fail2ban.actions        [773]: NOTICE  [sshd] Ban 51.161.153.48
2024-08-28 09:42:44,848 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:42:44
2024-08-28 09:44:11,137 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:44:10
2024-08-28 09:45:23,912 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:45:23
2024-08-28 09:45:23,929 fail2ban.actions        [773]: NOTICE  [sshd] Ban 1.234.58.136
2024-08-28 09:57:28,398 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 09:57:28
2024-08-28 10:10:03,236 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:304:200::6a05 - 2024-08-28 10:10:03
2024-08-28 10:12:18,780 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 10:12:18
2024-08-28 10:17:56,698 fail2ban.filter         [773]: INFO    [sshd] Found 202.190.50.129 - 2024-08-28 10:17:56
2024-08-28 10:28:07,991 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 10:28:07
2024-08-28 10:44:49,312 fail2ban.filter         [773]: INFO    [sshd] Found 51.68.143.159 - 2024-08-28 10:44:48
2024-08-28 10:48:23,088 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 10:48:23


Status for the jail: sshd
|- Filter
|  |- Currently failed:	4
|  |- Total failed:	2595
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	8
   |- Total banned:	611
   `- Banned IP list:	83.81.169.235 83.81.169.236 83.81.169.237 183.81.169.238 193.32.162.79 1.234.58.136 1.234.58.137 1.234.58.142


With cert-based ssh-authentication these tools are no more needed as a valid client-ssh-cert is mandatory.

I will create in the next days a howto based on the smallstep-solution to show the pros (and cons) for cert-based ssh from an DevOps perspective.


Duplicati RPi setup on 64-bit Ubuntu OS Jammy (22.04)

Setting up Duplicati on Ubuntu Jammy (22.04) for Raspberry Pi (RPI) is a great way to ensure your data is securely backed up. Duplicati is a free, open-source backup solution that allows you to store encrypted, incremental, and compressed backups on various cloud storage services and remote file servers. It supports a wide range of storage options, including Amazon S3, Google Drive, Dropbox, Onedrive and many more.

Duplicati uses AES-256 encryption to secure your data before it is uploaded, ensuring your privacy is protected from unauthorized access. The software is designed to handle network issues and interrupted backups efficiently, making it a reliable choice for online backups.

This guide will walk you through the process of installing and configuring Duplicati on your RPI running the 64-bit ARM version of Ubuntu Jammy.

Steps to setup Duplicati:
-Download the latest Duplicati-package (2.0.8.1 as of writing this page)

wget https://updates.duplicati.com/beta/duplicati_2.0.8.1-1_all.deb


Output:
--2024-08-26 11:12:53--  https://updates.duplicati.com/beta/duplicati_2.0.8.1-1_all.deb
Resolving updates.duplicati.com (updates.duplicati.com)... 2606:4700:7::60, 2a06:98c1:58::60, 172.66.0.96, ...
Connecting to updates.duplicati.com (updates.duplicati.com)|2606:4700:7::60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32505580 (31M) [application/x-debian-package]
Saving to: ‘duplicati_2.0.8.1-1_all.deb.1’

duplicati_2.0.8.1-1_all.deb.1                   100%[====================================================================================================>]  31.00M  11.3MB/s    in 2.7s

2024-08-26 11:12:56 (11.3 MB/s) - ‘duplicati_2.0.8.1-1_all.deb.1’ saved [32505580/32505580]

-Install the package

apt install ./duplicati_2.0.8.1-1_all.deb

Output:
.....
snip
.....
137 new root certificates were added to your trust store.
Import process completed.
Done
done.
Setting up duplicati (2.0.8.1-1) ...
Setting up mono-devel (6.8.0.105+dfsg-3.2) ...
update-alternatives: using /usr/bin/mono-csc to provide /usr/bin/cli-csc (c-sharp-compiler) in auto mode
update-alternatives: using /usr/bin/resgen to provide /usr/bin/cli-resgen (resource-file-generator) in auto mode
update-alternatives: using /usr/bin/al to provide /usr/bin/cli-al (assembly-linker) in auto mode
update-alternatives: using /usr/bin/sn to provide /usr/bin/cli-sn (strong-name-tool) in auto mode
Setting up mono-xsp4-base (4.2-2.3) ...
Setting up mono-xsp4 (4.2-2.3) ...
Using Mono XSP 4 port: 8084
Binding Mono XSP 4 address: 0.0.0.0
Use of uninitialized value $libs in concatenation (.) or string at /usr/sbin/mono-xsp4-update line 216.
Setting up monodoc-http (4.2-3.1) ...
Use of uninitialized value $libs in concatenation (.) or string at /usr/sbin/mono-xsp4-update line 216.
Processing triggers for libgdk-pixbuf-2.0-0:arm64 (2.42.8+dfsg-1ubuntu0.3) ...

As next we start the Duplicati-service and check if the service is running:

systemctl start duplicati
systemctl status duplicati

Output:
● duplicati.service - Duplicati web-server
     Loaded: loaded (/lib/systemd/system/duplicati.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-08-26 11:24:22 CEST; 4s ago
   Main PID: 1129 (mono)
      Tasks: 16 (limit: 4422)
     Memory: 55.7M
        CPU: 2.516s
     CGroup: /system.slice/duplicati.service
             ├─1129 DuplicatiServer /usr/lib/duplicati/Duplicati.Server.exe
             └─1133 /usr/bin/mono-sgen /usr/lib/duplicati/Duplicati.Server.exe

Aug 26 11:24:22 raspi24n systemd[1]: Started Duplicati web-server.

Duplicati uses port 8200 and binds the service per default to the loopback-address.

-To get access from remote to the Duplicati Gui edit /etc/default/duplicati and add “–webservice-interface=any” to the DAEMON_OPTS

vi /etc/default/duplicati

# Defaults for duplicati initscript
# sourced by /etc/init.d/duplicati
# installed at /etc/default/duplicati by the maintainer scripts

#
# This is a POSIX shell fragment
#

# Additional options that are passed to the Daemon.
DAEMON_OPTS="--webservice-interface=any"

-Restart the service

systemctl restart duplicati

Now you are able to access the Duplicati-GUI from remote on port 8200:
http://[IP-RPI]:8200

Make sure to set a password , close the page with <OK>:

In case you want to have the service activated at boot-time enable the service at startup:

systemctl enable duplicate

Output:
Created symlink /etc/systemd/system/multi-user.target.wants/duplicati.service → /lib/systemd/system/duplicati.service.

et voilà ! Duplicati is now up and running in a very basic setup.

To increase security make the Duplicati-page secure with adding a PKCS12-Container to have HTTPS-security enabled.

The Daemon-opts statement needs to be enhanced with:
--webservice-sslcertificatefile=[path]
--webservice-sslcertificatepassword=[password (empty is ok]
(in case the PKCS12-Container is unprotected(=no password set)), please keep in mind to add the second statement with an empty password. Additional instructions can be found on the Duplicati-docs.

Example-section:

ADD-ON: example to backup a folder to Onedrive
Duplicati’s robust features, including AES-256 encryption, ensure that your data is securely backed up and protected from unauthorized access. 

This small guide shows you how to create a daily, incremental Backup of a local RPI-folder to Onedrive:

Local folder to Backup: /daten/important_data
Onedrive Backup folder: /Backup/rpi_backup

Access the Duplicati-Gui and add a Backup:

Bosch Heat-pump, Influx, Grafana and Home-Assistant: get the water-temperature presented

Used Hard/Software:
Home Assistant Frontend 20240806.1
Core 2024.8.0
Supervisor 2024.08.0
Operating System 12.4

HA-Integration: Bosch custom component v0.26.1

It was a hard time to find the right Grafana-settings to get the water-temperature presented.
Bosch/Home-Assistant transports this value based on the state-series:

Hence Grafana needs a statement as shown:

Example output:

Done 🙂

iPad Samba file transfer fails

To be able to copy files over from iPad to Samba-shares the enhanced compatibility with Apple SMB clients and interoperability with Netatalk 3 AFP fileserver must be enabled.

Vfs-module is typically missing if you see in tshark messages like:

310 Create Response, Error: STATUS_OBJECT_NAME_INVALID;Read Response, Error: STATUS_OBJECT_NAME_INVALID;Close Response, Error: STATUS_OBJECT_NAME_INVALID

To get this fixed add to the [global] section in /etc/samba/smb.conf the following command:

vfs objects = fruit streams_xattr

The package samba-vfs-modules must be installed prior restarting the smb service.
Check if the package is installed with commands like:

ubuntu: dpkg -l samba-vfs-modules
RHEL. : rpm -qa | grep  samba-vfs-modules
etc.

a positive output looks similar to:
root@raspi24:/home/ugu5ma# dpkg -l samba-vfs-modules

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                    Version                   
+++-=======================-=========================
ii  samba-vfs-modules:arm64 2:4.15.13+dfsg-0ubuntu1.5

If the package is missing add it with:

ubuntu: apt install samba-vfs-modules
RHEL. : yum install samba-vfs-modules
etc.

Restart Samba:

systemctl restart smbd

Done!