Category: Security

Secure your SSH communication with certificates

How about securing your SSH-Server to only support login-attempts including a valid signed certificate from a trusted CA ?


This sounds pretty cool, but there are a couple of pitfalls which should be outlined first:

  • OpenSSH supports cert-based authentication since version 5.4 (in 2010)
  • OpenSSH does not support x.509-certificates !
  • OpenSSH has implemented its own certificate format

OpenSSH’s decision to use its own proprietary SSH certificates instead of X.509 certificates, as outlined in RFC 6187 (no draft, proposed standard!), is rooted in several practical and technical reasons. Let’s dive into the details:

Simplicity and Efficiency

  1. Simplicity: OpenSSH certificates are designed to be simple and efficient. They contain only the necessary information for SSH authentication, such as the public key, name, expiration date, and associated permissions. This simplicity makes them easier to implement and manage compared to the more complex X.509 certificates, which include a broader range of attributes and extensions.
  2. Efficiency: The lightweight nature of OpenSSH certificates means they are faster to process and verify. This efficiency is particularly important in environments with a large number of SSH connections, where performance can be a critical factor.

Security and Flexibility

  1. Security: OpenSSH certificates offer several security advantages. They are digitally signed, which means they cannot be altered without invalidating the signature. Additionally, they support short-lived certificates, which automatically expire after a set period, reducing the risk of unauthorized access if a certificate is compromised.
  2. Flexibility: OpenSSH certificates provide flexibility in terms of configuration and usage. They allow for custom validity periods, source restrictions, command restrictions, and option enforcement. This level of customization is not as easily achievable with X.509 certificates, which are designed for a broader range of applications beyond SSH.

Management and Usability

  1. Centralized Management: OpenSSH certificates simplify the management of SSH access. Instead of managing individual public keys for each user and server, administrators can use a single Certificate Authority (CA) to issue and revoke certificates. This centralized approach makes it easier to onboard and offboard users, as well as manage access permissions.
  2. Usability: The proprietary SSH certificate format is tailored specifically for SSH use cases, making it more user-friendly for administrators and developers who work primarily with SSH. The familiarity and ease of use of OpenSSH certificates can lead to better adoption and fewer implementation issues.

Is there any way to still use X.509-certificates with SSH ? Sure!
There are various products on the market available supporting X.509-based certificates like:
–PKIX-SSH secure shell with X.509 v3 certificate support (OpenSSH patch for X.509-support)
–Tectia® SSH Client/Server
–wolfSSL
-and so on and so forth. This is no complete list 🙂

Keep in mind that big players like RedHat rely on the proprietary certificate-solution of OpenSSH

My (personal) Summary:
while X.509 certificates are widely used and supported for various applications, OpenSSH’s proprietary certificates offer a more streamlined, secure, and manageable solution for SSH authentication. The decision to use a proprietary format is driven by the need for simplicity, efficiency, security, flexibility, and ease of management. A patch of the OpenSSH-libraries is not needed.

When you lock down your SSH-daemon to only allow logins with valid certificates of your SSH-CA you start creating an additional security-layer for your SSH-Service.
Just think of securing the SSH-service on an internet-facing (Bastion-)hosts:
Without ssh-certs you need tools like Crowdsec, SshGuard, Fail2ban to e.g. jail hacking attempts, but you get still a lot of noise in your logs.
Fail2ban for example creates time-based filters based on the Source-IP of the hacking-attempt:

dynamic FW-entries:
To                         Action      From
--                         ------      ----
Anywhere                   REJECT      1.234.58.136               # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      51.161.153.48              # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      189.241.227.175            # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      193.32.162.79              # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.238             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.235             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.237             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.236             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      1.234.58.142               # by Fail2Ban after 2 attempts against sshd

logs:
2024-08-28 07:48:00,596 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 07:48:00
2024-08-28 08:01:05,385 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:01:05
2024-08-28 08:04:25,692 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 08:04:25
2024-08-28 08:13:23,523 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 08:13:23
2024-08-28 08:16:29,521 fail2ban.actions        [773]: NOTICE  [apache-noscript] Unban 64.227.153.228
2024-08-28 08:20:49,352 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:20:49
2024-08-28 08:27:37,117 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.128.142.238
2024-08-28 08:27:38,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 112.163.28.218
2024-08-28 08:27:54,621 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.134.110.112
2024-08-28 08:28:22,790 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.97.177.217
2024-08-28 08:31:20,214 fail2ban.actions        [773]: NOTICE  [sshd] Unban 117.83.178.140
2024-08-28 08:36:06,460 fail2ban.actions        [773]: NOTICE  [sshd] Unban 207.172.160.36
2024-08-28 08:36:20,119 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 08:36:19
2024-08-28 08:36:31,386 fail2ban.filter         [773]: INFO    [apache-noscript] Found 167.172.208.130 - 2024-08-28 08:36:31
2024-08-28 08:38:00,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.73.131
2024-08-28 08:38:15,008 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.221.80.92
2024-08-28 08:38:57,178 fail2ban.actions        [773]: NOTICE  [sshd] Unban 177.53.215.134
2024-08-28 08:39:12,549 fail2ban.actions        [773]: NOTICE  [sshd] Unban 115.73.209.212
2024-08-28 08:39:25,317 fail2ban.actions        [773]: NOTICE  [sshd] Unban 78.153.149.132
2024-08-28 08:39:54,106 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.239.254
2024-08-28 08:40:03,800 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:40:03
2024-08-28 08:40:29,317 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:40:29
2024-08-28 08:41:04,291 fail2ban.actions        [773]: NOTICE  [sshd] Unban 180.131.108.240
2024-08-28 08:43:08,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 36.26.76.62
2024-08-28 08:44:26,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 27.254.207.91
2024-08-28 08:59:30,066 fail2ban.filter         [773]: INFO    [sshd] Found 146.59.228.24 - 2024-08-28 08:59:30
2024-08-28 08:59:52,008 fail2ban.filter         [773]: INFO    [sshd] Found 1.218.138.131 - 2024-08-28 08:59:51
2024-08-28 09:01:00,569 fail2ban.filter         [773]: INFO    [sshd] Found 47.245.28.86 - 2024-08-28 09:01:00
2024-08-28 09:11:10,660 fail2ban.filter         [773]: INFO    [sshd] Found 140.143.171.137 - 2024-08-28 09:11:10
2024-08-28 09:15:49,726 fail2ban.filter         [773]: INFO    [sshd] Found 103.200.20.12 - 2024-08-28 09:15:49
2024-08-28 09:17:24,908 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 09:17:24
2024-08-28 09:26:50,534 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 09:26:50
2024-08-28 09:27:17,395 fail2ban.actions        [773]: NOTICE  [sshd] Unban 14.40.8.125
2024-08-28 09:27:46,027 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 09:27:46
2024-08-28 09:28:14,571 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:28:14
2024-08-28 09:30:05,778 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:30:05
2024-08-28 09:36:10,170 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:10
2024-08-28 09:36:33,524 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:36:33
2024-08-28 09:36:57,262 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:57
2024-08-28 09:36:57,711 fail2ban.actions        [773]: NOTICE  [sshd] Ban 189.241.227.175
2024-08-28 09:37:34,771 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:37:34
2024-08-28 09:37:35,064 fail2ban.actions        [773]: NOTICE  [sshd] Ban 51.161.153.48
2024-08-28 09:42:44,848 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:42:44
2024-08-28 09:44:11,137 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:44:10
2024-08-28 09:45:23,912 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:45:23
2024-08-28 09:45:23,929 fail2ban.actions        [773]: NOTICE  [sshd] Ban 1.234.58.136
2024-08-28 09:57:28,398 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 09:57:28
2024-08-28 10:10:03,236 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:304:200::6a05 - 2024-08-28 10:10:03
2024-08-28 10:12:18,780 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 10:12:18
2024-08-28 10:17:56,698 fail2ban.filter         [773]: INFO    [sshd] Found 202.190.50.129 - 2024-08-28 10:17:56
2024-08-28 10:28:07,991 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 10:28:07
2024-08-28 10:44:49,312 fail2ban.filter         [773]: INFO    [sshd] Found 51.68.143.159 - 2024-08-28 10:44:48
2024-08-28 10:48:23,088 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 10:48:23


Status for the jail: sshd
|- Filter
|  |- Currently failed:	4
|  |- Total failed:	2595
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	8
   |- Total banned:	611
   `- Banned IP list:	83.81.169.235 83.81.169.236 83.81.169.237 183.81.169.238 193.32.162.79 1.234.58.136 1.234.58.137 1.234.58.142


With cert-based ssh-authentication these tools are no more needed as a valid client-ssh-cert is mandatory.

I will create in the next days a howto based on the smallstep-solution to show the pros (and cons) for cert-based ssh from an DevOps perspective.


Pi-hole: send pihole.log and pihole-FTL.log to external Syslog-server

The option to send Pihole-logs to a remote syslogserver is not implemented.
Good thing is that Pi-hole creates log-files.
How about adding these logfiles to the local (R-)Syslog-daemon and send it over to the external Syslog-Server via port 514 TCP/UDP ?

I use Dietpi as the OS on my Raspi3+ and added Pi-hole as one of the supported apps .

Dietpi has per default noSyslog-daemon activated.
The first step is to install a Syslog-daemon, in my case Rsyslog.
Install with root-privileges (sudo su):

apt install rsyslog

cd to /etc/rsyslog.d and create two files:

nano pihole.conf

$InputFileName /var/log/pihole.log
$InputFileTag pihole
$InputRunFileMonitor
$InputFilePersistStateInterval 1000
nano piholeftl.conf

$InputFileName /var/log/pihole-FTL.log
$InputFileTag pihole-ftl
$InputRunFileMonitor
$InputFilePersistStateInterval 1000

as next alter rsyslog.conf and add the following lines

nano /etc/rsyslog.conf

$ModLoad imfile
$InputFileName /var/log/pihole.log
$InputFileName /var/log/pihole-FTL.log
*.* @10.50.100.5:514

10.50.100.5 is the external Syslog-Server. Dietpi sends now the syslog-information to the Syslog-Server 10.50.100.5 via TCP port 514.
Change the IP 10.50.100.5 to the Syslog-Server IP you want to use.

restart Rsyslog-daemon:

systemctl restart rsyslog

The Syslog-daemon receives now the Pi-hole logs:

Addition: If you want to see also the DNS-queries go to Settings and Enable query logging:

This brings also the DNS content to the external Syslog-Server:

May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: query[A] bier.de from 10.50.100.13
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: forwarded bier.de to 1.0.0.1
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: dnssec-query[DS] bier.de to 1.0.0.1
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: reply bier.de is no DS
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: reply bier.de is 212.53.128.75
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: query[AAAA] bier.de from 10.50.100.13
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: forwarded bier.de to 1.0.0.1
May 14 09:58:27 DietPi local0 pihole May 14 09:58:26 dnsmasq[15529]: reply bier.de is NODATA-IPv6

Done!