Do you use NFTables or IPTables (or both) ?

Most major Linux distributions have adopted nftables as their default firewall framework, often using it under the hood for iptables commands. Here are some of the key distributions that support nftables:

  1. Debian: Starting with Debian Buster, nftables is the default backend for iptables.
  2. Ubuntu: From Ubuntu 20.10 (Groovy Gorilla) onwards, nftables is included and can be used as the default firewall framework.
  3. Fedora: Fedora has integrated nftables and uses it as the default firewall framework.
  4. Arch Linux: Arch Linux includes nftables and provides packages for easy installation and configuration.
  5. Red Hat Enterprise Linux (RHEL): RHEL 8 and later versions use nftables as the default packet filtering framework.

Let’s examine a fresh installed Ubuntu 24.04 LTS on an RPI:

What is iptables -V telling me ?

┌──$(root㉿raspi24n)-[/]
└─# iptables -V
iptables v1.8.10 (nf_tables)

The system does not use the legacy iptables framework, instead it uses the nf_tables version of iptables which provides a bridge to the nftables infrastructure/framework.

to complete the knowledge we check the symbolic link of iptables:

┌──$(root㉿raspi24n)-[/]
└─# ls -al /usr/sbin/iptables
lrwxrwxrwx 1 root root 26 Apr  8  2024 /usr/sbin/iptables -> /etc/alternatives/iptables

└─# ls -l /etc/alternatives/iptables
lrwxrwxrwx 1 root root 22 Aug 27 16:29 /etc/alternatives/iptables -> /usr/sbin/iptables-nft

Iptables-nft ruleset appears in the rule listing of nftables.

Is iptables-nft and nftables then the same ? No, but they share the infrastructure of nftables.

Here’s how they work together:

Compatibility Layer
iptables-nft: This is a variant of iptables that uses the nftables kernel API. When you use iptables commands, they are translated into nftables rules by this compatibility layer. This allows you to continue using familiar iptables commands while benefiting from the advanced features of nftables.
iptables-legacy: This is the traditional iptables that directly interacts with the kernel’s iptables API. If you use iptables-legacy, it operates independently of nftables and does not translate rules into nftables format.
Interaction
Rule Management: When you use iptables-nft, the rules you create are managed by nftables under the hood. This means that nftables takes precedence, and the rules are stored in the nftables ruleset.
Kernel API: Both iptables-nft and nftables use the same kernel API for packet filtering. This ensures that the packet matching and filtering behavior is consistent, regardless of which tool you use to create the rules.
Coexistence: If you use both iptables-legacy and nftables, they can coexist, but it’s generally recommended to stick with one framework to avoid conflicts and ensure consistency.

Best Practices

Transition to nftables: If you’re starting fresh or looking to modernize your firewall management, transitioning to nftables is recommended. It offers better performance, more features, and a simpler syntax.
Use iptables-nft: If you prefer using iptables commands, use the iptables-nft variant to take advantage of nftables’ capabilities while maintaining familiarity with iptables syntax.
By understanding how iptables and nftables interact, you can make informed decisions about managing your firewall rules and ensure a smooth transition to nftables.

Check out the official nftables wiki: http://wiki.nftables.org/

MacOS: Streamline Your App Management with MAS

Managing apps on your Mac can sometimes feel like a chore, especially when you have to manually update, install, or remove them.

Enter MAS (Mac App Store command line interface), a powerful tool that simplifies these tasks and more.
In this blog post I explore how MAS can make your life easier and why you should consider adding it to your toolkit.

What is MAS?

MAS is a command-line interface for the Mac App Store, allowing you to manage your apps directly from the terminal. With MAS, you can search for, install, update, and remove apps without ever opening the App Store. This can be particularly useful for developers, system administrators, or anyone who prefers using the terminal for their workflows.

Installing MAS

Before you can start using MAS, you’ll need to install it. The easiest way to do this is through Homebrew, a popular package manager for macOS. If you don’t have Homebrew installed yet, you can install it by running the following command in your terminal:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Once Homebrew is installed, you can install MAS with the following command:

brew install mas

Benefits of Using MAS

  1. Efficiency: MAS manage your apps quickly and efficiently from the terminal. No more clicking through the App Store interface—just type a few commands, and you’re done.
  2. Automation: With MAS, you can script your app management tasks. This is particularly useful for setting up new machines or maintaining multiple systems.
  3. Consistency: Using MAS ensures that your app management process is consistent across all your devices. You can easily replicate the same setup on different machines.
  4. Control: MAS gives you more control over your app management. You can see detailed information about your installed apps and manage them in a way that suits your needs.

Disadvantages of the Old App Store App

App Store App

While the App Store app on macOS has its merits, it also comes with several disadvantages:

  1. Cumbersome Interface: The graphical interface can be slow and cumbersome, especially when managing multiple apps. Navigating through various menus and options can be time-consuming.
  2. Lack of Automation: The App Store app does not support automation, making it difficult to script repetitive tasks. This can be a significant drawback for users who need to manage multiple devices or frequently install and update apps.
  3. Limited Control: The App Store app provides limited control over app management. Users cannot easily access detailed information about installed apps or manage them in bulk.
  4. Inconsistent Performance: The performance of the App Store app can be inconsistent, with occasional slowdowns and crashes. This can be frustrating, especially when trying to quickly install or update apps.

Getting Started with MAS

Here are a few basic commands to get you started with MAS:

  • Search for an app: mas search <app-name>
  • Install an app: mas install <app-id>
  • List installed apps: mas list
  • Update all apps: mas upgrade
  • Remove an app: mas uninstall <app-id>

Example to upgrade your current installed apps:

┌──(ugu5ma㉿iMacPro)-[~]
└─$ mas upgrade
Upgrading 8 outdated applications:
Final Cut Pro (10.8.1) -> (11.0)
Microsoft PowerPoint (16.90.2) -> (16.91)
Motion (5.8) -> (5.9)
iMovie (10.4.2) -> (10.4.3)
Microsoft Remote Desktop (10.9.10) -> (11.0.6)
Logic Pro (11.0.1) -> (11.1)
Compressor (4.8) -> (4.9)
Microsoft Word (16.90.2) -> (16.91)
==> Downloaded Final Cut Pro (11.0)
==> Installing Final Cut Pro (11.0)
==> Installed Final Cut Pro (11.0)
==> Downloading Microsoft PowerPoint (16.91)
==> Downloaded Microsoft PowerPoint (16.91)
==> Installing Microsoft PowerPoint (16.91)
==> Installed Microsoft PowerPoint (16.91)
==> Downloading Motion (5.9)
==> Downloaded Motion (5.9)
==> Installing Motion (5.9)
==> Installed Motion (5.9)
==> Downloading iMovie (10.4.3)
==> Downloaded iMovie (10.4.3)
==> Installing iMovie (10.4.3)
==> Installed iMovie (10.4.3)
==> Downloading Windows App (11.0.6)
==> Downloaded Windows App (11.0.6)
==> Installing Windows App (11.0.6)
==> Installed Windows App (11.0.6)
==> Downloading Logic Pro (11.1)
==> Downloaded Logic Pro (11.1)
==> Installing Logic Pro (11.1)
==> Installed Logic Pro (11.1)
==> Downloading Compressor (4.9)
==> Downloaded Compressor (4.9)
==> Installing Compressor (4.9)
==> Installed Compressor (4.9)
==> Downloading Microsoft Word (16.91)
==> Downloaded Microsoft Word (16.91)
==> Installing Microsoft Word (16.91)
==> Installed Microsoft Word (16.91)

┌──(ugu5ma㉿iMacPro)-[~]
└─$

For a full list of commands and options, you can check the MAS documentation.

MAS is a powerful tool that can save you time and effort when managing your Mac apps. Whether you’re a developer, a system administrator, or just someone who loves the terminal, MAS is worth checking out. Give it a try and see how it can streamline your app management process.

Lab setup: Secure your SSH communication with certificates

When you Ssh the first time to a host the screen shows something like:

ssh test@10.50.100.110
The authenticity of host '10.50.100.110 (10.50.100.110)' can't be established.
ED25519 key fingerprint is SHA256:jCJ0TIJkKnjgu3RTv5eGER7p4IN5Tb/JpTEVJNMfpMs.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Be honest: Do you just accept the shown Fingerprint of the remote host or do you really doublecheck the presented fingerprint before you accept ?

My guess: Most of the time the presented fingerprint gets accepted without any additional check.

This procedure is called TOFU (Trust On First Use).
TOFU assumes that the first time you connect to a server, the server’s key is trustworthy. However, this can leave you vulnerable to man-in-the-middle attacks if an attacker intercepts the initial connection. The attacker grabs your user/password credentials and can get access.

Here are some implementations who are using TOFU:

Android Enterprise Networks: Android supports TOFU for enterprise networks by allowing devices to trust an enterprise network by installing the root CA used by the server and setting its domain name in a saved network.

Signal: In Signal, endpoints initially trust the identifier blindly and display non-blocking warnings when it changes. Users can verify the identifier by scanning a QR code or exchanging a Safety Number, which then changes the nature of identifier change warnings from non-blocking to blocking.

WhatsApp: WhatsApp clients initially trust the identifier blindly and, by default, do not display warnings when the identifier changes. Users can enable non-blocking warnings by accessing the key fingerprint (called Security Code) and verifying it.

XMPP Client Conversations: This client uses Blind Trust Before Verification, where all identifiers are blindly trusted until the user authenticates endpoints by scanning a QR code. Once verified, the client displays a shield symbol for authenticated messages.

By using certificates for SSH authentication, you can eliminate the risks associated with TOFU. Key approval (and distribution) is no more necessary with certificate-based Ssh.

Certificates provide a more robust and secure method of verifying identities, ensuring that both the client and server can trust each other without relying on the initial Ssh-connection’s integrity.

This lab-setup is based on the open-source version of Smallstep, a powerful tool that simplifies the process of managing and issuing certificates.
Want to test cert-based ssh without smallstep ? go ahead, here are some getting started urls:
–https://goteleport.com/blog/how-to-configure-ssh-certificate-based-authentication/
–https://blog.habets.se/2011/07/OpenSSH-certificates.html
–https://www.sweharris.org/post/2016-10-30-ssh-certs/

Before we dive into the lab, let’s first take a look at some key pros and cons related to this topic.

Advantages when using certificate-based ssh:

  • Enhanced Security: Certificates provide a higher level of security compared to traditional SSH keys. They are issued by a trusted Certificate Authority (CA), which ensures that both the client and server are who they claim to be.
  • Elimination of TOFU Risks: By using certificates, you eliminate the Trust On First Use (TOFU) risks associated with traditional SSH key exchanges. This reduces the chances of man-in-the-middle attacks during the initial connection.
  • Simplified Key Management: Certificates simplify the management of SSH keys. They can be easily issued, renewed, and revoked by the CA, making it easier to manage credentials in large environments.
  • Scalability: Certificate-based authentication scales well in large environments. Automated tools can be used to handle the issuance, renewal, and revocation of certificates, making it easier to manage a large number of users and systems.
  • Improved Compliance: Using certificates can help meet regulatory and compliance requirements for secure communications. This provides an added layer of assurance for your organization.
  • Revocation Capability: If a certificate is compromised, it can be revoked by the CA, rendering it invalid. This provides a way to quickly respond to security incidents and prevent unauthorized access.
  • Expiration and Renewal: Certificates have a defined validity period, which means they need to be renewed periodically. This ensures that outdated or potentially compromised credentials are regularly updated.


Disadvantages when using certificate-based ssh:

  • Initial Setup Time: The initial setup of a certificate authority (CA) and the issuance of certificates can be time-consuming. This includes configuring the CA, generating certificates, and distributing them to clients and servers.
  • Complexity: Setting up and managing a certificate-based SSH infrastructure can be more complex compared to traditional key-based authentication. It requires a good understanding of Public Key Infrastructure (PKI) and certificate management.
  • Compatibility: Not all systems and applications may support certificate-based SSH authentication out of the box. Additional configuration or software may be required to enable this feature.
  • Maintenance: Regular maintenance is required to keep the certificate infrastructure secure. This includes renewing certificates before they expire, revoking compromised certificates, and ensuring the CA remains secure.
  • Dependency on CA: The security of the entire system relies on the integrity and security of the CA. If the CA is compromised, the entire certificate-based authentication system can be at risk.

OK. Ready to go ?

This is the lab setup:

HostnameMac-addressIP-address
step 00:0c:29:fd:94:15 10.50.100.110
clientstep100:0c:29:3d:c8:a5 10.50.100.111
clientstep200:0c:29:41:a1:5c 10.50.100.112
rogueclient00:0c:af:ee:af:fe 
# install needed packages
wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.rpm
rpm -i step-cli_amd64.rpm
wget https://dl.smallstep.com/certificates/docs-ca-install/latest/step-ca_amd64.rpm
sudo rpm -i step-ca_amd64.rpm

#check version(s)
[root@step step]# step version
Smallstep CLI/0.27.2 (linux/amd64)
Release Date: 2024-07-18T18:15:09Z

[root@step step]# step-ca version
Smallstep CA/0.27.2 (linux/amd64)
Release Date: 2024-07-18T21:29:11Z
[root@step step]#

#setup step ca
[root@step step]# step ca init -ssh
Use the arrow keys to navigate: ↓ ↑ → ←
? What deployment type would you like to configure?:
  ▸ Standalone - step-ca instance you run yourself
    Linked - standalone, plus cloud configuration, reporting & alerting
    Hosted - fully-managed step-ca cloud instance run for you by smallstep
✔ Deployment Type: Standalone
What would you like to name your new PKI?
✔ (e.g. Smallstep): step
What DNS names or IP addresses will clients use to reach your CA?
✔ (e.g. ca.example.com[,10.1.2.3,etc.]): step,10.50.100.110
What IP and port will your new CA bind to? (:443 will bind to 0.0.0.0:443)
✔ (e.g. :443 or 127.0.0.1:443): :443█
What would you like to name the CA's first provisioner?
✔ (e.g. you@smallstep.com): ugu5ma@step
Choose a password for your CA keys and first provisioner.
✔ [leave empty and we'll generate one]:

Generating root certificate... done!
Generating intermediate certificate... done!
Generating user and host SSH certificate signing keys... done!

✔ Root certificate: /root/.step/certs/root_ca.crt
✔ Root private key: /root/.step/secrets/root_ca_key
✔ Root fingerprint: fb31925c37a688d0821420eb25e5f1e6c03ca0c7d51e48516b14bdc13ff5ccdd
✔ Intermediate certificate: /root/.step/certs/intermediate_ca.crt
✔ Intermediate private key: /root/.step/secrets/intermediate_ca_key
✔ SSH user public key: /root/.step/certs/ssh_user_ca_key.pub
✔ SSH user private key: /root/.step/secrets/ssh_user_ca_key
✔ SSH host public key: /root/.step/certs/ssh_host_ca_key.pub
✔ SSH host private key: /root/.step/secrets/ssh_host_ca_key
✔ Database folder: /root/.step/db
✔ Templates folder: /root/.step/templates
✔ Default configuration: /root/.step/config/defaults.json
✔ Certificate Authority configuration: /root/.step/config/ca.json

Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

FEEDBACK 😍 🍻
  The step utility is not instrumented for usage statistics. It does not phone
  home. But your feedback is extremely valuable. Any information you can provide
  regarding how you’re using `step` helps. Please send us a sentence or two,
  good or bad at feedback@smallstep.com or join GitHub Discussions
  https://github.com/smallstep/certificates/discussions and our Discord
  https://u.step.sm/discord.
[root@step step]#

#run your ssh-ca
[root@step step]# step-ca $(step path)/config/ca.json
badger 2024/09/06 10:50:53 INFO: All 1 tables opened in 1ms
badger 2024/09/06 10:50:53 INFO: Replaying file id: 0 at offset: 7306
badger 2024/09/06 10:50:53 INFO: Replay took: 9.758µs
Please enter the password to decrypt /root/.step/secrets/intermediate_ca_key:
Please enter the password to decrypt /root/.step/secrets/ssh_host_ca_key:
Please enter the password to decrypt /root/.step/secrets/ssh_user_ca_key:
2024/09/06 10:51:00 Building new tls configuration using step-ca x509 Signer Interface
2024/09/06 10:51:00 Starting Smallstep CA/0.27.2 (linux/amd64)
2024/09/06 10:51:00 Documentation: https://u.step.sm/docs/ca
2024/09/06 10:51:00 Community Discord: https://u.step.sm/discord
2024/09/06 10:51:00 Config file: /root/.step/config/ca.json
2024/09/06 10:51:00 The primary server URL is https://step:443
2024/09/06 10:51:00 Root certificates are available at https://step:443/roots.pem
2024/09/06 10:51:00 Additional configured hostnames: 10.50.100.110
2024/09/06 10:51:00 X.509 Root Fingerprint: fb31925c37a688d0821420eb25e5f1e6c03ca0c7d51e48516b14bdc13ff5ccdd
2024/09/06 10:51:00 SSH Host CA Key: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKliAvQoL0eRNGMXMUFSSOBzo8fjzsnb1ztakctwFJUnsgzSCCWhXDky5B59CQcw/m8fb/0DDWv0Vyw7YYRkLJM=
2024/09/06 10:51:00 SSH User CA Key: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWRNNWJj9uhoM5PZ4rkQP0yzeW9F2+73UqCmQSAdDcukUzHmMlVet5yDpbOqfkjAVwokW68cS9OfzEXetd41y0=
2024/09/06 10:51:00 Serving HTTPS on 10.50.100.110:443 ...
  • we need later the shown fingerprint, this is a good time to copy the string (fb319….cdd).
  • Setup clientstep1 as ssh-daemon who only accept ssh-certificates of our new ssh ca:
wget https://dl.smallstep.com/cli/docs-cli-install/latest/step-cli_amd64.rpm
rpm -i step-cli_amd64.rpm
[root@clientstep1 ugu5ma]# step version
Smallstep CLI/0.27.2 (linux/amd64)
Release Date: 2024-07-18T18:15:09Z
[root@clientstep1 ugu5ma]#

*** To configure step to access your CA from a new host, run step ca bootstrap --ca-url [CA URL] --fingerprint [CA fingerprint] :
[root@clientstep1 ugu5ma]# step ca bootstrap --ca-url https://10.50.100.110:443 --fingerprint fb31925c37a688d0821420eb25e5f1e6c03ca0c7d51e48516b14bdc13ff5ccdd
The root certificate has been saved in /root/.step/certs/root_ca.crt.
The authority configuration has been saved in /root/.step/config/defaults.json.
#### trust your SSH user CA 
[root@clientstep1 ugu5ma]# step ssh config --roots > /path/to/ssh_user_key.pub
bash: /path/to/ssh_user_key.pub: No such file or directory
[root@clientstep1 ugu5ma]# step certificate install $(step path)/certs/root_ca.crt
Certificate /root/.step/certs/root_ca.crt has been installed.
X.509v3 Root CA Certificate (ECDSA P-256) [Serial: 1777...5197]
  Subject:     step Root CA
  Issuer:      step Root CA
  Valid from:  2024-09-06T08:40:14Z
          to:  2034-09-04T08:40:14Z
#### let's sign a host-certificate for:
##### principal1      = clientstep1
##### principal2      = clientstep1.fritz.box
##### certificate-id  = ubuarmlts 
[root@clientstep1 ugu5ma]# sudo -E step ssh certificate ubuarmlts /etc/ssh/ssh_host_ecdsa_key.pub --host --sign --principal clientstep1  --principal clientstep1.fritz.box
Use the arrow keys to navigate: ↓ ↑ → ←
What provisioner key do you want to use?
  ▸ ugu5ma@step (JWK) [kid: DFYsQuqHCCpjx2uXeDYiZEI0V8aH2tAcK54qGPyHbzA]
    sshpop (SSHPOP)
Please enter the password to decrypt the provisioner key:
✔ CA: https://10.50.100.110:443
✔ Certificate: /etc/ssh/ssh_host_ecdsa_key-cert.pub
#### 
#### run the following command and add the following to your SSHD configuration (vi /etc/ssh/sshd_config)
####
[root@clientstep1 ssh]# step ssh config --roots > /etc/ssh/ssh_user_key.pub
#### add the following to SSHD config(vi /etc/ssh/sshd_config)
# This is our host private key and certificate:
HostKey /etc/ssh/ssh_host_ecdsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
*** Configure SSH clients to trust your host CA
** To view the host key, run: step ssh config --host --roots
  • Setup clientstep2 as cert-based ssh-client 
#
#>>>bootstrap the host as shown on stepclient1<<<
#
#Let's create an SSH user certificate for user ugu5ma
[ugu5ma@clientstep2 ~]$ cd /home/ugu5ma/.ssh
[ugu5ma@clientstep2 ~]$ step ssh certificate ugu5ma@step id_ecdsa
Use the arrow keys to navigate: ↓ ↑ → ←
What provisioner key do you want to use?
  ▸ ugu5ma@step (JWK) [kid: DFYsQuqHCCpjx2uXeDYiZEI0V8aH2tAcK54qGPyHbzA]
    sshpop (SSHPOP)
✔ Provisioner: ugu5ma@step (JWK) [kid: DFYsQuqHCCpjx2uXeDYiZEI0V8aH2tAcK54qGPyHbzA]
Please enter the password to decrypt the provisioner key:
✔ CA: https://10.50.100.110:443
Please enter the password to encrypt the private key: █
✔ Private Key: id_ecdsa
✔ Public Key: id_ecdsa.pub
✔ Certificate: id_ecdsa-cert.pub

### Trust your ssh-CA, run command:
[ugu5ma@clientstep2 .ssh]$ step ssh config --host --roots
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKliAvQoL0eRNGMXMUFSSOBzo8fjzsnb1ztakctwFJUnsgzSCCWhXDky5B59CQcw/m8fb/0DDWv0Vyw7YYRkLJM=

### add the output to .ssh/known_hosts (vi /home/ugu5ma/.ssh/known_hosts)
@cert-authority * ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKliAvQoL0eRNGMXMUFSSOBzo8fjzsnb1ztakctwFJUnsgzSCCWhXDky5B59CQcw/m8fb/0DDWv0Vyw7YYRkLJM=

### verify

[ugu5ma@clientstep2 .ssh]$ cat /home/ugu5ma/.ssh/known_hosts
@cert-authority * ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKliAvQoL0eRNGMXMUFSSOBzo8fjzsnb1ztakctwFJUnsgzSCCWhXDky5B59CQcw/m8fb/0DDWv0Vyw7YYRkLJM=
  • next we try to connect from clientstep2 to clientstep1:
### let's ssh to clientstep1 and verify the communication:

[ugu5ma@clientstep2 .ssh]$ ssh ugu5ma@clientstep1 -vv
OpenSSH_8.7p1, OpenSSL 3.0.7 1 Nov 2022
#
# uninteresting output deleted...
#
debug2: resolving "clientstep1.fritz.box" port 22
debug1: Connecting to clientstep1.fritz.box [10.50.100.111] port 22.
debug1: Connection established.
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to clientstep1.fritz.box:22 as 'ugu5ma'
debug1: Server host certificate: ecdsa-sha2-nistp256-cert-v01@openssh.com SHA256:5OW2e/VviIjRptZ9L5exCm6xW3jwUI2BEM9Zx8MAb0A, serial 5287235044766806569 ID "ubuarmlts" CA ecdsa-sha2-nistp256 SHA256:jC3GukYmJNtdbqDF0J17DiFU98TW2/TlFNyQ2XG58PE valid from 2024-09-06T11:11:00 to 2024-10-06T11:12:00
debug2: Server host certificate hostname: clientstep1
debug2: Server host certificate hostname: clientstep1.fritz.box
debug1: Host 'clientstep1.fritz.box' is known and matches the ECDSA-CERT host certificate.
debug1: Found CA key in /home/ugu5ma/.ssh/known_hosts:1
debug1: Will attempt key: /home/ugu5ma/.ssh/id_ecdsa ECDSA-CERT SHA256:wBhbhAOlVpUXCKsuuAvN8+LV6ZLZuMpsv7GPySw3874 agent
debug2: pubkey_prepare: done
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Server accepts key: /home/ugu5ma/.ssh/id_ecdsa ECDSA-CERT SHA256:wBhbhAOlVpUXCKsuuAvN8+LV6ZLZuMpsv7GPySw3874 agent
debug2: sign_and_send_pubkey: using private key "/home/ugu5ma/.ssh/id_ecdsa" from agent for certificate
Authenticated to clientstep1 ([10.50.100.111]:22) using "publickey".
[ugu5ma@clientstep1 ~]$
  • Looks good! No TOFU as well 🙂
  • Lets try as next an ssh attempt from rogueclient without certificates:
ugu5ma@rogueclient:ssh ugu5ma@10.50.100.111 -vv
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host certificate: ecdsa-sha2-nistp256-cert-v01@openssh.com SHA256:5OW2e/VviIjRptZ9L5exCm6xW3jwUI2BEM9Zx8MAb0A, serial 5287235044766806569 ID "ubuarmlts" CA ecdsa-sha2-nistp256 SHA256:jC3GukYmJNtdbqDF0J17DiFU98TW2/TlFNyQ2XG58PE valid from 2024-09-06T11:11:00 to 2024-10-06T11:12:00
debug2: Server host certificate hostname: clientstep1
debug2: Server host certificate hostname: clientstep1.fritz.box
debug1: Next authentication method: publickey
debug1: Trying private key: /home/ugu5ma/.ssh/id_rsa
debug1: Trying private key: /home/ugu5ma/.ssh/id_ecdsa
debug1: Trying private key: /home/ugu5ma/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/ugu5ma/.ssh/id_ed25519
debug1: Trying private key: /home/ugu5ma/.ssh/id_ed25519_sk
debug1: Trying private key: /home/ugu5ma/.ssh/id_xmss
debug1: Trying private key: /home/ugu5ma/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
ugu5ma@10.50.100.111: Permission denied (publickey).

any ssh-attempt from rogueclient is denied because the client is not equipped with a valid ssh-certificate (best practice setting of “PasswordAuthentication no” and “PubkeyAuthentication yes“in sshd_config on clientstep1 must be enabled to avoid circumventing the security framework. ).

Is the shown setup suitable for productive use ? NO!
Smallstep has a nice overview what topics needs to be considered before going live.


This page is just for setting up a small dev-lab to get handy with ssh-certificates.
As always, take backups and redundant console-access before changing anything 🙂


Troubleshooting-commands

###check ssh host-certificate on clientstep1
[root@clientstep1 ssh]# cat /etc/ssh/ssh_host_ecdsa_key-cert.pub | step ssh inspect
-:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com host certificate
        Public key: ECDSA-CERT SHA256:5OW2e/VviIjRptZ9L5exCm6xW3jwUI2BEM9Zx8MAb0A
        Signing CA: ECDSA SHA256:jC3GukYmJNtdbqDF0J17DiFU98TW2/TlFNyQ2XG58PE (using ecdsa-sha2-nistp256)
        Key ID: "ubuarmlts"
        Serial: 5287235044766806569
        Valid: from 2024-09-06T11:11:00 to 2024-10-06T11:12:00
        Principals:
                clientstep1
                clientstep1.fritz.box
        Critical Options: (none)
        Extensions: (none)
        Signature:
                00:00:00:20:33:ac:f7:8e:4d:4c:60:26:5f:5e:83:18:
                a0:ef:25:3c:4f:49:3d:56:1d:c8:96:b0:3e:63:ee:1d:
                c5:27:8b:5e:00:00:00:21:00:d5:07:b2:13:ea:07:2d:
                57:4c:95:15:aa:5f:0c:95:71:f5:ec:11:86:a7:d9:f3:
                95:b3:12:d6:2a:15:39:03:07
[root@clientstep1 ssh]#

###check ssh user-certificate on clientstep2
[ugu5ma@clientstep2 .ssh]$ cat /home/ugu5ma/.ssh/id_ecdsa-cert.pub | step ssh inspect
-:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:wBhbhAOlVpUXCKsuuAvN8+LV6ZLZuMpsv7GPySw3874
        Signing CA: ECDSA SHA256:H+rgQ6gIM10MCNti1+tJs7H3nUCzrPQ3P5nCH2BqTJ8 (using ecdsa-sha2-nistp256)
        Key ID: "ugu5ma@step"
        Serial: 4249190036314894073
        Valid: from 2024-09-06T11:33:43 to 2024-09-07T03:34:43
        Principals:
                ugu5ma
                ugu5ma@step
        Critical Options: (none)
        Extensions:
                permit-port-forwarding
                permit-pty
                permit-user-rc
                permit-X11-forwarding
                permit-agent-forwarding
        Signature:
                00:00:00:21:00:99:07:e1:37:9f:2c:99:28:e1:b3:65:
                e9:8b:a6:4f:f0:d1:93:66:1f:f1:2d:fe:fa:21:ad:55:
                15:8b:54:69:fd:00:00:00:20:25:69:6b:00:99:e1:6a:
                6d:b4:04:af:02:9b:18:ae:0a:85:84:b8:fa:c1:f5:1c:
                d8:82:cc:f1:e9:07:81:90:38
[ugu5ma@clientstep2 .ssh]$

You see the lifetime of the user-certificate ? It is valid for 16 hours.
You can tweak the cert-lifetime during the creation with the parameter not-before and not-after, example:

# This certificate will be valid starting 10 minutes from now, until 10 days from now:
[ugu5ma@clientstep2 .ssh]$ step ssh certificate ugu5ma@step id_ecdsa --not-before=10m --not-after=240h
###check 
[ugu5ma@clientstep2 .ssh]$ cat /home/ugu5ma/.ssh/id_ecdsa-cert.pub | step ssh inspect
-:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:GOvbmiBstPbiiov0kNF4fApyjeSUhdnvm2xS5QolBFg
        Signing CA: ECDSA SHA256:H+rgQ6gIM10MCNti1+tJs7H3nUCzrPQ3P5nCH2BqTJ8 (using ecdsa-sha2-nistp256)
        Key ID: "ugu5ma@step"
        Serial: 1320427072549296678
        Valid: from 2024-09-06T14:20:58 to 2024-09-16T14:10:58
        Principals:
                ugu5ma
                ugu5ma@step
        Critical Options: (none)

The maxUserSSHCertDuration is per default set to 24hours. If you want to extent the User-cert lifetime you have to adjust this parameter (vi $(step path)/config/ca.json)

"authority": {
		"provisioners": [
			{
				"type": "JWK",
				"name": "ugu5ma@step",
				"key": {
					"use": "sig",
					"kty": "EC",
					"kid": "DFYsQuqHCCpjx2uXeDYiZEI0V8aH2tAcK54qGPyHbzA",
					"crv": "P-256",
					"alg": "ES256",
					"x": "2rF1jfj1sdsSD2yCNLZgDzixPCGVKPP40TJHVHtlpdc",
					"y": "_b5YAseg2MzHHOW1it3l0ueJxm0Wm7j-9h5JWk9ROg8"
				},
				"encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjYwMDAwMCwicDJzIjoialBOcE53ellPM3RsODcwdUVRMjBMdyJ9.ss7QL4e38LDTsgyjrVLH6lNrcPlNBgCuFfQmx2ei_vkN_sMge-stQg.K5alPL_-AOKnyU2i.WN-58v-uDkeAw7kKpZVEFpScBq3eaGnMn6D6E3ARhI6Xy1w3EQgfz6oCbCHd1DidwSmy1I7bhX7hquvjVkznZuJZYb4FcF8ZJjcPDuh6-_DkHDUR8QBy3qbvkPJNKl1sO-TabgxW3og_L1PIbyeH66f5bPmTfICtJ5RCAevGNma6-vmJGga9hkd6Fmyvr2TToMqKx5pHYHCgUuNwpnfpk7PzyKAPp9Gx9VtykuJ3ikQVil9nMZmNPdy2CY9ZdOyb64NmIwolNmgl8vrdRSXu-HJxzVptttf5G6vn6MvUJnjXBJbf0GabNEsGqTBjm-qA2vWcfg0AlXYwiUDsb3o.dMb9vHds1gxo-_K8rLhkFw",
				"claims": {
					"enableSSHCA": true,
                                        "minTLSCertDuration": "5m",
                                        "maxTLSCertDuration": "480h",
                                        "defaultTLSCertDuration": "24h",
					"minHostSSHCertDuration": "5m",
                                        "maxHostSSHCertDuration": "1680h",
                                        "defaultHostSSHCertDuration": "720h",
                                        "minUserSSHCertDuration": "5m",
                                        "maxUserSSHCertDuration": "480h",
                                        "defaultUserSSHCertDuration": "16h"
}

Have fun!

Secure your SSH communication with certificates

How about securing your SSH-Server to only support login-attempts including a valid signed certificate from a trusted CA ?


This sounds pretty cool, but there are a couple of pitfalls which should be outlined first:

  • OpenSSH supports cert-based authentication since version 5.4 (in 2010)
  • OpenSSH does not support x.509-certificates !
  • OpenSSH has implemented its own certificate format

OpenSSH’s decision to use its own proprietary SSH certificates instead of X.509 certificates, as outlined in RFC 6187 (no draft, proposed standard!), is rooted in several practical and technical reasons. Let’s dive into the details:

Simplicity and Efficiency

  1. Simplicity: OpenSSH certificates are designed to be simple and efficient. They contain only the necessary information for SSH authentication, such as the public key, name, expiration date, and associated permissions. This simplicity makes them easier to implement and manage compared to the more complex X.509 certificates, which include a broader range of attributes and extensions.
  2. Efficiency: The lightweight nature of OpenSSH certificates means they are faster to process and verify. This efficiency is particularly important in environments with a large number of SSH connections, where performance can be a critical factor.

Security and Flexibility

  1. Security: OpenSSH certificates offer several security advantages. They are digitally signed, which means they cannot be altered without invalidating the signature. Additionally, they support short-lived certificates, which automatically expire after a set period, reducing the risk of unauthorized access if a certificate is compromised.
  2. Flexibility: OpenSSH certificates provide flexibility in terms of configuration and usage. They allow for custom validity periods, source restrictions, command restrictions, and option enforcement. This level of customization is not as easily achievable with X.509 certificates, which are designed for a broader range of applications beyond SSH.

Management and Usability

  1. Centralized Management: OpenSSH certificates simplify the management of SSH access. Instead of managing individual public keys for each user and server, administrators can use a single Certificate Authority (CA) to issue and revoke certificates. This centralized approach makes it easier to onboard and offboard users, as well as manage access permissions.
  2. Usability: The proprietary SSH certificate format is tailored specifically for SSH use cases, making it more user-friendly for administrators and developers who work primarily with SSH. The familiarity and ease of use of OpenSSH certificates can lead to better adoption and fewer implementation issues.

Is there any way to still use X.509-certificates with SSH ? Sure!
There are various products on the market available supporting X.509-based certificates like:
–PKIX-SSH secure shell with X.509 v3 certificate support (OpenSSH patch for X.509-support)
–Tectia® SSH Client/Server
–wolfSSL
-and so on and so forth. This is no complete list 🙂

Keep in mind that big players like RedHat rely on the proprietary certificate-solution of OpenSSH

My (personal) Summary:
while X.509 certificates are widely used and supported for various applications, OpenSSH’s proprietary certificates offer a more streamlined, secure, and manageable solution for SSH authentication. The decision to use a proprietary format is driven by the need for simplicity, efficiency, security, flexibility, and ease of management. A patch of the OpenSSH-libraries is not needed.

When you lock down your SSH-daemon to only allow logins with valid certificates of your SSH-CA you start creating an additional security-layer for your SSH-Service.
Just think of securing the SSH-service on an internet-facing (Bastion-)hosts:
Without ssh-certs you need tools like Crowdsec, SshGuard, Fail2ban to e.g. jail hacking attempts, but you get still a lot of noise in your logs.
Fail2ban for example creates time-based filters based on the Source-IP of the hacking-attempt:

dynamic FW-entries:
To                         Action      From
--                         ------      ----
Anywhere                   REJECT      1.234.58.136               # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      51.161.153.48              # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      189.241.227.175            # by Fail2Ban after 3 attempts against sshd
Anywhere                   REJECT      193.32.162.79              # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.238             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.235             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.237             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      183.81.169.236             # by Fail2Ban after 2 attempts against sshd
Anywhere                   REJECT      1.234.58.142               # by Fail2Ban after 2 attempts against sshd

logs:
2024-08-28 07:48:00,596 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 07:48:00
2024-08-28 08:01:05,385 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:01:05
2024-08-28 08:04:25,692 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 08:04:25
2024-08-28 08:13:23,523 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 08:13:23
2024-08-28 08:16:29,521 fail2ban.actions        [773]: NOTICE  [apache-noscript] Unban 64.227.153.228
2024-08-28 08:20:49,352 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:20:49
2024-08-28 08:27:37,117 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.128.142.238
2024-08-28 08:27:38,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 112.163.28.218
2024-08-28 08:27:54,621 fail2ban.actions        [773]: NOTICE  [sshd] Unban 43.134.110.112
2024-08-28 08:28:22,790 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.97.177.217
2024-08-28 08:31:20,214 fail2ban.actions        [773]: NOTICE  [sshd] Unban 117.83.178.140
2024-08-28 08:36:06,460 fail2ban.actions        [773]: NOTICE  [sshd] Unban 207.172.160.36
2024-08-28 08:36:20,119 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 08:36:19
2024-08-28 08:36:31,386 fail2ban.filter         [773]: INFO    [apache-noscript] Found 167.172.208.130 - 2024-08-28 08:36:31
2024-08-28 08:38:00,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.73.131
2024-08-28 08:38:15,008 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.221.80.92
2024-08-28 08:38:57,178 fail2ban.actions        [773]: NOTICE  [sshd] Unban 177.53.215.134
2024-08-28 08:39:12,549 fail2ban.actions        [773]: NOTICE  [sshd] Unban 115.73.209.212
2024-08-28 08:39:25,317 fail2ban.actions        [773]: NOTICE  [sshd] Unban 78.153.149.132
2024-08-28 08:39:54,106 fail2ban.actions        [773]: NOTICE  [sshd] Unban 103.140.239.254
2024-08-28 08:40:03,800 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 08:40:03
2024-08-28 08:40:29,317 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 08:40:29
2024-08-28 08:41:04,291 fail2ban.actions        [773]: NOTICE  [sshd] Unban 180.131.108.240
2024-08-28 08:43:08,475 fail2ban.actions        [773]: NOTICE  [sshd] Unban 36.26.76.62
2024-08-28 08:44:26,650 fail2ban.actions        [773]: NOTICE  [sshd] Unban 27.254.207.91
2024-08-28 08:59:30,066 fail2ban.filter         [773]: INFO    [sshd] Found 146.59.228.24 - 2024-08-28 08:59:30
2024-08-28 08:59:52,008 fail2ban.filter         [773]: INFO    [sshd] Found 1.218.138.131 - 2024-08-28 08:59:51
2024-08-28 09:01:00,569 fail2ban.filter         [773]: INFO    [sshd] Found 47.245.28.86 - 2024-08-28 09:01:00
2024-08-28 09:11:10,660 fail2ban.filter         [773]: INFO    [sshd] Found 140.143.171.137 - 2024-08-28 09:11:10
2024-08-28 09:15:49,726 fail2ban.filter         [773]: INFO    [sshd] Found 103.200.20.12 - 2024-08-28 09:15:49
2024-08-28 09:17:24,908 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 09:17:24
2024-08-28 09:26:50,534 fail2ban.filter         [773]: INFO    [sshd] Found 2a03:b0c0:2:d0::89:2001 - 2024-08-28 09:26:50
2024-08-28 09:27:17,395 fail2ban.actions        [773]: NOTICE  [sshd] Unban 14.40.8.125
2024-08-28 09:27:46,027 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:8:3b79:: - 2024-08-28 09:27:46
2024-08-28 09:28:14,571 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:28:14
2024-08-28 09:30:05,778 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:30:05
2024-08-28 09:36:10,170 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:10
2024-08-28 09:36:33,524 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:36:33
2024-08-28 09:36:57,262 fail2ban.filter         [773]: INFO    [sshd] Found 189.241.227.175 - 2024-08-28 09:36:57
2024-08-28 09:36:57,711 fail2ban.actions        [773]: NOTICE  [sshd] Ban 189.241.227.175
2024-08-28 09:37:34,771 fail2ban.filter         [773]: INFO    [sshd] Found 51.161.153.48 - 2024-08-28 09:37:34
2024-08-28 09:37:35,064 fail2ban.actions        [773]: NOTICE  [sshd] Ban 51.161.153.48
2024-08-28 09:42:44,848 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:42:44
2024-08-28 09:44:11,137 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:44:10
2024-08-28 09:45:23,912 fail2ban.filter         [773]: INFO    [sshd] Found 1.234.58.136 - 2024-08-28 09:45:23
2024-08-28 09:45:23,929 fail2ban.actions        [773]: NOTICE  [sshd] Ban 1.234.58.136
2024-08-28 09:57:28,398 fail2ban.filter         [773]: INFO    [sshd] Found 116.122.157.203 - 2024-08-28 09:57:28
2024-08-28 10:10:03,236 fail2ban.filter         [773]: INFO    [sshd] Found 2001:41d0:304:200::6a05 - 2024-08-28 10:10:03
2024-08-28 10:12:18,780 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.254 - 2024-08-28 10:12:18
2024-08-28 10:17:56,698 fail2ban.filter         [773]: INFO    [sshd] Found 202.190.50.129 - 2024-08-28 10:17:56
2024-08-28 10:28:07,991 fail2ban.filter         [773]: INFO    [sshd] Found 194.169.175.37 - 2024-08-28 10:28:07
2024-08-28 10:44:49,312 fail2ban.filter         [773]: INFO    [sshd] Found 51.68.143.159 - 2024-08-28 10:44:48
2024-08-28 10:48:23,088 fail2ban.filter         [773]: INFO    [sshd] Found 85.209.11.27 - 2024-08-28 10:48:23


Status for the jail: sshd
|- Filter
|  |- Currently failed:	4
|  |- Total failed:	2595
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	8
   |- Total banned:	611
   `- Banned IP list:	83.81.169.235 83.81.169.236 83.81.169.237 183.81.169.238 193.32.162.79 1.234.58.136 1.234.58.137 1.234.58.142


With cert-based ssh-authentication these tools are no more needed as a valid client-ssh-cert is mandatory.

I will create in the next days a howto based on the smallstep-solution to show the pros (and cons) for cert-based ssh from an DevOps perspective.