Modify postfix for webhook plugin for Jellyfin

bgstack15 — Mon, 04 Jul 2022 12:48:40 GMT

With the recent Gmail change that requires oauth2 for sending authenticated gmail (covered in Postfix use oauth2 for gmail), my jellyfin Webhook plugin that includes an smtp option has finally stopped working.

First of all, I had to ensure that I had network connectivity to my smtp server which is available over my wireguard connection.

nc server2.ipa.internal.com 25
Ncat: Connection refused.

So I had to modify the nftables rules on server2. That took me a while, but I finally got it. For a real-time modification, I used this command.

sudo nft add rule 'inet filter' input position 4 iif wg0 accept

This rule means "for input interface wg0 [wireguard], accept all packets." And insert this rule in a certain position, and not just at the end (so after the infamous "DROP ALL" of a well-behaved firewall.

And my full ruleset is now in /etc/nftables.conf.

flush ruleset

table inet filter {
   chain input {
      type filter hook input priority 0;
      # accept any localhost traffic
      iif lo accept
      iif wg0 accept comment "trust all wireguard traffic"
      # accept traffic that originated from this system

      # accept traffic originated from us
      ct state established,related accept

      # this {} array is comma-separated
      tcp dport { 22 } ct state new accept

      # count and drop any other traffic
      counter drop
   }
   chain forward {
      type filter hook forward priority 0;
   }
   chain output {
      type filter hook output priority 0;
   }
}

So finally my netcat worked.

$ nc server2.ipa.internal.com 25
220 server2.ipa.internal.com ESMTP Postfix (Debian/GNU)

So when I trigger a notification in Jellyfin, I get this error.

Jun 26 18:27:50 server2 postfix/smtpd[14319]: connect from server1.remote.internal.com[10.198.0.14] Jun 26 18:27:50 server2 postfix/smtpd[14319]: warning: TLS library problem: error:0A000126:SSL routines::unexpected eof while reading:../ssl/record/rec_layer_s3.c:308: Jun 26 18:27:50 server2 postfix/smtpd[14319]: lost connection after STARTTLS from server1.remote.internal.com[10.198.0.14] Jun 26 18:27:50 server2 postfix/smtpd[14319]: disconnect from server1.remote.internal.com[10.198.0.14] ehlo=1 starttls=1 commands=2

Researching on the Internet for "jellyfin webhook smtp starttls" led to information mostly about disabling starttls. I didn't even realize I had it enabled. So I made some changes to my postfix to disable the silly snakeoil TLS certificate.

And then I logged in again, and got this message in my postfix logs! So this is progress.

Jun 26 18:34:49 server2 postfix/smtpd[15802]: NOQUEUE: reject: RCPT from server1.remote.internal.com[10.198.0.14]: 454 4.7.1 <example@gmail.com>: Relay access denied; from=<example@gmail.com> to=<example@gmail.com> proto=ESMTP helo=<[192.168.58.18]>
Jun 26 18:34:49 server2 postfix/smtpd[15802]: lost connection after RSET from server1.remote.internal.com[10.198.0.14]
Jun 26 18:34:49 server2 postfix/smtpd[15802]: disconnect from server1.remote.internal.com[10.198.0.14] ehlo=1 mail=1 rcpt=0/1 rset=1 commands=3/4

After all the changes, my postfix main.cf includes at least these lines:

# Important to comment these out!
#smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
#smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
#smtpd_tls_security_level=may
smtpd_use_tls=no
# This already existed, but...
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
# I added my wireguard subnet here.
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 10.198.0.0/24

And now I can receive notifications when my users visit my Jellyfin server.

And just for completeness's sake, here is my smtp notification information.

Add smtp destination.
Name: smtp2
Webhook Url: (not relevant) https://www.example.com/internal/jellyfin-webhook2.html
Items: playback start, playback stop, session start
User filter: (all users)
Item time: (all)
Send all properties: no
Template:
<pre>Username: {{Username}}
Action: {{NotificationType}}
Timestamp: {{UtcTimestamp}}
Title:: {{Name}}
{{#if_exist SeriesName}}
Series: {{SeriesName}}
Season: {{SeasonNumber00}}
Episode: {{EpisodeNumber00}}
{{/if_exist}}
DeviceName: {{DeviceName}}
ClientName: {{ClientName}}
PlaybackPosition: {{PlaybackPosition}}
</pre>
END TEMPLATE CONTENTS
Sender: example@gmail.com
Receiver: example@gmail.com
smtp server address: server2.ipa.internal.com
smtp port: 25
Use credentials: no
Use ssl: no
Is html body: yes
Subject template: Jellyfin activity for {{Username}}

Update 2022-07-12:

I have since learned that the nftables.conf contents should NOT have double-quotes around the wg0 interface name. The output of nft list table 'inet filter' shows double-quotes, but these do not work when placed in the rules file.

Show all TLS ports and their cert info

bgstack15 — Mon, 14 Mar 2022 12:59:31 GMT

I wanted to conduct an audit of what TLS certificates are in use on my system. This command should be run from the system to be scanned, but the connection is made to the main IP address and not loopback. So for host server1, run this command:

{ for word in $( sudo ss -tlnu | awk '{print $5}' | awk -F ':' '!x[$2]++{print $2}' | sort -n ) ; do timeout 3s sslscanner server1:${word} | sed -r -e "s/^/${word}: /;" ; done ; } 2>&1 | grep -vE '^Terminated$'

Observe that this command does depend on the sslscanner script.

The sample output is:

443: subject= /O=IPA.EXAMPLE.COM/CN=server1.ipa.internal.com
443: issuer= /O=IPA.EXAMPLE.COM/CN=Certificate Authority
443: notBefore=May  7 19:03:38 2021 GMT
443: notAfter=May  8 19:03:38 2023 GMT
443: san=www.example.com
443: san=server1.ipa.internal.com
443: san=www.ipa.internal.com
443: san=www.internal.com
443: 
500: subject= /CN=www.example.com
500: issuer= /C=US/O=Let's Encrypt/CN=R3
500: notBefore=Feb 26 23:38:29 2022 GMT
500: notAfter=May 27 23:38:28 2022 GMT
500: san=www.example.com
500: 
500: subject= /C=US/O=Let's Encrypt/CN=R3
500: issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
500: notBefore=Sep  4 00:00:00 2020 GMT
500: notAfter=Sep 15 16:00:00 2025 GMT
500: 
500: subject= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
500: issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3
500: notBefore=Jan 20 19:14:03 2021 GMT
500: notAfter=Sep 30 18:14:03 2024 GMT
500:

This oneliner makes it simple to see which certificates are in use, on what port.

Knowledge Base (Posts about tls)

Modify postfix for webhook plugin for Jellyfin

Update 2022-07-12:

Show all TLS ports and their cert info