Knowledge Base (Posts about gmail)

Postfix use oauth2 for gmail

bgstack15 — Thu, 10 Mar 2022 14:22:50 GMT

I've previously written about how to send authenticated gmail from cli with mailx, and a slightly more generic send authenticated gmail from command line. But with a recent change to Google's behaviors, I have to use some baloney scheme designed to make system admin's lives difficult just to send email.

Overview

Gmail requires the use of Oauth2 (aka 'xoauth2', or maybe just the library is known as that) to send authenticated mail from custom applications. This document describes how to set up postfix to relay to gmail as an authenticated user under the new scheme. My goal includes only sending messages, not receiving. Everything outbound just comes from my one account, bgstack15@gmail.com. It is possible the references include guidance for allowing different accounts for different users.

The test environment is d2-03a, and production is server2.remote.example.com.

Dependencies

On the system where postfix needs to run:

Devuan Ceres

The libpython2-stdlib is important for libimap.

sudo apt-get install postfix python2-minimal libpython2-stdlib libsasl2-module-xoauth

Onetime setup

Google side

I had to use Chromium to log into cloud.console.google.com. I found out later that LibreWolf does work. Here, set up "Oauth consent screen" and add my email address to the list of testers. Then, set up an oauth 2.0 client id ref 1 which provides the client id and secret.

On a Linux system where you can clone the gmail-oauth2-tools utilities, run the one-time command to get the access token and refresh token.

python2 gmail-oauth2-tools/python/oauth2.py --user=bgstack15@gmail.com --generate_oauth2_token --client_id=2748037O9251-ssj18r8tli6krklewtus3m2n3m7lvtiw.apps.googleusercontent.com --client_secret=GODSNX-m2MnUnpEac3tQU-1nm4VN54nop3m

It will direct you to a link you need to open in the browser to sign in and allow this application to control email. Paste the response back into the python2 program and it will generate an access token and refresh token.

Refresh Token: 1//01E-dJkGQzpa3CgYIARAAGAESNwF-L9Irl1pOeMY42_5uBGzVveXggTfg1Car290BgVHdEGspZxWpSheTHWXPySu-9uXvim8mFWg
Access Token: ya29.A0ARrdaM-PO3kNGo28gmKSGOuwkglampwoij3482GM26iTLiw4xMGNE3wE1Te54MvBo_RgmlIBEYd4qEMY522kTm4xnoIozpW5nL43nGmLap3kMfmsZ_sUt4Qenk_JDFMVGIxsmwXWJxObeR_-LSJ61IN4Bi4r
Access Token Expiration Seconds: 3599

Save the refresh token contents to /etc/postfix/refresh-token.

Postfix

Reference 3 or 4 include the readme that describes the process for configuring postfix.

A custom plugin is necessary, and is buildable from the above links. Link 4 includes a dpkg recipe. The plugin generates /usr/lib/x86_64-linux-gnu/sasl2/libxoauth2.so.0.0.0 or similar. The package libsasl2-module-xoauth2 is in my internal repo but can be rebuilt from link 4.

Postfix file main.cf needs quite a few entries, including but not limited to:

# everything normal it has, so these go at the bottom:
# gmail struggles with ipv6, or my net does or something.
inet_protocols = ipv4
# client
relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/saslpasswd
smtp_sasl_mechanism_filter = xoauth2
smtp_sasl_security_options =
smtp_tls_security_level = may
smtp_tls_policy_maps = hash:/etc/postfix/tls_policy

Establish file /etc/postfix/saslpasswd:

[smtp.gmail.com]:587    bgstack15@gmail.com:OAUTH2-TOKEN-CONTENTS-LONG-STRING

Then generate the /etc/postfix/saslpasswd.db with postmap:

# postmap /etc/postfix/saslpsswd

Establish file /etc/postfix/tls_policy:

[smtp.gmail.com]:587 encrypt

Generate its db file:

# postmap /etc/postfix/tls_policy

Generate in the ${sasl_plugin_dir}, the file nominally /etc/postfix/sasl/smtpd.conf.

log_level: DEBUG
sql_engine: sqlite3
sql_database: /etc/sasldb2.sqlite3
sql_select: SELECT props.value WHERE props.id = 2
xoauth2_scope: https://gmail.com/
auxprop_plugin: sql
mech_list: xoauth2

Install sqlite3 package if necessary, and establish sqlite3 database file /etc/sasl2.sqlite3 with the following:

# sqlite3 /etc/sasldb2.sqlite3
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE props (id INTEGER PRIMARY KEY, name VARCHAR, value VARCHAR);
INSERT INTO props VALUES(1,'userPassword','*');
INSERT INTO props VALUES(2,'oauth2BearerTokens','token');
COMMIT;

These values are string literals. Insert into the database an asterisk, as well as the word token.

Reload postfix.

sudo service postfix reload

Send a test email.

mail -a 'From:B. Stack <bgstack15@example.com' -s 'Test with oauth2 part11' bgstack15@gmail.com <<EOF
hello from the command line
at 13:31
EOF

The From field pretty name is used by gmail, but the <email address> section is discarded because we are sending from the one authenticated gmail account.

Custom token rotation script

Oauth2 access tokens expire (google likes to use 59 minutes), so a cron entry can be used to use the refresh token to get a new access token. The following script does not support accepting a new refresh token, but I do not know if that would ever need to happen.

Copy oauth2.py from that gmail oauth tools project to /usr/local/bin/oauth2.py2.

Establish file /usr/local/bin/refresh-oauth2-token:

	`#!/usr/bin/env sh`
	`# File: /usr/local/bin/refresh-oauth2-token`
	`# Startdate: 2022-03-04 13:36`
	`# Purpose: gets new access token with the cached refresh token`
	`# Usage: in cron every 30 minutes, because the access token lasts for 59 minutes. Must be run as root.`
	`# Dependencies:`
	`# already-established refresh token in /etc/postfix/refresh-token`
	`# Documentation:`
	`# /mnt/public/Support/Programs/oauth2-for-gmail/README-oauth2-for-gmail.md`
	`. /etc/default/postfix-oauth2`
	`export PATH=/usr/bin:/usr/sbin:/bin:/sbin`
	`test -z "${REFRESH_FILE}" && REFRESH_FILE="/etc/postfix/refresh-token"`
	`test -z "${SASLPASSWD_FILE}" && SASLPASSWD_FILE="/etc/postfix/saslpasswd"`
	`test -z "${OAUTH2_SCRIPT}" && OAUTH2_SCRIPT="/usr/local/bin/oauth2.py2"`
	`test -z "${USERNAME}" && USERNAME="bgstack15@gmail.com"`
	`test -z "${CLIENT_ID}" && CLIENT_ID="2748037O9251-ssj18r8tli6krklewtus3m2n3m7lvtiw.apps.googleusercontent.com"`
	`test -z "${CLIENT_SECRET}" && CLIENT_SECRET="GODSNX-m2MnUnpEac3tQU-1nm4VN54nop3m"`
	`test -z "${SMTP_SERVER}" && SMTP_SERVER=smtp.gmail.com`
	`test -z "${SMTP_PORT}" && SMTP_PORT=587`
	`refresh_token="$( cat "${REFRESH_FILE}" )"`
	`result="$( python2 "${OAUTH2_SCRIPT}" \`
	`--client_id="${CLIENT_ID}" \`
	`--client_secret="${CLIENT_SECRET}" \`
	`--refresh_token="${refresh_token}" \`
	`)"`
	`access_token="$( echo "${result}" \| awk '/Access Token:/{print $NF}' )"`
	`# Generate new /etc/saslpasswd`
	`echo "[${SMTP_SERVER}]:${SMTP_PORT} ${USERNAME}:${access_token}" > "${SASLPASSWD_FILE}"`
	`postmap "${SASLPASSWD_FILE}"`
	`service postfix reload`

Establish file /etc/default/postfix-oauth2:

# dot-sourced by /usr/local/bin/refresh-oauth2-token
REFRESH_FILE="/etc/postfix/refresh-token"
SASLPASSWD_FILE="/etc/postfix/saslpasswd"
OAUTH2_SCRIPT="/usr/local/bin/oauth2.py2"
USERNAME="bgstack15@gmail.com"
CLIENT_ID="2748037O9251-ssj18r8tli6krklewtus3m2n3m7lvtiw.apps.googleusercontent.com"
CLIENT_SECRET="GODSNX-m2MnUnpEac3tQU-1nm4VN54nop3m"
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587

Establish cron entry in /etc/cron.d/50_rotate_oauth2_token_cron.

# File /etc/cron.d/50_rotate_oauth2_token_cron
# Documentation: /mnt/public/Support/Programs/oauth2-for-gmail/README-oauth2-for-gmail.md
20,50 *  *  *  *  root    /usr/local/bin/refresh-oauth2-token 1>/dev/null 2>&1

Summary of files that are part of this project

Modified
/etc/postfix/main.cf
New to this project
/etc/postfix/saslpasswd Will get updated by the cron entry
/etc/postfix/saslpasswd.db
/etc/postfix/tls_policy
/etc/postfix/tls_policy.db
/etc/postfix/refresh-token
/etc/postfix/sasl/smtpd.conf
/usr/lib file libxoauth2.so somewhere, hopefully from an rpm/dpkg of the cyrus-sasl-xoauth2 project
/usr/local/bin/oauth2.py2 from the gmail-oauth2-tools project.
/usr/local/bin/refresh-oauth2-token
/etc/default/postfix-oauth2
/etc/cron.d/50_rotate_oauth2_token_cron

References

https://console.cloud.google.com/apis/credentials?project=smtp1-343114&supportedpurview=project
https://github.com/google/gmail-oauth2-tools a. exact link for oauth2.py2
https://github.com/moriyoshi/cyrus-sasl-xoauth2
Reference 3 but on salsa for the dpkg recpipe

Send authenticated gmail from cli with mailx

bgstack15 — Sat, 05 Aug 2017 13:18:05 GMT

Overview

I've shown how to send authenticated gmail from the command line before. That uses msmtp which takes some configuration. This document shows how to use mailx itself to send authenticated gmail.

tl;dr

echo "this is the message" | mailx -s "Subject line here" \
-S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login \
-S smtp=smtp://smtp.gmail.com:587 -S from="bgstack15@gmail.com(B Stack)" \
-S smtp-auth-user="bgstack15@gmail.com" \
-S smtp-auth-password="${SMTPPASSWORD}" -S ssl-verify=ignore \
-S nss-config-dir=/etc/pki/nssdb/ destination@example.com

Explanation

You need a certificate chain somewhere. You could also try nss-config- dir=~/.mozilla/firefox/xxxxxxxx.default. If you use the whole command in the tl;dr section, you don't need any config file. Of course, be aware that any parameter passed on the command line is visible to any other program running, so passing in the password like seen above is risky. You can redirect standard in from a file if you wish, of course, or from a here- document. For a dedicated configuration, and better password security, consider adding in to your ~/.mailrc file:

set smtp-use-starttls
set nss-config-dir=/etc/pki/nss/
set ssl-verify=ignore
set smtp=smtp://smtp.gmail.com:587
set smtp-auth=login
set smtp-auth-user=bgstack15@gmail.com
set smtp-auth-password=QWERTYUIOP
set from="bgstack15@gmail.com(B Stack)"

And then just use:

mailx -s "Subject line" destination@example.com

References

Weblinks

Inspiration for entire contents https://www.systutorials.com/1411/sending-email-from-mailx-command-in-linux-using-gmails-smtp/

Send authenticated gmail from command line

bgstack15 — Mon, 03 Apr 2017 13:22:48 GMT

Overview

You can send basic email from the command line. That's been done a million times before, including by me! Did you know that you can send email from the command line, from your gmail account that you authenticated to? That's a big deal for some people. This document describes how to do that.

Sending gmail from command line

Install packages msmtp and mutt. yum install msmtp mutt Write the conf file and fill in the pertinent information. # write conf file tf=/etc/msmtprc touch "${tf}"; chmod 0600 "${tf}"; chown root:root "${tf}"; cat <<EOF >"${tf}" account default tls on tls_certcheck off auth on host smtp.gmail.com port 587 user bgstack15@gmail.com from bgstack15@gmail.com password plaintextpassword # account second tls on tls_certcheck off auth on host smtp.gmail.com port 587 user secondaccount@gmail.com from secondaccount@gmail.com password plaintextpassword EOF Now you can send an email with this command. echo -e "From: Pretty Name\r\nSubject: Example subject\r\nContent goes here." | msmtp --debug --from=default -t destination@example.com

Sending attachments

Configure mutt if you want to send attachments from the command line. cat <<EOF >> /etc/Muttrc.local set sendmail="/usr/bin/msmtp" set use_from=yes set realname="Pretty Name" set from=bgstack15@gmail.com set envelope_from=yes EOF Send an email in html format and with an attachment. subject="This is the subject Mime-Version: 1.0 Content-Type: text/html" mutt -a ~/attachment.txt -s "${subject}" -- destination@example.com << EOF <html><body><pre> This will be fixed width font. I find it useful for sending code fragments or log files. </pre></body></html> EOF

Authenticating to gmail from command line

The Arch Linux wiki provides an important reminder:

Tip: If using Gmail you'll need to either

Allow "Less Secure Apps" in Settings > Security. Make sure to sign out of your other Gmail accounts first because the security settings part of Google Accounts can not manage concurrent sessions of more than one account.

Enable two factor authentication and create an app password.

References

Send.sh from bgscripts package https://gitlab.com/bgstack15/bgscripts/-/blob/master/src/usr/bin/send
https://wiki.archlinux.org/index.php/Msmtp#Test_functionality