Debugging authentication problems

A common problem after installing courier-imap or sqwebmail is that you try to login and just get "Invalid username or password" as the response. This document shows how to use courier's debugging features to pinpoint the problem.

1. Turn on debugging

For courier-imap, you need to set one of the following values in /etc/courier/pop3d and/or /etc/courier/imapd:

DEBUG_LOGIN=1    # turn on authentication debugging
DEBUG_LOGIN=2    # turn on authentication debugging AND show passwords

and then restart the daemons. For sqwebmail, you can either put this in /etc/courier/authdaemonrc, or you can start sqwebmail like this:

# DEBUG_LOGIN=1 /usr/lib/courier/authlib/authdaemond start
# DEBUG_LOGIN=1 /usr/lib/courier/sqwebmail/sqwebmaild start

At this point, all debugging output goes to syslog at level 'debug', which is normally not shown. You will probably need to change your /etc/syslog.conf file to be able to see these messages. If you have an existing entry which says "mail.info" (which means facility 'mail', level 'info' or higher) then you can just change this to "mail.debug". Alternatively you can add a new entry like this:

*.debug			/var/log/debug

Don't forget to create this file, and to send a HUP signal to syslogd to make it re-read its configuration:

# touch /var/log/debug
# killall -1 syslogd

If you don't want to mess around with your syslog configuration, you can also get authdaemond to log to a file. Just change the last line of /usr/lib/courier/authlib/authdaemond as follows:

exec $DIR/$AUTHDAEMOND -stderr=/var/tmp/authdaemond.log $1
# was: 
# exec $DIR/$AUTHDAEMOND -stderrlogger=${sbindir}/courierlogger -stderrloggername=$AUTHDAEMOND $1

2. Issue a manual login

For courier-imap, you will get much better information by not using a mail client and manually logging in using 'telnet'. The transcript of this telnet session may give useful information as to what is going on. If you are going to report a problem to the mailing list, you should certainly include this transcript as well as the corresponding debugging output.

-- to debug POP3 --
# telnet x.x.x.x 110
user USERNAME
pass PASSWORD
stat
quit

-- to debug IMAP --
# telnet x.x.x.x 143
a login USERNAME PASSWORD
a examine inbox
a logout

-- to debug POP3 over SSL --
# openssl s_client -connect x.x.x.x:995
(then use same commands as POP3 example)

-- to debug IMAP over SSL --
# openssl s_client -connect x.x.x.x:993
(then use same commands as IMAP example)

This isn't an option for sqwebmail of course - just login through the web interface and check the authentication debug log which is generated.

3. Interpret the debug output

First, a brief explanation of courier's authentication system. There are a number of standalone client modules which are executed, one after the other, resulting in one of three conditions:

In a typical courier installation, there is only one client module: authdaemon. This formats the authentication request and sends it down a socket to a pool of authdaemond processes (note the extra "d" on the end) which perform the actual work. authdaemond, in turn, contains other authentication modules such as authpam, authmysql, and so on.

You can see this reflected in the configuration files:

[ pop3d/imapd client: /etc/courier/pop3d and /etc/courier/imapd ]
AUTHMODULES="authdaemon"

[ sqwebmail client: /etc/courier/authmodulelist ]
authdaemon

[ authdaemond server: /etc/courier/authdaemonrc ]
authmodulelist="authpam authuserdb"

What happens if authdaemond is not running? Then the client module will time out trying to connect to it, and you will get debug output like this:

Apr 14 13:45:46 billdog pop3d: authdaemon: starting client module
Apr 14 13:46:01 billdog pop3d: authdaemon: s_connect() failed: Operation timed out
Apr 14 13:46:01 billdog pop3d: [Hint: perhaps authdaemond is not running?]
Apr 14 13:46:02 billdog pop3d: authdaemon: TEMPFAIL - no more modules will be tried

If authdaemond is running successfully, then it will in turn run each of the modules it is linked against. If any one returns REJECT then the next is tried; if any returns TEMPFAIL or ACCEPT then no further modules are tried.

So a typical example might look like this:

Apr 14 14:07:15 billdog pop3d: authdaemon: starting client module
Apr 14 14:07:15 billdog authdaemond.plain: received auth request, service=pop3, authtype=login
Apr 14 14:07:15 billdog authdaemond.plain: authcustom: trying this module
Apr 14 14:07:15 billdog authdaemond.plain: authcustom: nothing implemented in do_auth_custom()
Apr 14 14:07:15 billdog authdaemond.plain: authcustom: REJECT - try next module
Apr 14 14:07:15 billdog authdaemond.plain: authcram: trying this module
Apr 14 14:07:15 billdog authdaemond.plain: cram: only supports authtype=cram-*
Apr 14 14:07:15 billdog authdaemond.plain: authcram: REJECT - try next module
Apr 14 14:07:15 billdog authdaemond.plain: authuserdb: trying this module
Apr 14 14:07:15 billdog authdaemond.plain: userdb: opened /etc/userdb.dat
Apr 14 14:07:15 billdog authdaemond.plain: userdb: looking up 'brian'
Apr 14 14:07:15 billdog authdaemond.plain: userdb: entry not found
Apr 14 14:07:15 billdog authdaemond.plain: authuserdb: REJECT - try next module
Apr 14 14:07:15 billdog authdaemond.plain: authpam: trying this module
Apr 14 14:07:15 billdog authdaemond.plain: authpam: sysusername=brian, sysuserid=<null>, sysgroupid=1001, homedir=/home/brian, address=brian, fullname=Brian Candler, maildir=<null>, quota=<null>, options=<null>
Apr 14 14:07:15 billdog authdaemond.plain: pam_service=pop3, pam_username=brian
Apr 14 14:07:15 billdog authdaemond.plain: dopam successful
Apr 14 14:07:15 billdog authdaemond.plain: authpam: ACCEPT, username brian
Apr 14 14:07:15 billdog pop3d: authdaemon: ACCEPT, username brian
Apr 14 14:07:15 billdog pop3d: LOGIN, user=brian, ip=[::ffff:127.0.0.1]

What's happening here?

• The 'authdaemon' client module has been started
• The request was received by 'authdaemond.plain'
• It tries 'authcustom' - this module does nothing unless you have customised it yourself, so it REJECTs the request
• It tried 'authcram', but since this was a request with authtype=login (rather than authtype=cram-md5, say), this module cannot handle it so it REJECTs
• 'authuserdb' has a go. In this case there is an /etc/userdb.dat file for it to look in, but the requested username 'brian' does not exist in there, so it REJECTs
• 'authpam' has a go. It finds the username and home directory in /etc/passwd, and then calls the PAM subsystem to authenticate. The authentication is successful.

So, in principle, debugging is straightforward. Watch the modules operate, search for the one which you think should be authenticating the user, and if it is not, check for REJECT (user not known or password mismatch) or TEMPFAIL (internal error) status. Additional messages should indicate why this status was returned.

Why "authdaemond.plain"? Well, in fact you can have up to four separate authdaemond modules built:

They are separated this way to make it easier to package and distribute the software, although it does mean that you can't have a single configuration which authenticates against two different database backends.

The startup script /usr/lib/courier/authlib/authdaemond makes an arbitary decision as to which one to run. If it makes the wrong choice you can override it: for example,

[ /etc/courier/authdaemonrc ]
version="authdaemond.ldap"

4. Read the documentation

Most of the configuration files like authldaprc, authmysql are well documented with comments.

For the nitty-gritty details of authentication modules, see man authlib. There is probably a copy of this manpage installed on your system; if that command doesn't work, try one of these:

# man -M /usr/man authlib
or
# cd /path/to/sources
# cd authlib
# nroff -mandoc authlib.7.in | less

If you are using userdb authentication, you definitely need to read man makeuserdb, man userdb, and man userdb.

5. Use the mailing list

Please read through the common problems and solutions at the bottom of this document. The next thing to do, of course, is search the web to see if your particular problem has been seen before and solved. Google is very good for this.

If you still cannot work out what the problem is, then you can ask on the courier-imap or sqwebmail mailing lists. But before you post, please gather together all the following information:

• The operating system and version you are running
• The version of courier-imap/sqwebmail you have installed
• The ./configure command line you gave to build it
• If you didn't build it yourself, where you got the package from (and if possible, find out from the packager what options they used to build it)
• The versions of any other relevant software which you are linking against, e.g. vpopmail, openldap, mysql, pgsql
• The transcript of the 'telnet' session you used to test [courier-imap]
• The corresponding debug output which was generated for that session
• The contents of the 'pop3d' and 'imapd' configuration files [courier-imap]
• The contents of the 'authmodulelist' configuration file [sqwebmail]
• The contents of the any other relevant configuration files, e.g. authldaprc, authmysqlrc
• A copy of the database entry you are trying to authenticate against: e.g. the line from your userdb file, an LDAP entry, a row from your mysql table, the line in /etc/password, etc.

If you include all this, you are much more likely to get a helpful response.

Frequently seen authentication problems and solutions

See also the Courier MTA FAQ

I get intermittent authentication problems with vpopmail

This is a known problem with vpopmail 5.2.1 and earlier. You need to upgrade to 5.2.2 or later, and then recompile courier-imap/sqwebmail.
As a workaround, instead you can just recompile courier-imap/sqwebmail with the flag "--without-authdaemon". However that's less efficient.

Compiling fails with "cannot find -lvpopmail"

This is usually a permissions problem on the vpopmail library. Try:

# chmod +rx /home/vpopmail /home/vpopmail/lib

# CPPFLAGS="-I/home/vpopmail/include"; export CPPFLAGS
# LDFLAGS="-L/home/vpopmail/lib"; export LDFLAGS
# ./configure ...

When I try to login with POP3 using telnet, the server disconnects immediately after the "PASS" command, without giving a -ERR response

The reason for this error will probably be found in your mail logs. Usually it indicates either that the home directory does not exist (chdir failed), or the Maildir has not been created. See 'man maildirmake'.

"authdaemon start" returns immediately with AUTHFAILURE

You missed the "d" from the end of "authdaemond". The right command is

# /usr/lib/courier/authdaemond start

I get logged "pop3d: /usr/lib/courier/authlib/authfoo: No such file or directory"

If you are running a standard configuration with authdaemond, then you must leave

AUTHMODULES="authdaemon"

PAM authentication says "pam_start failed, result 4 [Hint: bad PAM configuration?]"

Probably your PAM configuration is bad. If you have /etc/pam.d/other, then try simply removing /etc/pam.d/pop3 and /etc/pam.d/imap and see if it works (this is sufficient for FreeBSD). Otherwise, try copying one of your existing /etc/pam.d/xxx files to /etc/pam.d/pop3, imap or webmail respectively.
The result value is a PAM_XXXX constant from /usr/include/security/pam_constants.h (this file may be in a different location on your system). Under FreeBSD, 4 is PAM_SYSTEM_ERR.

When I connect on the SSL ports (995 or 993), the server accepts the connection but then immediately disconnects

You probably didn't install any SSL certificates. Courier-imap comes with scripts you can run to do this for you:

# /usr/sbin/mkimapdcert
# /usr/sbin/mkpop3dcert

I expected courier-imap/sqwebmail to compile authmysql (or some other module), but it's not there

Firstly, remember that in the normal arrangement with authdaemond, there will be no standalone "authmysql" module. What you will get instead is a version of authdaemond installed which has the authmysql module built in; this program will be called "authdaemond.mysql" and you'll normally find it in /usr/lib/courier/authlib/.
If this doesn't exist, then perhaps ./configure was unable to find your mysql libraries (you can read through the file 'config.log' in the authlib subdirectory to see what it found). You may need to force it to look in the right place, as follows:

# ./configure --with-authmsql --with-mysql-libs=/usr/local/mysql/lib  \
              --with-mysql-includes=/usr/local/mysql/include

On some systems (e.g. FreeBSD), the mysqlclient library dpends on the math and compression libraries. For these systems, try:

# LDFLAGS="-lm -lz" ./configure --with-authmysql ... same as before