samhain file integrity scanner | online documentation


Frequently Asked Questions for Samhain


Rainer Wichmann

FAQ Revised: Saturday 14 April 2007 17:05:58

Table of Contents

1. Most frequently
2. Build and install
3. File checking
4. Client/Server
5. Email
6. Misc
7. Database

1. Most frequently

1.1. Owner not trustworthy / Group writeable and member not trustworthy
An untrusted user (might be an untrusted group member for group writeable files/directories) owns or can write to an element in the path listed in the error message. This concerns the configuration file, the log file, and the database file. The offending element in the path is identified as obj=/xxx in the error message. To fix the problem, see next entry.

1.2. samhain exits with the message "Untrusted path" for config/log/pid/database files
Paths to critical files (e.g. the configuration file) must be writeable by trusted users only. If a path element is group writeable, all group members must be trusted. By default, only root and the (effective) user of the program are trusted. To add trusted users, use the compile time option 
$ ./configure --with-trusted=0,...
or the configure file option: 
[Misc]
TrustedUser=username
If the path to the configuration file itself is writeable by other users than root and the effective user these must be defined as trusted already at compile time.

1.3. It does not log anything / Can't stop logging to console
(1) There is a section in the manual dealing with logging and filtering.
(2) To log to the console: 
$ samhain -p info ...
or in the configuration file: 
[Log]
PrintSeverity=info
To stop logging to the console: 
$ samhain -p none ...
or in the configuration file: 
[Log]
PrintSeverity=none
Defining /dev/null as console device works as well, but is a bad idea, because samhain will open the device and write (i.e. it is a very inefficient method).

1.4. Client cannot self-resolve, but nslookup works fine
  • Nslookup is a program to query Internet domain name servers.
  • Applications (like samhain) are not supposed to query DNS servers directly. Rather, they are supposed to query the resolver library that:
    • is provided by the operating system,
    • configured by the system administrator,
    • may use several different method to determine host names, as configured in /etc/nsswitch.conf, and
    • usually is configured to give precedence to the /etc/hosts file.
  • Therefore, whether nslookup gives correct answers may be completely irrelevant. For self-resolving the own hostname, the resolver library probably will use /etc/hosts, rather than querying a DNS server.

Below you can find some examples of good and bad /etc/hosts files: 

        # CORRECT
	#
        127.0.0.1  localhost
        xxx.xxx.xxx.xxx myhost.mydomain.tld  myhost

        # CORRECT
	#
        127.0.0.1  localhost.localdomain localhost
        xxx.xxx.xxx.xxx myhost.mydomain.tld  myhost

        # BAD
	#
        127.0.0.1  myhost.mydomain.tld  localhost
        xxx.xxx.xxx.xxx myhost.mydomain.tld  myhost

        # BAD
	#
        127.0.0.1  localhost myhost
        xxx.xxx.xxx.xxx myhost.mydomain.tld  myhost


2. Build and install

2.1. [Fedora Core] Cannot compile with --enable-khide
The Fedora Core kernel is patched to unconditionally deny reading from /dev/kmem. Compiling the stealth kernel modules is not possible under these circumstances.

2.2. [Fedora Core] Cannot compile with --with-kcheck
The Fedora Core kernel is patched to unconditionally deny reading from /dev/kmem. Checking the kernel for the presence of rootkits is not possible under these circumstances.

2.3. "make" loops infinitely !
This may happen (e.g. when building via NFS for multiple architectures) if the relative timestamps in the source directory are wrong (time not in sync on different machines) or some intermediate target is unusable (up-to-date, but built for a different OS). Use "touch * && make distclean" in the source directory to recover.

2.4. Why does static compiling (--enable-static) on Solaris fail ?
Ingo Rogalsky has provided the following information: It isn't possible to link Samhain statically with Solaris. This is a Solaris issue (see Sun Infodoc ID12624) and not a samhain problem.

2.5. Compilation fails with '/usr/bin/ld: cannot find -lnss_files'
For Linux, this is a known problem with --enable-static if you compile in MySQL support. The problem is that the mysql_config that comes as part of the MySQL distribution script incorrectly lists dependencies on the libnss_files and libnss_dns libraries which are only available as shared libraries, so the linker cannot find the static libraries. You can check this by inspecting the output of mysql_config --libs. The version of mysql_config that comes with the RedHat mysql RPM (RedHat 9) does not have this bug; the one distributed by the MySQL people has. You can fix the problem by editing mysql_config: search for the client_libs variable, and remove all instances of -lnss_files and -lnss_dns.

2.6. The executable is corrupted after installation
The executable will get stripped during the installation. On suitable systems (i386 Linux/FreeBSD currently), additionally the "sstrip" utility (copyright 1999 by Brian Raiter, under the GNU GPL) will be used to strip the executable even more, to prevent debugging with the GNU "gdb" debugger. The "strip" utility cannot handle the resulting executable, therefore trying to strip manually after installation will corrupt the executable.

2.7. --enable-xml-log has no effect
If you have compiled for stealth, you won't see much, because if obfuscated, then both a 'normal' and an XML logfile look, well ... obfuscated. Use samhain -jL /path/to/logfile to view the logfile.

2.8. ./install-sh: strip: not found (Solaris)
Install the SUNWbtool package.

2.9. What is sh_tiger1.s?
This is a precompiled assembly file for the i386 architecture generated from sh_tiger1.c using gcc 3.4.0 with the following options, that were found to generate the fastest code: 
 -O1 -fno-delayed-branch -fexpensive-optimizations -fstrength-reduce 
     -fpeephole2 -fschedule-insns2 -fregmove -frename-registers -fweb 
     -momit-leaf-frame-pointer -funroll-loops
These options were determined using acovea 5.1.1 by Scott Robert Ladd. The file is provided as precompiled assembly because different versions of gcc can have very different performance, require different options to compile optimal code, and it would be impossible to maintain a library of optimal compile options for every version of gcc.

2.10. Why does static compiling (--enable-static) on MaxOS X fail ?
Static linking is not supported on MacOS X, see Technical Q&A QA1118. This is a MacOS X issue and not a bug in samhain.

2.11. Why does compiling with MySQL fail on Solaris ?
The reason is often the shell script 'mysql_config' that comes as part of MySQL. This script is intended to print appropriate compiler flags for compiling applications that use MySQL. Unfortunately, since Sun compiles MySQL with the Solaris compiler, this script outputs options for the Solaris compiler (i.e. unsuitable for gcc). To solve this problem, you need to move this script (i.e. 'mysql_config') out of your PATH before running ./configure (unless of course you are using the Solaris compiler rather than gcc).

3. File checking

3.1. How can I exclude a (sub-)directory ?
[IgnoreAll]
dir=-1/ignore/this/subdirectory


3.2. In messages about policy violations, what does the code after POLICY [XYZ] mean ?
This code indicates which items are modified (e.g. C = checksum). You can find a description in section 5.4.9 in the user manual. It is there because then you can see in the message list of the Beltane web console what has been modified, without the need to look at the message in detail.

3.3. Does samhain support prelink ?
Yes. There is a special checking policy [Prelink]. Directories with prelinked executables / shared libraries (see /etc/prelink.conf) should be placed under this policy, rather than under the [ReadOnly] policy.

3.4. I get error messages about 'subdirectory count != hardlinks'
Some filesystems do not always follow the rule that the number of directory hardlinks equals the number of subdirectories. E.g. the root directory of reiserfs partitions generally seems to have two additional hardlinks. To account for such exceptions, you can either switch off the hardlink check globally, or specify exceptions: 
[Misc]
# Switch off hardlink check
#
UseHardlinkCheck=no

[Misc]
# Specify exceptions for the hardlink check
#
HardlinkOffset=N:/path
Here, N is the numerical offset (actual - expected hardlinks) for '/path'. For multiple exceptions, use this options multiple times (note that '/path N:/path2' would itself be a valid path, so using the option only once with multiple exceptions on the same line would be ambiguous).

4. Client/Server

4.1. I don't want to poke a hole into my firewall to let the client connect to the server !
Pat Smith has posted the following solution. On the client, create an iptable rule as follows (note: you probably don't need this if you configure / compile in 127.0.0.1 as the server address): 
iptables -t nat -A OUTPUT -p tcp -m tcp --dport 49777 -d server-ip -j REDIRECT
On the server, create an ssh tunnel for each client outside the firewall: 
ssh -f -C -R 49777:localhost:49777 -N client-ip
It is necessary that each client has a distinct name, and that the server knows the name of the client. With the setup above, each client will appear as "localhost" to the server, thus the server needs to trust the client name as reported by the client itself, and suppress all errors on resolving this name to the apparent address. In the server configuration: 
[Misc]
SetClientFromAccept = false
SeverityLookup = debug
Obviously, self-resolving must work on the client machine, otherwise you are in trouble (see next issue).

4.2. The client sends 127.0.0.1 (or some other numerical address) as its name to the log server
See 'Client cannot self-resolve' in the 'Most frequently' section

4.3. The server wants to send rc.ip-adress rather than rc.fqdn to the client
The client self-resolves to its ip address. See 'Client cannot self-resolve' in the 'Most frequently' section

4.4. Cannot resolve client name host=XXX
The server must be able to determine the client name.
This is because only authenticated connections from registered 
clients are allowed, and
the server must be able to check the client hostname against the list of
allowed hosts, and look up the password verifier for that
host.
There are two different ways to accomplish this. Unfortunately, judging from customer feedback as well from common sense, both do not work very well with a messed up local DNS (including /etc/hosts files) and/or überparanoid or misconfigured firewalls (in case of connections across one).

  • First method: Determine client name on client, and try to cross-check on server

    This does not work for a number of people because (1) the /etc/hosts file on the client machine has errors (yes, there are plenty machines with a completely messed up /etc/hosts file), (2) the server cannot resolve the client address because the local DNS is f***ed up, or (3) the client machine has multiple network interfaces, and the interface used is not the one the client name resolves to.

    If the client uses the wrong interface on a multi-interface machine, there is a config file option SetBindAddress=IP address that allows to choose the interface the client will use for outgoing connections.

    If you want to download the config file from the server, you should instead use the corresponding command line --bind-address=IP address to select the interface.

    If you encounter problems, you may (1) fix your /etc/hosts file(s), (2) fix your local DNS, or (3) switch to the second method.

    Errors in name resolving/cross-checking can be avoided by setting a very low severity (lower than the logging threshold), e.g.

    SeverityLookup=debug

    in the Misc section of the server configuration, if you prefer running unsafe at any speed instead of fixing the problem (you have been warned). Doing so will allow an attacker to pose as the client.

  • Second method: Use address of connecting entity as known to the communication layer

    This has been dropped as default long ago because it may not always be the address of the client machine. To enable this method, use

    SetClientFromAccept=true

    in the Misc section of the server configuration file. If the address cannot be resolved, or reverse lookup of the resolved name fails, no error message will be issued, but the numerical address will be used.



4.5. Cannot resolve socket peer IP for client host=XXX peer=YYY
See above

4.6. Reverse lookup of socket peer failed host=XXX peer=YYY obj=ZZZ
See above

4.7. No socket peer alias matches client name host=XXX peer=YYY
See above

4.8. Session key negotiation failed
See the document HOWTO client+server troubleshooting

4.9. Invalid connection attempt: Not in client list
See the document HOWTO client+server troubleshooting

4.10. Invalid connection attempt: Session key mismatch
See the document HOWTO client+server troubleshooting

4.11. How do I update the file signature database ?
If you keep the file signature database on the server, the database is supposed to be updated on the server, using the beltane web-based console (currently in beta) and the log messages from the client.

Alternatively, you can scp the database to the client, run samhain -t update -l none --foreground (you need to avoid logging because otherwise you will get in conflict with the running samhain daemon), and then scp the database back to the server. Actually, with a properly set up "ssh", using RSA/DSA authentication and ssh-agent you could write a script to automate this.

4.12. Time limit exceeded
The respective client for that this message is generated has not sent anything for some interval of time (default 84600 sec = 1 day). The interval can be set as follows: 
        [Misc]
	# unit is seconds
        SetClientTimeLimit=NNN
This feature has the purpose to detect if a client is dead. You might want to ensure that timestamps are sent to the server: 
        [Log]
	ExportSeverity=mark
If you don't want to use this feature, set the time limit to some very large value.

4.13. Invalid connection attempt: Signature mismatch
Clients sign their messages using a session key negotiated with the server. The message indicates that the server could not verify the signature. This may be caused by a running two instances of samhain on the same client machine, both of them accessing the server (and negotiating different session keys ...). The system will recover automatically from the problem by forcing the failed client to negotiate a fresh session key.

4.14. [Server] PANIC .. Address already in use   subroutine=bind
The server cannot bind to its port because the port is already used. Maybe you have accidentially already an instance of the server running.

5. Email

5.1. Reverse lookup failed
Fix your DNS (reverse lookup: numerical IP address to FQDN, to verify FQDN to numerical IP address). 
Whether "nslookup" works is not very informative, because 
"nslookup" does not use the resolver library of the operating
system. Therefore,
it is not exactly the
best tool for debugging name resolving problems (see the book
"DNS and bind").


5.2. From daemon@example.com
samhain fails to resolve the self-address of the host. See 'Client cannot self-resolve' in the 'Most frequently' section.

5.3. How do I define more than one email addresses ?
Use SetMailAddress=... multiple times (upt to eight addresses are possible, with at most 63 characters per address): 
[Misc]
SetMailAddress=aaa@foo.com
SetMailAddress=bbb@foo.com


6. Misc

6.1. Error message: "Invalid line XYZ in configuration file"
This message indicates that line XYZ in the configuration file contains an unrecognized directive. The primary reasons are:
(a) The directive should be placed into a particular section of the configuration file, but the section header is not present (or you forgot to uncomment it).
(b) Samhain is compiled without support for this directive.
(c) You have a typo in the directive.


6.2. Why do I get a local logfile if I log to the server ?
Because you can use all log facilities in parallel. You should switch off in the config file what you don't want/need: 
        [Log]
        # local log file
        LogSeverity=none


6.3. Why is there no NIS support with a static samhain executable on Linux ?
Some functions (including NIS) require libraries that are only available as shared libraries with modern GLIBC versions. While you can always compile a static executable, normally it would still open the shared library at runtime. As of version 1.8.11, samhain avoids this by providing replacement functions from uClibc. However, these do not include NIS support.

6.4. Why do I get hundreds of messages about modified CTIME ?
This happens because some backup applications reset the atime/mtime timestamps, which causes the ctime timestamp to be modified (rootkits avoid this by temporarily resetting the system clock to the original ctime ...).

To fix this problem, read the manual of your backup application, or redefine the ReadOnly policy to not check the ctime timestamp: 

        [Misc]
        RedefReadOnly=-CTM

        Order matters - you must first redefine 
        ReadOnly before you use it


6.5. PANIC — File not accessible
Most likely permission denied because of unsufficient privileges.

6.6. How can I avoid error messages for invalid UIDs (no such user) ?
Set SeverityNames to a low value 
[EventSeverity]
SeverityNames=debug


6.7. [Redhat] The /etc/init.d/(samhain|yule) init script hangs
Redhat uses "initlog" (see man initlog) in initscripts. If it hangs, most probably samhain/yule runs in the foreground rather than as daemon. Set daemon mode in the configuration file: 
[Misc]
Daemon=yes


6.8. The /etc/init.d/(samhain|yule) init script exits with: execvp: No such file or directory
Either the program is not installed, or it is not in the PATH (the one used by the init script, which may be different from your PATH).

6.9. Why am I not receiving the "BEGIN LOGKEY" message by email ?
This message (which contains the key to verify the log file) is generated when logging to the log file starts. It has the severity "ALRT", thus you should make sure that you have set the logging threshold for email correctly to receive it.

6.10. Why does console logging fail if I compile with --enable-(micro-)stealth ?
The default logging options are more "stealthy". Set the threshold explicitely rather than relying on the default.

6.11. I need a list for my schedule !
You can have the same effect with a list of schedules. See the section "Timing file checks" in the manual.

6.12. The hiding kernel module has no effect !
Most probably you compiled using the wrong "System.map" file.

6.13. What does the message "Large lstat/open overhead" mean ?
Your system needs several seconds to proceed from an lstat() system call to an open() system call. This is a tremenduous overhead, and indicates that either your system has a really severe performance problem, or someone tries to slow down samhain.

6.14. What does the message "Device not available path=/dev/random" mean ? I have /dev/random !
/dev/random blocks unless there is some entropy it can deliver. Samhain will time out and fall back on /dev/urandom after some seconds to avoid hanging for a potentially long time. It will try /dev/random again next time it needs entropy.

6.15. Logging to an external program fails; the program receives no data on stdin !
Probably your program is not designed to wait for input, but exits if reading fails (because there is no data yet). You may want to let your program wait for the terminating "[EOF]" line.

6.16. SIGILL on AIX
For each scanned file, samhain needs to store some information in memory (e.g. to recognize changes that have already been reported, and avoid duplicate reports). On AIX, if you are checking a really huge number of files, memory usage may exceed the default limit of 256 MB, and the process may terminate with SIGILL.

The problem can be solved by linking with the flag -bmaxdata:0x80000000. This allows the application to access up to 8 segments (where each segment is 256MB).

If you are using gcc, you need to use instead the flag -Wl,bmaxdata:0x80000000, which tells gcc to pass on the bmaxdata flag to the AIX linker. You can use the LDFLAGS environment variable to pass linker flags to the configure script: 

     export LDFLAGS="-Wl,bmaxdata:0x80000000"


7. Database

7.1. Why are client messages corrupted / incompletely stored in the DB ?
Because the messages are not in XML format, and therefore incorrectly parsed. The most frequent reasons are: 
        1.) Your server is compiled with --enable-xml-log, but your client(s)
        is/are not.

        2.) In your client or server configuration file, you are using
        the option for a custom message header, but without paying attention
        to preserving the XML format.


7.2. I want / don't want the server timestamps (for client messages) in the SQL database
[Database]
SetDBServerTstamp = true/false
This will enable/disable logging of the server timestamp for client messages. The server timestamp will be written to a seperate record, with log_ref set to the value of log_index of the corresponding client message.

7.3. I don't want the client TIMESTAMP messages in the SQL database
     Sending timestamps from the client allows the server to detect if
     a client is not running anymore (use SetClientTimeLimit=NNN in the
     [Misc] section of the server config file to set the number of seconds
     after which the server will issue an error message if no timestamp has
     been received).
However, you might not want to log these timestamps to the database (or other log facilities). To filter them, you can use two methods (examples are for the SQL database). The first one has the disadvantage that only messages of severity err or higher will be logged: 
     [Misc]
     UseClientSeverity=yes

     [Log]
     DatabaseSeverity=err
The second method is more specific — log everything not belonging to the STAMP class of messages: 
     [Misc]
     UseClientClass=yes

     [Log]
     DatabaseClass=PANIC RUN FIL TCP ERR ENET EINPUT


7.4. What does the log_ref field mean ?
NULL are client messages. Nonzero integer is a server timestamp for a client message (where log_ref indicates the log_index entry number of the corresponding client message). Zero indicates a message by the server itself (e.g. the server's start message).

7.5. How can I check what is in the database ?
Use a command line client to login to the database and query it: 
     sh$ mysql -u <user_name> -p <database_name>
     Enter password: ****
     mysql> SELECT log_index,log_ref,log_host,log_sev,log_msg,path FROM <table_name> WHERE entry_status = 'NEW' ORDER BY log_index;
     ....
     mysql> \q


Copyright (c) 2004 Rainer Wichmann

This list of questions and answers was generated by makefaq.