Complete Reference Guide To Creating a Secure Remote Log Server
Author: Loki (loki@fatelabs.com)
www.fatelabs.com


Introduction
Welcome to the first issue of a series of security-related HOWTO papers. In these documents both Fyodor and I aimed to give various tips, hints and tricks on various aspects of Internet security. Comments and suggestions are welcome and can be emailed to us directly at the contact information provided at the bottom of this paper.

We hope you find it useful and recommend hammering away at the README files for all utilities being referenced in this paper.

In this paper we make references to and as headers. In this, SERVER, refers to nothing more then the machine that is configured to receive logs from remote machines (CLIENTS). CLIENT, referring to the machine that is configured to send its logs to the REMOTE LOG SERVER.

Loki
Founder/Senior Research Scientist
loki@fatelabs.com

What is a Remote Log Server
A remote log server is nothing more then a system preconfigured at install-time to provide hard drive space for other systems to log to. This system must be completely secured and locked down. No unencrypted remote access should be allowed, all RPC Daemons and other misc. services should be turned off as well. The only data allowed to the machine should be UDP/Port 514. We will be walking you through a step-by-step process that details how to configure, install, and deploy a remote log server. Utilizing some of the most renowned security experts across the globe for input, I've compiled a comprehensive, and easy to understand guide on ensuring this to be a successful launch.

Recompiling syslogd
The first step in turning on remote logging capabilities on your machine is recompiling syslogd to read a completely different syslog.conf file. This is for security reasons, which we will detail more later in this paper.

NOTES

This was an idea derived from the "To Build a Honeypot" whitepaper written by Lance Spitzner. Thanks for your contributions to the security community, Lance.

The beautiful idea behind this strategy is to fool the intruder into thinking there is no remote logging setup on the machine. By leaving the old /etc/syslog.conf file in tact, also helps in tricking the unsuspecting cracker.

STEP 1: Compiling Our New Version

In our first step, we face somewhat of a problem in that syslogd ships with the operating system by default, so no source code is provided with it. You will need to download the source code from an outside party, such as Redhat, or even Joey's FTP site (the creator of syslogd). As of this writing, Joey's ftp site is down so I have provided an alternative resource to download the needed file. Your distribution however, would be in violation of Open Source policies by not providing the source code to you. So check your particular OS vendor for the source code if you do not want to download it from any of the links specified in this paper.

SITE 1: ftp://ftp.infodrom.north.de/pub/people/joey/sysklogd/ (currently down as of this writing)
or
SITE 2: ftp://ftp.sourceforge.net/pub/mirrors/redhat/redhat/redhat- .2/SRPMS/SRPMS/sysklogd-1.3.31-16.src.rpm

After 'rpm -iv -v sysklogd-1.3.31-16.src.rpm, the source code should have been installed to /usr/src/redhat/SOURCES/sysklogd-1.3-31 (that is, if you are running Redhat)

STEP 2: Changing the Default Config File Location

Here is the fun part. This is used to fool an intruder once he has compromised the system into thinking your log files are stored locally. We will be recompiling syslogd to read a completely separate configuration file, leaving the old default one in tact, just in case the intruder runs a 'cat' on the /etc/syslogd.conf file to check if remote logging has been enabled. By leaving the old configuration there proceeding our recompile of the new binary, ensures that we can fool the cracker into thinking otherwise.

Lets modify the default location of /etc/syslog.conf

[loki@fatelabs.com sysklogd-1.3-31]$ cd /usr/src/redhat/SOURCES/sysklogd-1.3-31

[loki@fatelabs.com sysklogd-1.3-31]$ vi syslogd.c do a /grep for syslog.conf

This will bring us to #define _PATH_LOGCONF "/etc/syslog.conf"

Change the above location value to something you prefer. I suggest something like /etc/.sys/CORE.conf or something ingenious like that.

Copy the new syslogd binary over the old (typically /sbin/syslogd)

BACKUP old version before doing this. As an admin, you should have already known that, right? :)

NOTES

The reason why we do not simply just start the old syslogd with the [-f conffile] switch is because if an attacker were to grab a 'ps' list of processes, the switch you used will show up, telling the intruder that we are specifying a completely different configuration file. Ultimately, defeating this purpose. I recommend leaving the old /etc/syslog.conf file in tact as it completely throws off the intruder. Also, keep local logs, as if the /var/log directory is empty, well, use your imagination on why.

I would have to make the suggestion to use a different filename, other then syslog.conf when specifying the new location. Understand that an intruder can also just do a find / -name "syslog.conf" and locate another copy of the configuration file.

STEP 3: Modify the (Real) Syslog Config File

For example, to forward ALL messages to a remote host use the following syslog.conf entries:

# Sample syslogd entry to forward all messages to a remote host.
*.* @hostname

# To forward all kernel messages to a remote host the configuration file would be as follows:
kern.* @hostname

# Remote Logging which includes local logging as well (for fault tolerance)
kern.crit @finlandia
kern.crit /dev/console

.. and so on.. repeat for all log files you wish to keep offsite. This includes third party applications.

The Client (Remote Log Server)
With the client server, (or the machine accepting all of the remote logs), all you have to do is launch syslogd with the -r switch, specifying remote logging capabilities. Make sure that you remember to ADD: syslog 514/udp to the /etc/services file as well, on the server, or it will not listen to the specified port.

NOTES
If the network consists of different domains you don't have to complain about logging fully qualified names instead of simple hostnames. You may want to use the strip-domain feature -s of this server. You can tell the syslogd to strip off several domains other than the one the server is located in and only log simple hostnames.

ADD 'syslog 514/udp' to the /etc/services file.

If this entry is missing syslogd neither can receive remote messages nor send them ADD '*.* @hostname' to the /etc/syslog.conf file. To forward all Kernel logging messages to a remote host,

Make sure that if you do not specify an IP ADDRESS that you add the host to the /etc/hosts file to avoid DNS RESOLUTION errors.

If the remote host is located in the same domain as the host, syslogd is running on, only the simple hostname will be logged instead of the whole fqdn. (fully qualified domain name)

Locking Down the Remote Log Server
One of the most crucial steps in this project is to fully lock down the machine that is receiving all of the logs, or referred to as the remote log server.

1. Turn off (ALL SERVICES)

Remember, this is ONLY a log server, so all other services should NOT be turned on. Lets walk through that process right now.

[root@myhost /etc]# cd /etc
[root@myhost /etc]# vi inetd.conf

1a. Turn off all inetd services

You will want to put a '#' comment in front of EVERY service listed in this file. The inetd.conf file is IMHO one of the largest mistakes of the Linux operating system :). But the ability to disable the inetd services makes it all better :)

You will want to comment out everything:

a. echo
b. discard
c. daytime
d. chargen
e. time
f. ftp
g. telnet
h. shell
i. login
j. exec
k. comsat
l. talk
m. ntalk
n. dtalk
o. pop-2
p. pop-3
q. imap
r. uucp
s. tftp
t. bootps
u. finger
v. cfinger
w. systat
x. netstat
y. auth
z. linuxconf
zz. swat

1b. DISABLE ALL RPC SERVICES
[root@myhost /etc]# cd /etc/rc.d

Anything listed in the rc. directories that start with a capital "S" means that the service will start at boot time. In order to disable it from starting, you will want to 'mv to (a lowercase 's'). e.g. mv S11portmap s11portmap // This disables portmapper from starting at boot time.

You will want to do this with all unneeded services in this directory. These include: nfslock, apmd, netfs, identd, autofs, portmap, atd, pcmcia, I would even disable the inetd startup script as well, isdn, sendmail, gpm, httpd, vmware, xfs,linuxconf, local, You are basically (other then the obvious services) are going to want to disable all of these rc controlled services.You will want to go through all of the rc0.d-rc6.d directories looking for these same files to disable.

2. Disable Accounts

[root@myhost /etc]# vi /etc/passwd

Go through your password file and remove practically every account that isn't being used. I would suggest downloading the tool, /bin/noshell (http://www.cerias.purdue.edu/coast/archive/data/categ50.html)and use this with all accounts in the password file that should not have remote login access. This is a very informative program that provides admins additional information on who is attempting to login with disabled accounts.

3. Install SSH

Since the release of OpenSSH,giving the GNU opensource community free access to secure shell, telnet should now be obsolete in a security centric environment. GET AWAY from using telnet. With packet sniffers getting easier and easier to use (tools such as dsniff, 'www.dsniff.org') allows even the biggest nitwit to snarf packet data. So case-in-point, don't use telnet.

I bet you are so happy, because guess what? I'm going to walk you through the installation of OpenSSH! (ohh goody)..

3a. Downloading OpenSSH

Point your web browser to www.openssh.com, as of this writing they are on version 2.1.1 which has support for both SSH1 and SSH2 protocols. The release date was June 08, 2000. After downloading the source tarball, go ahead and do the following:

[root@myhost ]# gzip -d openssh-2.1.1p2.tar.gz

[root@myhost ]# tar -xvf openssh-2.1.1p2.tar

I can't realistically walk you through everything, so please save me some heartache, as well as yourself, and just read the INSTALL file before continuing. I have no idea what particular requirements your system has. I will however, walk you through my install.

First you will need working installations of both Zlib and OpenSSL (yes their is an OpenSSL project as well)

Zlib:
http://www.freesoftware.com/pub/infozip/zlib/

OpenSSL 0.9.5a or greater:
http://www.openssl.org/

Or, if you're even more lazy then the audience this paper was written for, (God forbid), then grab the RPM's from the following link. RPMs of OpenSSL are available at http://violet.ibs.com.au/openssh/files/support

Alternatively, Jim Knoble has written an excellent X11 passphrase requester. This is maintained separately at: http://www.ntrnet.net/~jmknoble/software/x11-ssh-askpass/index.html

If you wish to build the GNOME passphrase requester, you will need the GNOME libraries and headers.

GNOME:
http://www.gnome.org/

Also, grab GNUmake if you do not yet have it, OpenSSH has only been tested with this program.

[root@myhost ]# ./configure && make && make install

This will install the binaries in /opt/{bin,lib,sbin}, but will place the config files in /etc/ssh

If you are using PAM, you will need to manually install a PAM control file as "/etc/pam.d/sshd" (or wherever your system prefers to keep them). A generic PAM configuration is included as "contrib/sshd.pam.generic", you may need to edit it before using it on your system. If you are using a recent version of Redhat Linux, the config file in contrib/redhat/sshd.pam should be more useful. Failure to install a valid PAM file may result in an inability to use password authentication.

[root@myhost ]# tar -xvf zlib.tar
[root@myhost ]# cd zlib-1.1.3

[root@myhost zlib-1.1.3]# ./configure && make && make install
You shouldn't get any errors, if you do, sorry to do this, but refer to the README file. What I like to do when compiling something and something goes wrong, I just grep a keyword from the error on the provided README file(s) to see if there is anything specifically mentioned about my error. (Just a suggestion)

To install OpenSSL, you will need:

Perl 5
an ANSI C compiler
a supported Unix operating system [root@myhost ]# tar -xvf openssl-0.9.5a.tar

[root@myhost ]# ./config && make && make test && make install

Now lets build OpenSSH!

[root@myhost ]# cd openssh-2.1.1p2

[root@myhost ]# ./configure --without-PAM && make && make install

NOTES

Because I opt for RSA authentication, I run configure with the --without-PAM option as I am not a big fan of basic password authentication. With the advancements in Cryptography and more secure forms of authentication, PAM and other password-type authentication mechanisms IMHO will become obsolete within a few years.

When you have successfully compiled and installed OpenSSH, you will receive the following information.

------ snip ------
Key generation complete.
Your identification has been saved in /usr/local/etc/ssh_host_key.
Your public key has been saved in /usr/local/etc/ssh_host_key.pub.
The key fingerprint is:
d5:74:83:d0:3f:c4:b4:d6:c5:39:1d:94:ee:9b:a8:61 root@fatelabs.com.com
Generating DSA parameter and key.
Your identification has been saved in /usr/local/etc/ssh_host_dsa_key.
Your public key has been saved in /usr/local/etc/ssh_host_dsa_key.pub.
The key fingerprint is:
ed:58:65:b9:8b:fe:05:81:c2:8c:06:c9:cb:ac:bb:e6 root@fatelabs.com.com
------ snap ------

3b. Configuring OpenSSH
Even though the default installation of OpenSSH should work, I would take a quick stab at the ssh config files to see if you want to change anything.

Because we configured OpenSSH without PAM support, we want to enable RSA Authentication. Uncomment it in the /usr/local/etc/ssh_config file respectively (this is for the system-wide client configuration).

Then go ahead and edit the SSHD config file, /usr/local/etc/sshd_config

Lets go ahead and disable PermitRootLogin (do this by changing 'yes' to 'no' Go ahead and save the file and quit.

After installation, a host key and DSA key should have been created for you. So lets go ahead and move to the next step.

NOTES ON SSH v2

THE SSH PROTOCOL v2 SSH protocol version 2

In SSH v2, each host has a host-specific DSA key used to identify the host. However, when the daemon starts, it does not generate a server key. Forward security is provided through a Diffie-Hellman key agreement. This key agreement results in a shared session key. The rest of the session is encrypted using a symmetric cipher, currently Blowfish, 3DES or CAST128 in CBC mode or Arcfour. The client selects the encryption algorithm to use from those offered by the server. Additionally, session integrity is provided through a cryptographic message authentication code (hmac-sha1 or hmac-md5).

Protocol version 2 provides a public key based user authentication method (DSAAuthentication) and conventional password authentication.

AUTHORIZED_KEYS File Format
The $HOME/.ssh/authorized_keys file lists the RSA keys that are permitted for RSA authentication in SSH protocols 1.3 and 1.5 Similarly, the $HOME/.ssh/authorized_keys2 file lists the DSA keys that are permitted for DSA authentication in SSH protocol 2.0. Each line of the file con- tains one key (empty lines and lines starting with a `#' are ignored as comments). Each line consists of the following fields, separated by spaces: options, bits, exponent, modulus, comment. The options field is optional; its presence is determined by whether the line starts with a number or not (the option field never starts with a number). The bits, exponent, modulus and comment fields give the RSA key; the comment field is not used for anything (but may be convenient for the user to identify the key).

Note that lines in this file are usually several hundred bytes long (be- cause of the size of the RSA key modules). You don't want to type them in; instead, copy the identity.pub file and edit it.

Important Files
Creating User Specific Key Files
With each user that you want to have access to the SSH server for, run /usr/local/bin/ssh-keygen on the machine you are going to be ssh'ing from. After running ssh-keygen, goto $HOME/.ssh/ and cp identity.pub -> authorized_keys
Key Location Overview
The system you want to login to (the SSH Server) is where you store the $HOME/.ssh/authorized_keys file. This is the public key file for your user. The server will require this to decrypt your private key on your host machine. The system you are logging in FROM is where you store the $HOME/.ssh/identity file. This is the private key file you generated.

NOTES

[root@myhost /usr/local/bin]# ./ssh-keygen -d
This -d flag will create the DSA Public Keyfile needed for DSA Authentication. (When generating host keys, you can not specify a password)

Once you have generated your key files, go ahead and plop your sshd module file into the /etc/pam.d/ directory. You are probably wondering, jeese, will Eric provide me with a working sshd module? Well the answer is YES :) See below

---- snip -----
#%PAM-1.0
auth required /lib/security/pam_unix.so shadow nodelay
auth required /lib/security/pam_nologin.so
account required /lib/security/pam_unix.so
password required /lib/security/pam_cracklib.so
password required /lib/security/pam_unix.so shadow nullok use_authtok
session required /lib/security/pam_unix.so
session required /lib/security/pam_limits.so
----- snap -----

Remember, this is being used on my RH 6.2 box only. If it does not work, consult the generic copy provided with the SSH distribution.

Overview
Now you should have ALL services turned off and SSH as the only means of connecting remotely to the machine for administration purposes. The next step we want to do is setup that infamous host-based Firewall!

The Firewall
Packet filtering doesn't really make sense on your logging box until you've got your tcp stack vulnerable to some fragmentation issue, since you have got all the services down anyway and you can control who is permitted to establish secure shell connections to the machine using ssh access control features.

It might be reasonable though to filter out syslog port (514/udp) but it doesn't really stop people from sending garbage to your syslog server, however it does make their job harder since it's udp which could be easily spoofed. This is at least until you have a multihomed machine where all logs come to internal interface so you can safely filter out everything that comes from the outside.

However here are basic recommended filtering rules for machine. For this purpose you can use either ipchains package if you're using Linux (last time I checked ipfilter didn't compile cleanly there) or try to use ipfilter package if you use something else (works perfectly on *BSD and Solaris machines at least).

For ipchains I recommend the following shell script.

-------------- snip --------------------
#!/bin/sh
PATH=/usr/sbin:/sbin:/bin:/usr/sbin
LOCAL_INTERFACE="192.168.1.1/32" # put your real IP address
LOCAL_NETWORK="192.168.1.0/24" # put your local net IP address/mask here
SSH_PERMITTED="192.168.1.2/32 192.168.2.3/32" # who allowed to ssh
SYSLOG_PERMITTED="192.168.1.5/32 192.168.5.2/32" # who allowed to log syslog messages

# deny everything
ipchains -P input DENY
ipchains -P output DENY
ipchains -P forward DENY
ipchains -F

#permit ssh
for ipaddr in $SSH_PERMITTED;
do
ipchains -A input -p tcp -s $ipaddr -d 0/0 22 -i $LOCAL_INTERFACE -j ACCEPT
done

# permit outgoing tcp
ipchains -A output -p tcp -i $LOCAL_INTERFACE -j ACCEPT
ipchains -A input -p tcp ! -y -i $LOCAL_INTERFACE -j ACCEPT

# permit syslog
for ipaddr in $SYSLOG_PERMITTED;
do
ipchains -A input -p udp -s $ipaddr -d $LOCAL_INTERFACE 514 -i $LOCAL_INTERFACE -j ACCEPT
done

# if you would like to log all the other connection attempts,

# uncomment these...
#ipchains -A input -p tcp -i $LOCAL_INTERFACE -l -j DENY
#ipchains -A input -p udp -i $LOCAL_INTERFACE -l -j DENY
#ipchains -A input -p icmp -i $LOCAL_INTERFACE -l -j DENY
----------- snap ---------------

For IP filter package, rules would look approximately like this:

----------- snip ---------------
# close everything on local interface
block in all on le0 from any to any
# pass secureshell
pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22
pass out on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22
# or you can replace these two rules with
#pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 keep state
#pass SYSLOG
pass in on le0 proto udp from 192.168.1.2/32 to 192.168.1.1/32 port = 514
----------- snap ----------------

Log Reporting
What good would a remote log server be if you weren't monitoring the log files. I would recommend several different programs.

Logcheck - www.psionic.com
or
Swatch - www.swatch.org

Below we've included a simple shell script (Courtesy of Mr. Bill Pennington) Which will archive the log files every day/hour/minute to an offsite location (using scp of course).

#!/bin/bash
#Simple script to rotate the log files on a daily basis
#Bill Pennington 1/19/2000


#Set the date variable
date=`date +%m-%d-%Y`

#Rename the messages file
mv /var/log/messages /var/log/messages.$date

#HUP the syslog daemon so it writes to a new file
killall -HUP syslogd

#Compress the file
/bin/gzip /var/log/messages.$date

#Rename the secure file
mv /var/log/secure /var/log/secure.$date

#HUP the syslog daemon so it writes to a new file
killall -HUP syslogd

#Compress the file
/bin/gzip /var/log/secure.$date

#Rename mail file
mv /var/log/maillog /var/log/maillog.$date

#HUP the syslog daemon so it writes to a new file
killall -HUP syslogd

#Compress the file
/bin/gzip /var/log/maillog.$date

#Then scp them somewhere

Time
It is vital to ensure that the logserver has the correct time/date. Get xntpd install it, then "ntpdate timeservername" at least once a day, I do it twice a day. If you want to use your logs in any type of legal proceeding you are going to need accurate time. You should be doing this on all your servers of course.

Other syslog Devices
There are many other devices including CISCO that can be setup to log to syslog.

Having all your cool *nix boxes logging to a central log server is way cool, but what about those other boxes on your network? To get a complete view of your network and servers, setup all your devices to log to syslog. Depending on the number of devices you have, you might want to dedicate a box for each type of device or server. i.e. unix log server, netgear log server, NT log server...

Cisco
All Cisco devices support logging to syslog. The commands to set your Cisco device to log to log messages to syslog vary from device to device. Some examples are provided here but you should check your Cisco documentation for more complete instructions.

Routers
In config mode:
logging
logging facility defaults to local7>

Pix Firewall
In config mode:
logging host
logging facility defaults to LOCAL4>
logging trap from emergencies to debug, be careful with debug you will get a ton of traffic!

Switches
In config mode:
set logging server enable
set logging server
set logging session enable
set logging level all 1 default

Again you will want to look at your Cisco docs or Cisco's web site for complete details but the above examples should get you started.

Windows NT
You can make NT servers forward events to central syslog servers as well. There are several shareware/freeware packages available to accomplish this. Search http://www.bhs.com for syslog to find one that is right for you.

Contributions
Lance Spitzner (Renowned security whitepaper author of the 'Know Your Enemy' series. Thank you for all your support Lance. http://www.enteract.com/~lspitz

Martin Roesch (Creator of the popular Intrusion Detection System (SNORT)). Had a blast with you at Defcon! Hope to see you next year. http://www.snort.org

Bill Pennington (Thanks for all the help with SNORT and your contributions to this paper!) Good luck with everything!

Frank Heidt (@Stake Security Team - Enjoyed the cigar Frank! Hope to see you at Defcon next year as well. Sorry I didn't end up implementing your syslog design into this paper. I didn't think I could explain as perfectly as you could;))

Fyodor Yarochkin (This paper wouldn't have been possible without you bud! Thanks!)

Bio
Loki has been working in Information Security for over 9 years and continues to play an active role in contributions to the open disclosure community, widely sought after from recent advisories on circumvening Virtual Private Network appliances and other security applications. Loki is the Founder and Chief Research Scientist of Fate Research Labs. Loki has authored many security whitepapers and is also the former Manager of Penetration Testing for SBC Datacomm, NUASIS Corporation, and also the former CEO of a now publicly traded company, Netstream, Inc.

A recognized name in the cryptanalysis sector, Loki continues to travel across the globe, from Europe to Asia conducting speeches on circumventing Virtual Private Networks. With deep knowledge in Cryptography, Intrusion Detection, and circumvening VPNs, Loki continues to be the driving force in identifying flaws within the security industry.