Welcome to the first issue of a series of security-related HOWTO papers. In
these documents both Fyodor and I aimed to give various tips, hints and tricks
on various aspects of Internet security. Comments and suggestions are welcome
and can be emailed to us directly at the contact information provided at the
bottom of this paper.
We hope you find it useful and recommend hammering away at the README files for all
utilities being referenced in this paper.
In this paper we make references to <SERVER> and <CLIENT> as headers. In this, SERVER,
refers to nothing more then the machine that is configured to receive logs from remote
machines (CLIENTS). CLIENT, referring to the machine that is configured to send its logs
to the REMOTE LOG SERVER.
- Loki
A. What is a Remote Log Server
A remote log server is nothing more then a system preconfigured at install-time to provide hard
drive space for other systems to log to. This system must be completely secured and locked
down. No unencrypted remote access should be allowed, all RPC Daemons and other misc. services
should be turned off as well. The only data allowed to the machine should be UDP/Port 514.
We will be walking you through a step-by-step process that details how to configure, install,
and deploy a remote log server. Utilizing some of the most renowned security experts across
the globe for input, I've compiled a comprehensive, and easy to understand guide on
ensuring this to be a successful launch.
B. Syslogd
Recompiling syslogd
The first step in turning on remote logging capabilities on your machine is recompiling
syslogd to read a completely different syslog.conf file. This is for security reasons,
which we will detail more later in this paper.
NOTES
This was an idea derived from the "To Build a Honeypot" whitepaper written by Lance
Spitzner. Thanks for your contributions to the security community, Lance.
The beautiful idea behind this strategy is to fool the intruder into thinking there is
no remote logging setup on the machine. By leaving the old /etc/syslog.conf file in
tact, also helps in tricking the unsuspecting cracker.
STEP 1: Compiling Our New Version
In our first step, we face somewhat of a problem in that syslogd ships with the
operating system by default, so no source code is provided with it. You will need to
download the source code from an outside party, such as Redhat, or even Joey's FTP
site (the creator of syslogd). As of this writing, Joey's ftp site is down so I have
provided an alternative resource to download the needed file. Your distribution
however, would be in violation of Open Source policies by not providing the
source code to you. So check your particular OS vendor for the source code if you
do not want to download it from any of the links specified in this paper.
SITE 1: ftp://ftp.infodrom.north.de/pub/people/joey/sysklogd/
(currently down as of
this writing)
or
SITE 2: ftp://ftp.sourceforge.net/pub/mirrors/redhat/redhat/redhat- .2/SRPMS/SRPMS/sysklogd-1.3.31-16.src.rpm
After 'rpm -iv -v sysklogd-1.3.31-16.src.rpm, the source code should have been
installed to /usr/src/redhat/SOURCES/sysklogd-1.3-31 (that is, if you are
running Redhat)
STEP 2: Changing the Default Config File Location
Here is the fun part. This is used to fool an intruder once he has compromised
the system into thinking your log files are stored locally. We will be
recompiling syslogd to read a completely separate configuration file, leaving
the old default one in tact, just in case the intruder runs a 'cat' on the
/etc/syslogd.conf file to check if remote logging has been enabled. By leaving the
old configuration there proceeding our recompile of the new binary, ensures
that we can fool the cracker into thinking otherwise.
Lets modify the default location of /etc/syslog.conf
[ehines@myhost sysklogd-1.3-31]$ cd /usr/src/redhat/SOURCES/sysklogd-1.3-31
[ehines@myhost sysklogd-1.3-31]$ vi syslogd.c
do a /grep for syslog.conf
This will bring us to
#define _PATH_LOGCONF "/etc/syslog.conf"
Change the above location value to something you prefer. I suggest something
like /etc/.sys/CORE.conf
or something ingenious like that.
Copy the new syslogd binary over the old (typically /sbin/syslogd)
BACKUP old version before doing this. As an admin, you should have already
known that, right? :)
NOTES
The reason why we do not simply just start the old syslogd with the [-f conffile]
switch is because if an attacker were to grab a 'ps' list of processes, the switch
you used will show up, telling the intruder that we are specifying a completely
different configuration file. Ultimately, defeating this purpose. I recommend
leaving the old /etc/syslog.conf file in tact as it completely throws off the
intruder. Also, keep local logs, as if the /var/log directory is empty, well,
use your imagination on why.
I would have to make the suggestion to use a different filename, other then
syslog.conf when specifying the new location. Understand that an intruder can
also just do a find / -name "syslog.conf" and locate another copy of the
configuration file.
STEP 3: Modify the (Real) Syslog Config File
For example, to forward ALL messages to a remote host use the following syslog.conf
entries:
# Sample syslogd entry to forward all messages to a remote host.
*.* @hostname
# To forward all kernel messages to a remote host the configuration file would be as
follows:
kern.* @hostname
# Remote Logging which includes local logging as well (for fault tolerance)
kern.crit @finlandia
kern.crit /dev/console
.. and so on.. repeat for all log files you wish to keep offsite. This includes third
party applications.
B. The Client (Remote Log Server)
With the client server, (or the machine accepting all of the remote logs), all
you have to do is launch syslogd with the -r switch, specifying remote logging
capabilities. Make sure that you remember to ADD: syslog 514/udp to the
/etc/services file as well, on the server, or it will not listen to the
specified port.
NOTES
If the network consists of different domains you don't have to complain
about logging fully qualified names instead of simple hostnames.
You may want to use the strip-domain feature -s of this server.
You can tell the syslogd to strip off several domains other than the one
the server is located in and only log simple hostnames.
ADD 'syslog 514/udp' to the /etc/services file.
If this entry is missing syslogd neither can receive remote messages nor send them
ADD '*.* @hostname' to the /etc/syslog.conf file. To forward all Kernel logging
messages to a remote host,
Make sure that if you do not specify an IP ADDRESS that you add the host
to the /etc/hosts file to avoid DNS RESOLUTION errors.
If the remote host is located in the same domain as the host, syslogd is running on,
only the simple hostname will be logged instead of the whole fqdn. (fully qualified
domain name)
C. Locking Down the Remote Log Server
One of the most crucial steps in this project is to fully lock down the machine that
is receiving all of the logs, or referred to as the remote log server.
1. Turn off (ALL SERVICES)
Remember, this is ONLY a log server, so all other services should NOT be turned on. Lets
walk through that process right now.
[root@myhost /etc]# cd /etc
[root@myhost /etc]# vi inetd.conf
1a. Turn off all inetd services
You will want to put a '#' comment in front of EVERY service listed in this file. The
inetd.conf file is IMHO one of the largest mistakes of the Linux operating system :).
But the ability to disable the inetd services makes it all better :)
You will want to comment out everything:
a. echo
b. discard
c. daytime
d. chargen
e. time
f. ftp
g. telnet
h. shell
i. login
j. exec
k. comsat
l. talk
m. ntalk
n. dtalk
o. pop-2
p. pop-3
q. imap
r. uucp
s. tftp
t. bootps
u. finger
v. cfinger
w. systat
x. netstat
y. auth
z. linuxconf
zz. swat
1b. DISABLE ALL RPC SERVICES
[root@myhost /etc]# cd /etc/rc.d
Anything listed in the rc. directories that start with a capital "S" means that the
service will start at boot time. In order to disable it from starting, you will want to
'mv to (a lowercase 's').
e.g. mv S11portmap s11portmap // This disables portmapper from starting at boot time.
You will want to do this with all unneeded services in this directory. These include:
nfslock, apmd, netfs, identd, autofs, portmap, atd, pcmcia, I would even disable the
inetd startup script as well, isdn, sendmail, gpm, httpd, vmware, xfs,linuxconf, local,
You are basically (other then the obvious services) are going to want to disable all of
these rc controlled services.You will want to go through all of the rc0.d-rc6.d
directories looking for these same files to disable.
2. Disable Accounts
[root@myhost /etc]# vi /etc/passwd
Go through your password file and remove practically every account that isn't being used.
I would suggest downloading the tool, /bin/noshell
(http://www.cerias.purdue.edu/coast/archive/data/categ50.html)and
use this with all
accounts in the password file that should not have remote login access.
This is a very informative program that provides admins additional information on who
is attempting to login with disabled accounts.
3. Install SSH</p>
Since the release of OpenSSH,giving the GNU opensource community free access to secure
shell, telnet should now be obsolete in a security centric environment. GET AWAY from
using telnet. With packet sniffers getting easier and easier to use (tools such as
dsniff, 'www.dsniff.org') allows even the biggest nitwit to snarf packet data. So
case-in-point, don't use telnet.
I bet you are so happy, because guess what? I'm going to walk you through the
installation of OpenSSH! (ohh goody)..
3a. Downloading OpenSSH
Point your web browser to www.openssh.com, as of this writing they are on version
2.1.1 which has support for both SSH1 and SSH2 protocols. The release date was
June 08, 2000. After downloading the source tarball, go ahead and do the following:
[root@myhost ]# gzip -d openssh-2.1.1p2.tar.gz
[root@myhost ]# tar -xvf openssh-2.1.1p2.tar
I can't realistically walk you through everything, so please save me some heartache,
as well as yourself, and just read the INSTALL file before continuing. I have no
idea what particular requirements your system has. I will however, walk you through
my install.
First you will need working installations of both Zlib and OpenSSL (yes their is an
OpenSSL project as well)
Zlib:
http://www.freesoftware.com/pub/infozip/zlib/
OpenSSL 0.9.5a or greater:
http://www.openssl.org/
Or, if you're even more lazy then the audience this paper was written for,
(God forbid), then grab the RPM's from the following link.
RPMs of OpenSSL are available at http://violet.ibs.com.au/openssh/files/support
Alternatively, Jim Knoble has written an excellent X11
passphrase requester. This is maintained separately at:
http://www.ntrnet.net/~jmknoble/software/x11-ssh-askpass/index.html
If you wish to build the GNOME passphrase requester, you will need the GNOME
libraries and headers.
GNOME:
http://www.gnome.org/
Also, grab GNUmake if you do not yet have it, OpenSSH has only been tested with this
program.
[root@myhost ]# ./configure && make && make install
This will install the binaries in /opt/{bin,lib,sbin}, but will place the config
files in /etc/ssh
If you are using PAM, you will need to manually install a PAM
control file as "/etc/pam.d/sshd" (or wherever your system
prefers to keep them). A generic PAM configuration is included as
"contrib/sshd.pam.generic", you may need to edit it before using it on
your system. If you are using a recent version of Redhat Linux, the
config file in contrib/redhat/sshd.pam should be more useful.
Failure to install a valid PAM file may result in an inability to
use password authentication.
I like to run configure with '--without-pam' to disable PAM support. PAM is
automatically detected and switched on if found. This is because I opt to use
RSA Authentication instead.
[root@myhost ]# tar -xvf zlib.tar
[root@myhost ]# cd zlib-1.1.3
[root@myhost zlib-1.1.3]# ./configure && make && make install
You shouldn't get any errors, if you do, sorry to do this, but refer to the README file.
What I like to do when compiling something and something goes wrong, I just grep a
keyword from the error on the provided README file(s) to see if there is anything
specifically mentioned about my error. (Just a suggestion)
To install OpenSSL, you will need:
- Perl 5
- an ANSI C compiler
- a supported Unix operating system
[root@myhost ]# tar -xvf openssl-0.9.5a.tar
[root@myhost ]# ./config && make && make test && make install
Now lets build OpenSSH!
[root@myhost ]# cd openssh-2.1.1p2
[root@myhost ]# ./configure --without-PAM && make && make install
NOTES
Because I opt for RSA authentication, I run configure with the --without-PAM
option as I am not a big fan of basic password authentication. With the advancements
in Cryptography and more secure forms of authentication, PAM and other password-type
authentication mechanisms IMHO will become obsolete within a few years.
When you have successfully compiled and installed OpenSSH, you will receive the following
information.
------ snip ------
Key generation complete.
Your identification has been saved in /usr/local/etc/ssh_host_key.
Your public key has been saved in /usr/local/etc/ssh_host_key.pub.
The key fingerprint is:
d5:74:83:d0:3f:c4:b4:d6:c5:39:1d:94:ee:9b:a8:61 root@soc1.priv.nuasis.com
Generating DSA parameter and key.
Your identification has been saved in /usr/local/etc/ssh_host_dsa_key.
Your public key has been saved in /usr/local/etc/ssh_host_dsa_key.pub.
The key fingerprint is:
ed:58:65:b9:8b:fe:05:81:c2:8c:06:c9:cb:ac:bb:e6 root@soc1.priv.nuasis.com
------ snap ------
3b. Configuring OpenSSH
Even though the default installation of OpenSSH should work, I would take a quick stab
at the ssh config files to see if you want to change anything.
Because we configured OpenSSH without PAM support, we want to enable RSA Authentication.
Uncomment it in the /usr/local/etc/ssh_config file respectively (this is for the
system-wide client configuration).
Then go ahead and edit the SSHD config file, /usr/local/etc/sshd_config
Lets go ahead and disable PermitRootLogin (do this by changing 'yes' to 'no'
Go ahead and save the file and quit.
After installation, a host key and DSA key should have been created for you. So lets
go ahead and move to the next step.
NOTES ON SSH v2
THE SSH PROTOCOL v2 SSH protocol version 2
In SSH v2, each host has a host-specific DSA key used to identify the host.
However, when the daemon starts, it does not generate a server key. Forward
security is provided through a Diffie-Hellman key agreement. This key agreement
results in a shared session key. The rest of the session is encrypted using a
symmetric cipher, currently Blowfish, 3DES or CAST128 in CBC mode or Arcfour.
The client selects the encryption algorithm to use from those offered by the
server. Additionally, session integrity is provided through a cryptographic message
authentication code (hmac-sha1 or hmac-md5).
Protocol version 2 provides a public key based user authentication method
(DSAAuthentication) and conventional password authentication.
AUTHORIZED_KEYS File Format
The $HOME/.ssh/authorized_keys file lists the RSA keys that are permitted
for RSA authentication in SSH protocols 1.3 and 1.5 Similarly, the
$HOME/.ssh/authorized_keys2 file lists the DSA keys that are permitted
for DSA authentication in SSH protocol 2.0. Each line of the file con-
tains one key (empty lines and lines starting with a `#' are ignored as
comments). Each line consists of the following fields, separated by
spaces: options, bits, exponent, modulus, comment. The options field is
optional; its presence is determined by whether the line starts with a
number or not (the option field never starts with a number). The bits,
exponent, modulus and comment fields give the RSA key; the comment field
is not used for anything (but may be convenient for the user to identify
the key).
Note that lines in this file are usually several hundred bytes long (be-
cause of the size of the RSA key modules). You don't want to type them
in; instead, copy the identity.pub file and edit it.
Important Files
Creating User Specific Key Files
With each user that you want to have access to the SSH server for, run
/usr/local/bin/ssh-keygen on the machine you are going to be ssh'ing from.
After running ssh-keygen, goto $HOME/.ssh/ and cp identity.pub -> authorized_keys
Key Location Overview
The system you want to login to (the SSH Server) is where you store the
$HOME/.ssh/authorized_keys file. This is the public key file for your user. The server
will require this to decrypt your private key on your host machine.
The system you are logging in FROM is where you store the
$HOME/.ssh/identity file. This is the private key file you generated.
NOTES
For those of you wishing to use SSH v2 (be careful, it's a pretty new version. I'd stick
with v1 to be stable and safe).
[root@myhost /usr/local/bin]# ./ssh-keygen -d
This -d flag will create the DSA Public Keyfile needed for DSA Authentication.
(When generating host keys, you can not specify a password)
Once you have generated your key files, go ahead and plop your sshd module file into the
/etc/pam.d/ directory. You are probably wondering, jeese, will Eric provide me with a
working sshd module? Well the answer is YES :) See below
---- snip -----
#%PAM-1.0
auth required /lib/security/pam_unix.so shadow nodelay
auth required /lib/security/pam_nologin.so
account required /lib/security/pam_unix.so
password required /lib/security/pam_cracklib.so
password required /lib/security/pam_unix.so shadow nullok use_authtok
session required /lib/security/pam_unix.so
session required /lib/security/pam_limits.so
----- snap -----
Remember, this is being used on my RH 6.2 box only. If it does not work, consult the
generic copy provided with the SSH distribution.
Overview
Now you should have ALL services turned off and SSH as the only means of connecting
remotely to the machine for administration purposes. The next step we want to do is
setup that infamous host-based Firewall!
4. The Firewall
Packet filtering doesn't really make sense on your logging box until you've got
your tcp stack vulnerable to some fragmentation issue, since you have got all
the services down anyway and you can control who is permitted to establish
secure shell connections to the machine using ssh access control features.
It might be reasonable though to filter out syslog port (514/udp) but it doesn't
really stop people from sending garbage to your syslog server, however it
does make
their job harder since it's udp which could be easily
spoofed. This is at least until you have a
multihomed machine where all logs
come to internal interface so you can safely
filter out everything that comes
from the outside.
However here are basic recommended filtering rules for machine. For this
purpose you can use either ipchains package if you're using Linux (last time
I checked ipfilter didn't compile cleanly there) or try to use ipfilter package
if you use something else (works perfectly on *BSD and Solaris machines at least).
For ipchains I recommend the following shell script.
-------------- snip --------------------
#!/bin/sh
PATH=/usr/sbin:/sbin:/bin:/usr/sbin
LOCAL_INTERFACE="192.168.1.1/32" # put your real IP address
LOCAL_NETWORK="192.168.1.0/24" # put your local net IP address/mask here
SSH_PERMITTED="192.168.1.2/32 192.168.2.3/32" # who allowed to ssh
SYSLOG_PERMITTED="192.168.1.5/32 192.168.5.2/32" # who allowed to log syslog messages
# deny everything
ipchains -P input DENY
ipchains -P output DENY
ipchains -P forward DENY
ipchains -F
#permit ssh
for ipaddr in $SSH_PERMITTED;
do
ipchains -A input -p tcp -s $ipaddr -d 0/0 22 -i $LOCAL_INTERFACE -j ACCEPT
done
# permit outgoing tcp
ipchains -A output -p tcp -i $LOCAL_INTERFACE -j ACCEPT
ipchains -A input -p tcp ! -y -i $LOCAL_INTERFACE -j ACCEPT
# permit syslog
for ipaddr in $SYSLOG_PERMITTED;
do
ipchains -A input -p udp -s $ipaddr -d $LOCAL_INTERFACE 514 -i $LOCAL_INTERFACE -j ACCEPT
done
# if you would like to log all the other connection attempts,
# uncommend these...
#ipchains -A input -p tcp -i $LOCAL_INTERFACE -l -j DENY
#ipchains -A input -p udp -i $LOCAL_INTERFACE -l -j DENY
#ipchains -A input -p icmp -i $LOCAL_INTERFACE -l -j DENY
----------- snap ---------------
For IP filter package, rules would look approximately like this:
----------- snip ---------------
# close everything on local interface
block in all on le0 from any to any
# pass secureshell
pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22
pass out on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22
# or you can replace these two rules with
#pass in on le0 proto tcp from 192.168.1.2/32 to 192.168.1.1/32 port = 22 keep state
#pass SYSLOG
pass in on le0 proto udp from 192.168.1.2/32 to 192.168.1.1/32 port = 514
----------- snap ----------------
5. Log Reporting
What good would a remote log server be if you weren't monitoring the log
files. I would recommend several different programs.
Logcheck - www.psionic.com
or
Swatch - www.swatch.org
Below we've included a simple shell script (Courtesy of Mr. Bill Pennington)
Which will archive the log files every day/hour/minute to an offsite
location (using scp of course).
#!/bin/bash
#Simple script to rotate the log files on a daily basis
#Bill Pennington 1/19/2000
#Set the date variable
date=`date +%m-%d-%Y`
#Rename the messages file
mv /var/log/messages /var/log/messages.$date
#HUP the syslog daemon so it writes to a new file
killall -HUP syslogd
#Compress the file
/bin/gzip /var/log/messages.$date
#Rename the secure file
mv /var/log/secure /var/log/secure.$date
#HUP the syslog daemon so it writes to a new file
killall -HUP syslogd
#Compress the file
/bin/gzip /var/log/secure.$date
#Rename mail file
mv /var/log/maillog /var/log/maillog.$date
#HUP the syslog daemon so it writes to a new file
killall -HUP syslogd
#Compress the file
/bin/gzip /var/log/maillog.$date
#Then scp them somewhere
6. Time
It is vital to ensure that the logserver has the correct time/date.
Get xntpd install it, then "ntpdate timeservername" at least
once a day, I do it twice a day. If you want to use your logs in any type of
legal proceeding you are going to need accurate time. You should be doing this
on all your servers of course.
7. Other Syslog Devices
There are many other devices including CISCO that can be setup to log to
syslog.
Having all your cool *nix boxes logging to a central log server is way
cool, but what about those other boxes on your network? To get a complete
view of your network and servers, setup all your devices to log to syslog.
Depending on the number of devices you have, you might want to dedicate a box
for each type of device or server. i.e. unix log server, netgear log
server,
NT log server...
Cisco
All Cisco devices support logging to syslog. The commands to set your Cisco
device to log to log messages to syslog vary from device to device. Some
examples are provided here but you should check your Cisco documentation for
more complete instructions.
Routers
In config mode:
logging <server ip address or name>
logging facility <facility> defaults to local7>
Pix Firewall
In config mode:
logging host <ip address of syslog server>
logging facility <facility> defaults to LOCAL4>
logging trap <level> from emergencies to debug, be careful with debug you will get a ton of traffic!
Switches
In config mode:
set logging server enable
set logging server <ip address of log server>
set logging session enable
set logging level all 1 default
Again you will want to look at your Cisco docs or Cisco's web site for
complete details but the above examples should get you started.
Windows NT
You can make NT servers forward events to central syslog servers as well.
There are several shareware/freeware packages available to accomplish this.
Search http://www.bhs.com for syslog to find one that is right for you.
8. Resources
Swatch - http://www.stanford.edu/~atkins/swatch/
OpenSSH - http://www.openssh.com
LogCheck - http://www.psionic.com/abacus/logcheck/
xntp and list of public NTP servers - http://www.eecis.udel.edu/~ntp/
Cisco - http://www.cisco.com
Windows NT software - http://www.bhs.com
Robert Graham's sniffing FAQ (hints to build `receive-only device' - http://www.robertgraham.com/pubs/sniffing-faq.html
IP Filter - TCP/IP filtering package - http://coombs.anu.edu.au/~avalon/
9. Contributions
Lance Spitzner (Renowned security whitepaper author of the 'Know Your Enemy' series. Thank you for all your support Lance. http://www.enteract.com/~lspitz
Martin Roesch (Creator of the popular Intrusion Detection System (SNORT)).
Had a blast with you at Defcon! Hope to see you next year.
http://www.snort.org
Bill Pennington (Thanks for all the help with SNORT and your contributions to
this paper!)
Good luck with everything!
Frank Heidt (@Stake Security Team - Enjoyed the cigar Frank! Hope to see you
at Defcon next year as well. Sorry I didn't end up implementing your syslog
design into this paper. I didn't think I could explain as perfectly as you
could;))
Fyodor Yarochkin (This paper wouldn't have been possible without you bud! Thanks!)
10. Shouts
Lamagra, Safety, #RootHat, zen-parse, lockdown, Crimson, Invader, LoA,
Fyodor, Faisal Jawdat, Art Stine, Vince Pereira, Ted Barban, Malverian,
Frank Heidt, Bill Pennington, Marty, Lance.. and all the rest.
"TIMMMMMY!"
11. Contact Information
Loki
loki@fatelabs.com
Loki is a 7 year member of the network security industry, Eric currently provides
security consulting to many Fortune 500 to Fortune 50 Corporations across the
US. Eric also gives keynote speakings on information security at numerous
different E-Commerce and E-Banking seminars. Eric has deployed several VPN
Extranets from British Columbia to the United States, as well as plays
active roles in contributions to many security-related mailing lists. Eric
enjoys creating custom attack signatures using packet analyzers for
Intrusion Detection Systems such as SNORT, and holds strong interest in
Intrusion Detection technology.