Wednesday 11 May 2011

Server Monitoring

If the server load is high then do the following tricks:

1) top or top -n1
check for the server load and watch for process

2) free -m
it will display the memory status
# vmstat : (virtual memory status)
# mymem : (which services are consuming memory)

3)if a perticular user is causing a high server laod then use “ps U username”: it will display the process that are running by the user . Then you can take action against that user.

4) netstat -an |grep :80 |wc -l :
Show how many active connections there are to apache (httpd runs on port 80)

5) netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort –n
it will display the no of connections form a certain ip

6)if there is MYSQL taking cpu usage then check for mysql status
# watch mysqladmin proc
# mysqladmin -u root processlist
check for the databases and suspend that user if any database is using high resources.

7)check /tmp if there is any process taking high resources : chown to root for that process and chmod 000 also if .sh process is running then chown it t0 root:root

8)if exim is taking high resources then check for exim logs : use this command :
replace :blackhole: : fail: — /etc/valiases/*

w : Current users

exim -bp | exiqsumm
ps aux | head -1;ps aux –no-headers| sort -rn +3 | head :

Use below mentioned command to get top memory consuming processes
ps aux | head -1;ps aux –no-headers | sort -rn +2
ps aux | head -1;ps aux –no-headers | sort -rn +2

———————————————————————————————————-

If Exim causes
exim –bpc
shows the total no of email in qmail

eximstats -nr -ne /var/log/exim_mainlog
Total mail server report

pidof exim
shows no of exim pids running

exim -bpr | grep frozen | wc -l
Shows no of frozen emails

exiqgrep -z -i | xargs exim –Mrm
exim -bp | exiqgrep -i | xargs exim -Mrm
it deletes the FROZEN mails from the server

tail -f /var/log/exim_mainlog | grep public_html
check for spamming if anybody is using php script for sending mail through public_html

tail -f /var/log/exim_mainlog | grep /tmp
Used for checking for who is spamming through the /tmp

tail -3000 /var/log/exim_mainlog |grep ‘rejected RCPT’ |awk ‘{print$4}’|awk -F\[ '{print $2} '|awk -F\] ‘{print $1} ‘|sort | uniq -c | sort -k 1 -nr | head -n 5
It will display the IP and no of tries done by the IP to send mail but rejected by the server.

netstat -plan|grep :25|awk {‘print $5′}|cut -d: -f 1|sort|uniq -c|sort -nk 1
shows the connections from a certain ip to the SMTP server

exim -bp | exiqsumm | more
It shows the domain name and the no of emails sent by that domain

If spamming from outside domain then you can block that domain or email id on the server
pico /etc/antivirus.exim
Add the following lines:
if $header_from: contains “name@domain.com”
then
seen finish
endif

eximstats -nr -ne /var/log/exim_mainlog

Catching spammer

exim -bp | exiqsumm | more

exim -bpr | grep “<*@*>” | awk ‘{print $4}’|grep -v “<>” | sort | uniq -c | sort -n
That will show you the maximum no of email currently in the mail queue have from or to the email address in the mail queue with exact figure.

exim -bpr | grep “<*@*>” | awk ‘{print $4}’|grep -v “<>” |awk -F “@” ‘{ print $2}’ | sort | uniq -c | sort -n
That will show you the maximum no of email currently in the mail queue have for the domain or from the domain with number.

Check if any php script is causing the mass mailing with
cd /var/spool/exim/input
egrep “X-PHP-Script” * -R
Just cat the ID that you get and you will be able to check which script is here causing problem for you.

To Remove particular email account email
exim -bpr |grep “ragnarockradio.org”|awk {‘print $3′}|xargs exim -Mrm

If Mysql causes :
mysqladmin -u root processlist
mysqladmin version
watch mysqladmin proc

If Apache causes :
netstat -ntu | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort –n
netstat -an |grep :80 |wc –l
netstat -n | grep :80 | wc -l;uptime ; netstat -n | wc –l
netstat –tupl
netstat -tupl

Other Commands

pidof php
shows the PIDs for php

history | netstat
shows no of connection details

lsof -p pid
shows the details of pid

netstat -na |grep :80 |sort

Use below mentioned command to get top memory consuming processes”

ps aux | head -1;ps aux –no-headers| sort -rn +3 | head

Use below command to get top cpu consuming processes:

ps aux | head -1;ps aux –no-headers | sort -rn +2 |more

You can check if any backup is going on, run the following commands:
# ps aux | grep “pkg”
# ps aux | grep “gzip”
# ps aux | grep “backup”
If any backup process is going on, kill that process.

We can trace the user responsible for high web server resource usage by the folowing command
cat /etc/httpd/logs/access_log | grep mp3
cat /etc/httpd/logs/access_log | grep rar
cat /etc/httpd/logs/access_log | grep wav etc

cat /etc/httpd/logs/access_log | grep 408
Can be used to check for DDOS attacks on the server.

cat /etc/httpd/logs/access_log | grep rar

Port scanning :
root@ [/tmp]# nmap localhost

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2007-10-04 13:22 MYT
Interesting ports on localhost (127.0.0.1):
Not shown: 1664 closed ports
PORT STATE SERVICE
1/tcp open tcpmux
21/tcp open ftp
25/tcp open smtp
53/tcp open domain
80/tcp open http
110/tcp open pop3
111/tcp open rpcbind
143/tcp open imap
443/tcp open https
465/tcp open smtps
631/tcp open ipp
783/tcp open spamassassin
953/tcp open rndc
993/tcp open imaps
995/tcp open pop3s
3306/tcp open mysql

Nmap finished: 1 IP address (1 host up) scanned in 0.125 seconds
root@ [/tmp]#

netstat -anp |grep :3306

DDOS ATTACK :

ps -aux|grep HTTP|wc –l : It will show you no of http connections to the server

netstat -lpn|grep :80 |awk ‘{print $5}’|sort : It will helpful to check the no of connections from a certain ip

Check for the ips and block them with firewalls as apf/csf/iptables

For iptables : iptables -A INPUT -s -j DROP
For apf : apf –d ip_address
For CSF : csf -d ip_adrress

If load due to SPAMD:

Issue
The load issues on a cpanel 11 box (later versions too) because of spamd are quite common, sometimes they may take more than 90% of the CPU. Here are some ways to resolve it.
[edit] Solution
We can use the following steps to resolve the issue in a considerable manner.

1. Edit the /etc/mail/spamassassin/local.cf file

vi /etc/mail/spamassassin/local.cf

then add the following lines to it

a) use_bayes 0 (this will disable some spamassassin functionalities)
b) lock_method flock

(Uncomment these options, if they are commented)

2) Run scripts2/autofixer/spamd_dbm_fix

a) Log into WHM and use another URL as “http://your server IP:2086//scripts2/autofixer”
then type the script name “spamd_dbm_fix” and run

b) Or you can also do the same from server back end as follows.

# wget http://httpupdate.cpanel.net/autofixer/spamd_dbm_fix
# chmod 755 spamd_dbm_fix
# ./spamd_dbm_fix

3) Run the following lines of codes

# rm -f /home/*/.spamassassin/__db.bayes_toks.new
# rm -f /home/*/.spamassassin/*.lock

NOTE : If the load issue is related to a particular user then there is no need for running this,
you can simply enter into the ‘.spamassassin/’ directory for that user and delete the
files ‘__db.bayes_toks.new’ and *.lock.

4) Restart spam assassin (restart Exim)

5) Hope this will help you in such a situation.

: memory usage serverwide.
ps aux | awk ‘{print $4″\t”$11}’ | sort | uniq -c | awk ‘{print $2″ “$1″ “$3}’ | sort –nr

[root@ ~]# mymem
Memory Usage
————
Current: 326.344 MB
Allocated: 492.012 MB
Guaranteed: 512 MB
Burstable: 768 MB

====================

5.1 1 spamd
0.7 1 spamd
0.6 1 /usr/bin/spamd
0.4 1 /usr/sbin/mysqld
0.2 1 cpsrvd
0.1 4 /etc/authlib/authProg
0.1 1 eximstats
0.1 1 cpdavd
0.1 1 cpbandwd
0.1 1 cpanellogd
0.1 1 cPhulkd
0.1 1 /usr/local/cpanel/bin/leechprotect
0.0 6 /usr/libexec/courier-authlib/authdaemond
0.0 4 /usr/sbin/courierlogger
0.0 4 -bash
0.0 3 sshd:
0.0 3 /usr/sbin/exim
0.0 3 /usr/lib/courier-imap/libexec/couriertcpd
0.0 2 sort
0.0 2 awk
0.0 11 /usr/local/apache/bin/httpd
0.0 1 xinetd
0.0 1 uniq
0.0 1 syslogd
0.0 1 pure-ftpd
0.0 1 ps
0.0 1 init
0.0 1 crond
0.0 1 chkservd
0.0 1 antirelayd
0.0 1 /usr/sbin/sshd
0.0 1 /usr/sbin/saslauthd
0.0 1 /usr/sbin/pure-authd
0.0 1 /usr/sbin/portsentry
0.0 1 /usr/sbin/named
0.0 1 /etc/authlib/authProg
0.0 1 /bin/sh
%MEM 1 COMMAND

====================================================

kill nobody process
ps aux | grep nobody | awk ‘{print $2}’ | xargs kill -9

To see only the memory resources occupied by each category of processes, such as Apache httpd, MySQL mysqld or Java, use the following command:

ps aux | awk ‘{print $4″\t”$11}’ | sort | uniq -c | awk ‘{print $2″ “$1″ “$3}’ | sort -nr
====================================================

ps -eo pid,%cpu,vsz,args,wchan

Shows every process, their pid, % of cpu, memory size, name, and what syscall they are currently executing. Nifty.

whereis locates source/binary and manuals sections for specified files. The supplied names are first stripped of leading pathname components and any (single) trailing
extension of the form .ext, for example, .c. Prefixes of s. resulting from use of source code control are also dealt with. whereis then attempts to locate the desired program in
a list of standard Linux places. For example find out where is ls command?
$ whereis ls

cd /var/spool/exim/input
egrep “X-PHP-Script” * -R

For netstat
netstat -nap | grep :110 | awk ‘{print $5}’ | cut -d”:” -f1 | sort | uniq -c | sort -nr | head
netstat -nap | grep :25 | awk ‘{print $5}’ | cut -d”:” -f1 | sort | uniq -c | sort -nr | head
netstat -n | grep :80 | wc -l;uptime ; netstat -n | wc –l
netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n
netstat -nap | grep :80 | awk ‘{print $5}’ | cut -d”:” -f1 | sort | uniq -c | sort -nr | head

root@ [~]# tail -f /var/log/slow.log
# tail -500 /var/log/mysql-slow-queries.log

# How to find Top 10 memomry & cpu consuming processes
ps -eo pcpu,%mem,pid,user,args | sort -k 1 -r | head -10

No comments:

Post a Comment