Monitoring User Activity with psacct or acct

by Mike on January 15, 2010 · 1 comment

in Server Management

Monitoring with psacct
An excellent program for monitoring users and applications is psacct.  This program will work in the background of your system recording what all users are doing on your system as well as the resources that are being consumed.  I originally started using this program when a client asked for a way to monitor his developers and what they were doing on the servers that they were working on.  psacct provided an excellent solution which told not only how long the developers were on the box but what commands they were using and how much of the resources they were using.  Another great application for this program is to use it to evaluate the resources that applications are consumed on your server.  So for example you could see the total resources consumed by apache, mysql, etc.  Whether you are a CentOS Administrator or an Ubuntu Server Administrator or a home user wanting to track activity on your machine, this is a good program to explore.

Install Process Accounting

yum install psacct
on Ubuntu
sudo apt-get install acct

Start Process Accounting

/etc/init.d/psacct start
Starting process accounting:                               [  OK  ]

on Ubuntu
sudo service acct start

Connect Time
The connect time in hours is based on logins and logouts.  The ac command provides a total.

ac
total      256.64

Accounting By Day
The system’s default login accounting file is /var/log/wtmp.

ac -d
Nov  1  total        8.78
Nov  3  total        6.15
Nov  6  total       13.68
Nov  7  total       15.50
Nov  8  total       10.54
Nov  9  total       10.56
Nov 10  total        8.41
Nov 13  total        8.66
—cut—
Today   total       19.33

Time Totals for Users

ac -p
root                                 5.83
mike                               251.29
total      257.13

Commands of Users
You can search out the commands of users with the lastcomm command which prints out the previously executed commands.

Explanation of Output:

Process  Flag    Username    Terminal   Time
ping         S       mike              pts/5      0.00 secs Thu Nov 30 13:39

Flags:
S -     executed as super-user
F -     executed after  but not following exec
D -     terminated with core file
X -     terminated with signal SIGTERM

lastcomm mike
bash                  X     mike     __         0.04 secs Thu Nov 30 12:49
bash                  X     mike     __         0.04 secs Thu Nov 30 12:57
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:39
nmap                        mike     pts/4      4.01 secs Thu Nov 30 13:38
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:37
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:34
ping                  S     mike     pts/5      0.00 secs Thu Nov 30 13:34
bash                  X     mike     __         0.03 secs Thu Nov 30 13:28
bash                   F    mike     pts/6      0.00 secs Thu Nov 30 13:28
consoletype             mike     pts/6      0.00 secs Thu Nov 30 13:28
bash                   F        mike     pts/6      0.00 secs Thu Nov 30 13:28
id                                  mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                            mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                        mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28
egrep                       mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28
dircolors                   mike     pts/6      0.00 secs Thu Nov 30 13:28
bash               F        mike     pts/6      0.00 secs Thu Nov 30 13:28

Search Logs for Commands
Using the lastcomm command you will be able to view each use of an individual command.

lastcomm grep
grep                    mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                    mike     pts/6      0.00 secs Thu Nov 30 13:28
grep                    mike     pts/5      0.00 secs Thu Nov 30 12:57
grep                    mike     pts/5      0.00 secs Thu Nov 30 12:57

lastcomm grep
grep                    mike     pts/7      0.00 secs Wed Mar 14 06:07
grep                    mike     pts/7      0.00 secs Wed Mar 14 06:07
grep                    root     pts/7      0.00 secs Wed Mar 14 05:39
grep                    root     pts/7      0.00 secs Wed Mar 14 05:39
grep                    root     pts/2      0.00 secs Wed Mar 14 05:16

Print Summary

The sa command will print a summary of commands that were executed.  It will also condense the information into a summary file called savacct which contains the number of times that the command was executed.  The useracct file keeps a summary of the commands by user.

Output Fields
cpu   -  sum of system and user time in cpu minutes
re    -  actual time in minutes
k     -  cpu-time averaged core usage, in 1k units
k*sec -  cpu storage integral (kilo-core seconds)
u     -  user cpu time in cpu minutes
s     -  system time in cpu minutes

/usr/sbin/sa

Print User Information
Use the -u option to provide information on individual users.

/usr/sbin/sa -u

/usr/sbin/sa -u
root       0.00 cpu      598k mem accton
root       0.00 cpu     1081k mem initlog
root       0.00 cpu      920k mem initlog
root       0.00 cpu     1172k mem touch
root       0.00 cpu     1402k mem psacct
mike       0.01 cpu     7282k mem kdeinit          *
mike       0.00 cpu     6232k mem gnome-panel      *
mike       0.02 cpu     4848k mem gnome-terminal

Display Number of Processes
An increase in these fields indicates a problem.  This prints the number of processes and the number of CPU minutes.  If these numbers continue to increase it is time to look into what is happening.

/usr/sbin/sa -m
195         220.31re           0.09cp     2220k
mike                                  65          198.37re           0.08cp     2135k
root                                  88          21.86re           0.00cp     1084k
postgres                              40          0.09re           0.00cp     4879k
smmsp                                 2           0.00re           0.00cp     1827k

Display All Names
This option will show each of the programs on your server so you may evaluate, real time, memory usage and which programs are running.

sa -a
221      83.36re       0.01cp     1414k
1       0.01re       0.00cp     1471k   rpmq
7       0.33re       0.00cp     2465k   sendmail*
1      40.78re       0.00cp     1844k   sshd
37       0.00re       0.00cp      964k   bash*
32       0.00re       0.00cp      604k   tmpwatch
27       0.00re       0.00cp     4984k   postmaster*
26       0.00re       0.00cp     1116k   df
15       0.00re       0.00cp      959k   id
11       0.00re       0.00cp      709k   egrep
8       0.00re       0.00cp      636k   sa
7       0.00re       0.00cp      817k   grep
6       0.00re       0.00cp      562k   ac
5       0.01re       0.00cp      789k   awk
3       0.41re       0.00cp     1219k   crond*
3       0.40re       0.00cp      674k   run-parts
3       0.00re       0.00cp      774k   dircolors
3       0.00re       0.00cp      673k   consoletype
2      40.98re       0.00cp     1344k   bash
2       0.14re       0.00cp     1628k   sshd*
2       0.00re       0.00cp      914k   logrotate

Sort by Percentage
This will provide the highest percentage users first.

sa -c
232  100.00%      83.36re  100.00%       0.01cp  100.00%     1482k
14    6.03%      41.09re   49.29%       0.01cp   58.82%      957k   ***other*
7    3.02%       0.33re    0.39%       0.00cp   41.18%     2465k   sendmail*
37   15.95%       0.00re    0.00%       0.00cp    0.00%      964k   bash*
32   13.79%       0.00re    0.00%       0.00cp    0.00%     4984k   postmaster*
32   13.79%       0.00re    0.00%       0.00cp    0.00%      604k   tmpwatch
31   13.36%       0.00re    0.00%       0.00cp    0.00%     1122k   df
15    6.47%       0.00re    0.00%       0.00cp    0.00%      959k   id
11    4.74%       0.00re    0.00%       0.00cp    0.00%      709k   egrep

Previous post:

Next post: