Wednesday, June 11, 2008

Not so familair log entries from sendmail

For a sysadmin or an application support person, log files are the key input to all they do. Mastering the skill of understanding what each log entry in a log file means or trying to tell you, is something that separate a super admin from a Joe Schmoe admin.

Some applications provide logs in a meaningful uniform format, while others just spits out strings to logs. From my experience, most open-source applications running on Unix flavors which uses syslog facility are excellent in this regard. On the other hand Most windows applications that logs via event log facility are pathetic. Output from tomcat on web servers are just plain funny, they don't even have a time stamp.

OK, back to the title of this post. I made some changes to a sendmail instance running on one of my boxes. The goal was to
  1. Stop processing the queue if the load average on the box rises above, say 20.
  2. Delay the incoming messages if the load average rises above, say 30.
sendmail is rich in this regard. I could use '-OQueueLA=20' to prevent the queue processors from processing the queue if/when the load grows above 20. I can use '-ODelayLA=30' on my sendmail listeners to delay in incoming volume by inserting a one second sleep when the load rises above 25. This is all good stuff. But I have to show my boss that these changes are actually working. Few lines from sendmail log will be the best. A bit of digging uncovered the gem I was looking for.

When the load average rise past 20, sendmail stops processing the queue and logs:

runqueue: Aborting queue run: load average too high

When load average rise past 30, sendmail delays incoming e-mails and logs:

delaying connections on daemon : load average=32 >= 30

Additionally, I think , if the load average goes above during an SMTP session, sendmail delay the commands of the session and logs:

delaying=EHLO, load average=32 >= 30
delaying=MAIL, load average=32 >= 30
delaying=DATA, load average=32 >= 30


Sweet!

No comments: