Sunday, November 27, 2011

Cron : ! MAXRUN (100) procs reached

Today I got a ticket for scheduled crons are not executing.

Let's see what is the issue.

Logged into the server and checked whether cron is running or not [ps -ef | grep cron ]. Yep its working.

1. The issue may be because of hung cron process. Find out which commands executed from cron is hung

 There was no hung process


2.Then what will be the issue. let's see the log


Log location for cron in HP-UX

# pwd
/var/adm/cron

root @ ux-memoirs[/var/adm/cron]
# tail log
! MAXRUN (100) procs reached Sat Nov 26 22:21:14 EST 2011
! rescheduling a cron job Sat Nov 26 22:21:14 EST 2011


Hmmm... this is the issue. What's this MAXRUN ? Where its defined.

Googled  and found it. its /var/adm/cron/queuedefs file.


"The queuedefs file describes the characteristics of the queues managed by cron"

Format  in the file will be :- q. [njob j][nice n][nwait w]

where,


q
The name of the queue, such that a is the default queue for jobs started by at (see at(1)), b is the queue for jobs started by batch (see at(1)), and c is the queue for jobs run from a crontab file (see crontab(1)). Queue names d through y designate user-defined queues.
njob
The maximum number of jobs that can be run simultaneously in that queue. Although any number can be specified here, cron (see cron(1M)) by default limits the number of jobs that can be run on all the queues to 100. This limitation can be removed by setting the variable DISABLE_MAXJOB_LIMIT to 1 in the /etc/default/cron file.
nice
The nice value to give to all jobs in that queue that are not run with a user ID of super-user (see nice(1)). The default value is 2.
nwait
The number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in that job's queue, or because more than 100 jobs were running in all the queues (see njob above).

Instead of limiting using DISABLE_MAXJOB_LIMIT I decided to limit cron to 200 jobs.

Here we go :-

Default queuedefs file will be like this
root @ ux-memoirs[/var/adm/cron]
# cat queuedefs
a.4j1n
b.2j2n90w

After modification it will be like this
root @ ux-memoirs[/var/adm/cron]
# cat queuedefs
a.4j1n
b.2j2n90w
c.200j

This means  
c          Cron
200j    Max No of Jobs can be processed at a time is 200

I have not put nice value and wait seconds.


Now restart cron
# /sbin/init.d/cron stop
cron stopped

# /sbin/init.d/cron start
cron started

let's see the logs..
# tail log
>  root 4075 c Sat Nov 26 22:35:00 EST 2011
>  CMD: /opt/xx/xx/bin/start.sh >/dev/null 2>&1 &
>  root 4076 c Sat Nov 26 22:35:00 EST 2011

Wow !! its working .