XRT HOWTO --- Restart HK at ISAS
K. Glotfelty, D. McKenzie, K. Reeves, M. Weber; 2007-Aug-16
If the XRT HouseKeeping (HK) appears to have stopped working, one
possibility is that the HK processes at ISAS have hung. This HOWTO
explains how to restart those processes.
NOTE (added 2008-11-10 by K. Reeves): This procedure only restarts
the HK is cron jobs are stuck at ISAS. The housekeeping software has
been reconfigured, so that this is much less likely. If the
housekeeping is not updating, it is probably due to the cron jobs at
SAO. Contact email@example.com if this appears to be
There are cronjobs at ISAS that run xskim and related
software routines in order to collect the HK information from the
telemetry pipeline. These cronjobs run about once per hour. There
is another cronjob at SAO that pulls that info to the SAO XRT HK
website, and that cronjob also runs about once per hour. Therefore,
in order to fix the hung HK jobs, you only have to kill the processes.
The cronjobs will automatically restart everything within an hour.
It may take up to about 2 hours for the new HK data to appear on
the SAO webpage. However, it is also possible that the new xskim
and mergeband processes will start immediately, and you do not
want to kill the new ones.
- Log on to xrtco workstation. Check the directory
- If there is recently created
data in this directory, the cron jobs at ISAS are working fine, and
the problem is at SAO. DO NOT follow the steps below - contact
- If there is no recently created data, the cron jobs at ISAS have
stopped; continue with the steps below.
- Log into ISAS workstation "norway". You will have to log in to the gateway machines (ssh1, ssh2 or ssh3) as user sbusxrt first.
- Look for the mergebands and xskim processes,
and note their process ID numbers:
- The mergebands processs look like:
[sbusxrt@pg2~]$ ps auwx | grep mergebands
sbusxrt 22521 0.0 0.0 2068 956 ? S 00:18 0:00 sh
- The xskim processes look like:
[sbusxrt@pg2~]$ ps auwx | grep xskim
sbusxrt 22734 0.0 0.0 3148 1608 ? S 00:26 0:08
/home/sbusxrt/install/xskim -f 08/09/2007 00:00:00 -t +23:59:59
-l -a sval -b x -v 0 -C mworking
- Kill the xskim job. (Using the ID number from this
- [sbusxrt@pg2~]$ kill -9 22734
- Wait until after any mergeband jobs are done.
- Often the new xskim and mergebandjobs will
begin right away. This is okay. But if it happens, you
should verify that the new xskim and mergebands
processes have new ID numbers.
(Use ps auwx and grep to check ID numbers.)
- Verify that the file ~/mworking is removed.
- You should NOT remove the mworking file until the
old merge jobs are finished. In fact, it should
remove itself automatically when the merge scripts
are done. This file tells the cronjob not to start another
job if it's working on a current one (the cron runs every
hour, and sometimes xskim takes longer than that to
- Send email (To: firstname.lastname@example.org; Cc:
email@example.com) announcing the time that
you killed the HK processes, so that others do not also try to
fix the problem. In the email, also remind people that it may
take up to about 2 hours before the new HK info is transmitted
to the SAO HK website.