XRT HOWTO --- Debug HK at SAO

2008-Nov-12, K. Reeves
Updated 2010-Jan-06, K. Reeves

If the XRT HouseKeeping (HK) appears to have stopped working, or is giving strange values, this process will help you evaluate if the problem is with the housekeeping software at SAO.

Overview

There are cronjobs at SAO that parse ascii files created by xskim and plot the hk values on the SAO HK website. These cronjobs run about once per hour. Some of the files are only accessible to those with access to SAO computers. A description of the HK pipeline process is given below.

Housekeeping pipeline

On the machine "norway" at ISAS, data are extracted from the data distributer and stored in .DAT format. Details on this process are in the document SolarB_hk_writeup.pdf.
The .DAT files are copied over to SAO via an rsynch job on the SAO machine "kurasuta" (user: xrt_data).
On the machine "machida" at SAO, under the user xrt_data, the housekeeping data stream is processed via the script /data/solarb/XRT/hk_pro/make_hk_plots_merge.cron. This script runs /data/solarb/XRT/hk_pro/xdump_install/merge_bands_sao.sh, which creates ascii files that contain the housekeeping data.
The ascii files are kept at SAO in the directory /archive/hinode/xrt/hk/yyyy/mm/dd/ and are named things like "xrt_sts1_yyyymmddt000000.txt.merge."
The files are then processed by another script, /data/solarb/XRT/hk_pro/parse_file.sh, which calls the perl script called parse_hk_ascii.pl that parses them out into two-column text files including the time and the HK value. These files can be found in /data/solarb/XRT/hk_cron/yyyy/mm/dd/. They have names like TMP04C_yyyymmdd_hh.txt. These are the files that are being plotted on the SAO website.
The plots are made (via the same cron job on machida that makes the two column text files) using a batch idl process. The main plotting routine is plot_all_hk_vs2.pro, and it can be found in /data/solarb/XRT/hk_pro/programs/.
The script /data/solarb/XRT/hk_pro/get_yesterdays_data.cron runs once a day, at 4:45am local time. This script processes the ascii files and the two-column text files from the previous day in order to avoid gaps in the HK that appear around midnight.

Procedure for Checking for Anomalous HK Values

1. If you have access to the SAO machines, you can identify problems by looking first at the two column text files on the SAO machines at /data/solarb/XRT/hk_cron/yyyy/mm/dd/. Looking at the ones from 11/11/2008, for example, P_P28IV__20081111_00.txt , you see:

UTC_Time P_P28IV_
[...]
2008/11/11_00:57:22.2 29.1797
2008/11/11_00:57:24.2 29.1797
2008/11/11_00:57:26.2 29.1797
2008/11/11_00:57:30.2 29.1797
2008/11/11_03:43:43.5 :03:21:30:07.747
2008/11/11_03:43:45.5 :03:21:30:09.748
2008/11/11_03:43:47.5 :03:21:30:11.748
2008/11/11_03:43:49.5 :03:21:30:13.749
2008/11/11_03:43:51.5 :03:21:30:15.75
2008/11/11_03:43:53.5 :03:21:30:17.75
2008/11/11_03:43:55.5 :03:21:30:19.751
2008/11/11_03:43:57.5 :03:21:30:21.751
2008/11/11_03:43:59.5 :03:21:30:23.752
2008/11/11_03:44:01.5 :03:21:30:25.753
[...]

The first few values are fine (29.197), but subsequent values look like times and indicate a problem with the perl script processing.

2. You should also check the master ascii file (in /archive/hinode/xrt/hk/yyyy/mm/dd/ at SAO) for that day to determine if the strange values are contained in the ascii file.

Creative use of the grep command is your friend when checking the ascii file - these files are huge. However, say you were worried about TMP11. You can do a command that looks something like this:

prompt: grep "Temp 11" xrt_sts1_20081111t000000.txt.merge

And you will get a list of all the TMP 11 values :

Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
et cetera.....

3. If anomalous two-column files are found, but there are not anomalous values in the ascii file, proceed to the next item to fix the two-column text files. If there are anomalous values in the ascii files, send email to xrt_manager@head.cfa.harvard.edu.

Procedure for Remaking Two-Column Files

Sometimes the perl script (parse_hk_ascii.pl) fails if two instances are run at the same time, producing improper values in the housekeeping stream. A symptom of this problem is that values will get placed in the wrong two-column file, i.e. values for TMP01 may be placed in the two-column text file that is supposed to contain TMP07 values. There is a script for re-parsing the ascii files in this case.

1) Remove the problematic two-column text files located in /data/solarb/XRT/hk_cron/yyyy/mm/dd

2) prompt: /data/solarb/XRT/hk_pro/parse_file_date.sh merge yyyy/mm/dd/

3) Do NOT run two instances of /data/solarb/XRT/hk_pro/parse_file_date.sh on the same directory at the same time. The perl script will fail. If the files you are trying to fix are recent (today or yesterday), you should turn off the /data/solarb/XRT/hk_pro/make_hk_plots_merge.cron cron job (on machida, user xrt_data) to avoid running conflicting instances of the perl script.

This procedure will properly reprocess the two column text files for the date entered.