XRT HOWTO --- Debug HK at SAO

2008-Nov-12, K. Reeves
Updated 2010-Jan-06, K. Reeves


If the XRT HouseKeeping (HK) appears to have stopped working, or is giving strange values, this process will help you evaluate if the problem is with the housekeeping software at SAO.

Overview

There are cronjobs at SAO that parse ascii files created by xskim and plot the hk values on the SAO HK website. These cronjobs run about once per hour. Some of the files are only accessible to those with access to SAO computers. A description of the HK pipeline process is given below.

Housekeeping pipeline

Procedure for Checking for Anomalous HK Values

1. If you have access to the SAO machines, you can identify problems by looking first at the two column text files on the SAO machines at /data/solarb/XRT/hk_cron/yyyy/mm/dd/. Looking at the ones from 11/11/2008, for example, P_P28IV__20081111_00.txt , you see:

UTC_Time P_P28IV_
[...]
2008/11/11_00:57:22.2 29.1797
2008/11/11_00:57:24.2 29.1797
2008/11/11_00:57:26.2 29.1797
2008/11/11_00:57:30.2 29.1797
2008/11/11_03:43:43.5 :03:21:30:07.747
2008/11/11_03:43:45.5 :03:21:30:09.748
2008/11/11_03:43:47.5 :03:21:30:11.748
2008/11/11_03:43:49.5 :03:21:30:13.749
2008/11/11_03:43:51.5 :03:21:30:15.75
2008/11/11_03:43:53.5 :03:21:30:17.75
2008/11/11_03:43:55.5 :03:21:30:19.751
2008/11/11_03:43:57.5 :03:21:30:21.751
2008/11/11_03:43:59.5 :03:21:30:23.752
2008/11/11_03:44:01.5 :03:21:30:25.753
[...]

The first few values are fine (29.197), but subsequent values look like times and indicate a problem with the perl script processing.

2. You should also check the master ascii file (in /archive/hinode/xrt/hk/yyyy/mm/dd/ at SAO) for that day to determine if the strange values are contained in the ascii file.

Creative use of the grep command is your friend when checking the ascii file - these files are huge. However, say you were worried about TMP11. You can do a command that looks something like this:

prompt: grep "Temp 11" xrt_sts1_20081111t000000.txt.merge

And you will get a list of all the TMP 11 values :

Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
Temp 11 (1-5): 46.571442 C (Mirror Support -X)
et cetera.....


3. If anomalous two-column files are found, but there are not anomalous values in the ascii file, proceed to the next item to fix the two-column text files. If there are anomalous values in the ascii files, send email to xrt_manager@head.cfa.harvard.edu.

Procedure for Remaking Two-Column Files

Sometimes the perl script (parse_hk_ascii.pl) fails if two instances are run at the same time, producing improper values in the housekeeping stream. A symptom of this problem is that values will get placed in the wrong two-column file, i.e. values for TMP01 may be placed in the two-column text file that is supposed to contain TMP07 values. There is a script for re-parsing the ascii files in this case.

1) Remove the problematic two-column text files located in /data/solarb/XRT/hk_cron/yyyy/mm/dd

2) prompt: /data/solarb/XRT/hk_pro/parse_file_date.sh merge yyyy/mm/dd/

3) Do NOT run two instances of /data/solarb/XRT/hk_pro/parse_file_date.sh on the same directory at the same time. The perl script will fail. If the files you are trying to fix are recent (today or yesterday), you should turn off the /data/solarb/XRT/hk_pro/make_hk_plots_merge.cron cron job (on machida, user xrt_data) to avoid running conflicting instances of the perl script.

This procedure will properly reprocess the two column text files for the date entered.