.:scripts:home

Hierarchical Script Structure

The graphic below shows the structure of the process_all scripts in the ADPAA library. Each script calls the ones beneath it (for example: the process_all_dir script will call the process_all_polcast3 script, which in turn calls the process_all script, and etc.).

Master Processing Scripts

process_all_dir

This simply goes through every PostProcessing directory and processes the *.sea file. Edits are also applied. To view a list of command line arguments that will work with process_all_dir, just enter process_all_dir into the command line

When running the process_all_dir script it is a requirement to be in the general time period directory. If the script is ran in any other directory nothing will happen.

The following is the list of proper names and directories:

   MC3E                   - Spring 2011 Oklahoma Project (UND Directory Structure)
   Goodrich               - Winter 2010 North Dakota Project (UND Directory Structure)
   POLCAST3               - Summer 2010 North Dakota Project (RAL Directory Structure)
   SaudiArabia_Spring2009 - Spring 2009 Saudi Arabia Project (RAL Directory Structure)
   Mali_Summer2008        - Summer 2008 Mali Project (RAL Directory Structure)
   SaudiArabia_Summer08   - Winter 2007/2008 Saudi Arabia Project (UND Directory Structure)
   POLCAST2               - Summer 2008 North Dakota Project (RAL Directory Structure)
   Mali_Summer2006        - Summer 2006 Mali Project (RAL Directory Structure)
   Mali_Summer2007        - Summer 2007 Mali Project (RAL Directory Structure)
   SaudiArabia_Winter0708 - Winter 2007/2008 Saudi Arabia Project (RAL Directory Structure)
   Saudi                  - Winter 2007/2008 Saudi Arabia Project (UND Directory Structure)
   Mali                   - July/August/September/October 2007 (UND Directory Structure)
   Harvesting             - June/July/August 2007
   Sikorsky3              - September/October 2005
   TAMDAR_Turbulence      - August/September 2005
   Sikorsky2              - January/February 2005
   L3Com                  - November/December 2004
   MPACE                  - September/October 2004
   IOP1                   - June/July 2004
   Sikorsky               - March/April 2004
   WISP4                  - February/March 2004
   THORpex                - November/December 2004
   NACP                   - May/June 2003
   Crystal                - July 2002
   Kwajex                 - August/September 1999

EX: process_all_dir POLCAST3

     This simply goes through every PostProcessing directory and processes the *.sea file. Edits are also applied.
     To view a list of command line arguments that will work with process_all_dir, just enter process_all_dir into the command line

Applications In order to change the program so that the cloud bases and temperatures could be determined during the 2010 seeding by the Cessna340_N37360 aircraft, it was necessary to find the time intervals that the aircraft was near a target. THis is due to the lack of equipment on the aircraft that would allow it to keep track of cloud bases during flight. The process used to determine this post flight is outlined below:

1. Open the data file for the date in question under /nas/ral/NorthDakota/Summer2010/Aircraft/Cessna340_N37360/FlightData, enter the Post_Processing file, and use cplot to the open the .pol3a or .pol file
2. Plot Latitude versus Longitude in cplot
3. Locate the areas that the aircraft seems to circle on the chart
4. Use a combination of Tools->Select Time Interval and Control->Time Interval in cplot to narrow down the flight just the circling times of the aircraft. As a rule of thumb, the aircraft must circle a minimum of 15 minutes in order for that region to be considered a target.
5. Now that you have found the time interval for the target, switch your x and y axis on cplot to Pressure_Alt and Air_Temp respectively. These will produce a rather random plot, but will allow you to find the average cloud height and temperature.
6. Use Tools-> Statistics to determine the approximate cloud height and temperature.

Single Field Project Scripts

process_all_*

For example, process_all_saudi calls all programs to process aircraft data for the Saudi Arabia 2007 project. You can test a process_all_* by using TestData files, for example:

 cd /usr/local/ADPAA/src/TestData/FlightData/20140429_152103  && process_all_iphex
 

Platform Scripts

aimmsprocessing_saudi

Handles and converts all of the AIMMs data that was saved on USB drives to UND Modified NASA format.

 aimmsprocessing_saudi
 Note: Needs to be executed from FlightData directory and process data within the "AIMMSData" directory
       e.g. SaudiArabia/Spring09/Aircraft/KingAir_N825ST/FlightData20090323_114454

Data Level Processing Scripts

AIMMS USB File (*.a?? and *.r??) Scripts

convert_adptonasa

Converts AIMMS adp.out to UND Modified NASA format.

 convert_adptonasa <inputfile>

Note: The times associated with this file is GPS time. This script currently only converts GPS time to UTC for 2009. For data from any other year, make sure that the time offset is correct.

convert_aimmstonasa

Convert AIMMS raw date files (*.aim) created during the Saudi Arabia 2009 project to UND Modified NASA format.

 convert_aimmstonasa file=inputfile
 Note: inputfile must have the *.aim extension.  This indicates the data file created by the concatenation of all of the *.a?? files.
       Files are assumed to be in the form of ????????.aim.

Note: The *.aim files were created by the concatenation of all the *.a?? files which were saved to the USB drive during flights. For example, cat 03231137.a?? > 03231137.aim

convert_aimtonasa

Converts AIMMs (YY_MM_DD_HH_MM_SS.aim.txt) to UND Modified NASA format.

 convert_aimtonasa <inputfile>

M300 Data File (*.sea) Scripts

extract_tables

This script will extract the tables in a given *.sea file. The syntax for this is as follows:

extract_tables <-vm> input_file

getstart_info

This subroutine determines the start time and date from a *.sea file. The syntax for this is:

getstart_info <-sfm> <-end> <-v> <-vm> input_file

process_raw

Process_raw does Level 1 data processing. The process_raw script takes the *.sea binary file produced by the M300/M200 aircraft acquisition system and creates instrument specific ASCII files. The syntax for this subroutine is:

process_raw <-d> <-v> <-vm> input_file

Example Syntax

 process_raw {$ADPAA_DIR}/src/TestData/FlightData/20140429_152103/PostProcessing/14_04_29_15_21_03.sea

process_all

Process_all script does data Level 1-4 processing. The process_all script starts with the *.sea data file produced by the M300/M200 data acquisition system and does all processing possible for that instrument data within the *.sea. The syntax for this subroutine is:

 process_all [options] inputfile

 Options:
   --fast
         Skips long duration processing.
   --final
         The data processing is to produce the final version of the data set
   --L1
         Do not process Level 1 files.
   --L2
         Do not process Level 2 files.
   --L3
         Do not process Level 3 files.
   inputfile - *.sea raw data file or any YY_MM_DD_HH_MM_SS.* based file name.

UND 'NASA' ASCII File Scripts

addedit

The addedit is a perl program that creates an edit file.

addedit [start=n] [end=n] [id='Editor Name'] [outputfile] name='Parameter' why='Reason for Edit' inputfile

Bracket entries are optional. Single quotation entries only are mandatory. If no start and end times are given the whole entire time period of the inputfile is used. If no id parameter is given the Flight Scientist name is used.The inpute file has to be a *.raw file. IF no outputfile is given the input file is used as the base of the output file.

Ex: addedit start=64100 end=64200 name='Pitot_Nose' why='Iced up' 08_06_13_18_19_05.physical.raw

apply_edits

The apply_edits file is used to create a clean file which can be used with cplot.

apply_edits <csh_error> input_filename edit_filename output_filename

input_filename - File in the starndard NASA/UND ASCII format to which edits are applied. edit_filename - File which contain the edits to apply to the input file. output_filename - File created by appling the edits to the input file.

Including the csh_error option causes the error to be return to the csh calling program, instead of occuring in the IDL code.

avgfields

Averages fields from a source file into a target file

 avgfields points=n|frequency=n [avgwithmvc=0|1] [stdfile=?] [exp_notation=0|1] [timeformat=string] < inputfile > outputfile 
    avgwithmvc = 0:  Average is missing value code if any values within the time period are a missing value code.
    avgwithmvc = 1:  Average is missing value code only if all values within the time period are missing value codes.
    exp_notation = 1:  Data is printed in exponential notation (e.g. 9.99999e+01). Default is data is printed in floating point notation.
    timeformat string is in standard notation, for example timeformat='%12.6f

Note: The points flag takes n points and averages them throughout the data set. The frequency flag makes n frequencies out of data by averaging the middle of n with a weight and weights the rest of the data accordingly. USE ONE OR THE OTHER (the points flag is the most often used).

combine_files

This script combines parted (different files on the same flight) files that have the same extension. To use this script call it in any ????????_? (YEARMMDD_#) named folder without specifying any arguments. The syntax for this script is:

combine_files <file_suffix>
file_suffix - Suffix after the first dot of the file to process.
              If file_suffix argument not given, default list of summary files is used.

drift_correction

The drift_correction is a python program that corrects for cloud drift due to wind speeds to show the hypothetical path a plane would follow if clouds did not drift due to wind speed. This can be particully useful when trying to determine when a cloud is passing through a cloud more than once. The syntax for running the code is:

drift_correction [start=n] [end=n] [coordinates=AIMMS] [speed=n] [direction=n] inputfile

Start and end represent the start and end times for which you want drift coordinates displayed for. Coordinates are whatever coordinates you want corrected (AIMMS or GPS). Speed and direction are optional parameters input by the user when the user does not want to rely on AIMMS to provide the wind direction and speed for the interval being corrected. If speed and direction are not provided, the program will choose the first valid AIMMS wind speed and direction in the interval selected from the inputfile The input file should be a *.sau file. The output file will be a *.drift file. this program may only be run on a computer with python 2.5 with numpy installed (as of now esxfedora and buster are the only computers with the program).

fileMath

This script performs mathematical operations on a UND "NASA" formatted file. Syntax for running script:

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=<parameter index 1> y=<constant (must have decimal), single parameter index, or multiple parameter indices (comma-separated)> Type=<operation> <inputfile> <outputfile>

The available operations are addition (add), subtraction (sub), multiplication (mult), and division (div). Param1 and param2 are the indices of the parameters in the NASA file that will be added, subtracted, multiplied, or divided. The input file can be any UND "NASA" formatted file. To overwrite the input file and add the newly derived data to the original file, use the name of the input file as the name of the output file. The output file does not have to be the original file, but to add the new data to the original file, it must be.

Example syntax:

Single Parameter

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=2 y=1 Type=div 15_08_08_14_57_48.air.ascent.1Hz 15_08_08_14_57_48.air.ascent.1Hz

This will divide the dew point temperature (parameter 2 in this NASA file) by the air temperature (parameter 1 in this NASA file) and add the new data to the original file, 15_08_08_14_57_48.air.ascent.1Hz. This NASA file, 15_08_08_14_57_48.air.ascent.1Hz, is located in /usr/local/ADPAA/src/TestData/FlightData/20150808_145748/Analysis/

Constant

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=2 y=0.5 Type=add 15_08_08_14_57_48.air.ascent.1Hz outfile.txt

This will add 0.5 to the dew point temperature (second parameter in this NASA file) and write the new data to a separate output file called "outfile.txt".

Multiple Parameters

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=1 y=2,3 Type=add 15_08_08_14_57_48.air.ascent.1Hz 15_08_08_14_57_48.air.ascent.1Hz

This will add together the first, second, and third parameters in this aircraft file. For addition and multiplication, the order of the parameter indices does not matter. By setting x=3 and y=1,2 , the resulting data would be exactly the same. However, for subtraction, the parameter from which you will be subtracting must be in the x spot. For example, to subtract parameters 1 and 2 from parameter 3, set x=3 and y=1,2 or y=2,1.

listparas

This subroutine program can be used to determine what parameters that are required by UND "NASA" formatted files. The syntax is:

listparas <-h> <-l> inputfile
-l, --longname                         
Include full variable description

mergefield

The purpose of the mergefield program is to merge parameters of one data file with another. The syntax of the program is as such:

mergefield field=n [target=n] file=mergefile [MVC[i]=x]+ [tolerance= ] [fill=interpolate|older] < infile > outfile

Field is the parameter(s) being taken from the input file and added to the mergefile. This is expressed as a number and can be found using listparas. Target is the number of the parameter being put into the output file (again, found using listparas).

For example, say you wanted the temperature parameter from the March 20, 2009 Saudi file to be added to the pcasp.raw data file. The temperature is the first parameter (column of data) in the .sau file; therefore, the field flag would be set to 1. If you want all parameters from the .sau file to be added, do not specify a field number. There are 17 parameters in the March 20, 2009 raw pcasp file, so the target flag would be set to the number 18, since you want to add it to the end of the new file (the 18th parameter). Please note that if you do not specify the target, it will automatically be set to the field number(s). The file flag must be set to whatever file the field is being obtained from, in this case it is 09_03_20_06_46_51.sau. The infile is whatever the file the field is being added to, in this case it is 09_03_20_06_46_51.conc.pcasp.raw. The outfile is the name of the new file which is being output by this program--in other words, what the file that contains the combined parameters will be named. For example, I could name this new file 09_03_20_06_46_51.temp.pcasp.merge.

Overall, the command would look like this:

mergefield field=1 target=18 file=09_03_20_06_46_51.sau < 09_03_20_06_46_51.conc.pcasp.raw > 09_03_20_06_46_51.temp.pcasp.merge

This would output a file named 09_03_20_06_46_51.temp.pcasp.merge to whatever directory the command is being executed from. Tolerance and fill are optional parameters and are rarely used.

pcaspscat

The pcaspscat program creates a *.550nm.scat.raw file (from whatever *.conc.pcasp.raw file in the directory the program is being executed from). This file displays atmospheric scattering, absorbtion, backscattering, and assymetry parameters all based on the pcasp channels. These parameters are all based on a wavelength of 550nm. The program is executed under the following syntax:

pcaspscat [-v] [-vm] input_file

1. v - Verbose mode.
2. vm - Execute code using IDL virtual machine.

The input file must be a *.conc.pcasp.raw file.

process_raw_file

This subset program edits the input file and creates a .1Hz file. This subset utilizes the avgfileds function which averages the fields from the source file into a target file. In this case the target file is the newly created .1Hz file. The syntax for this program is as follows:

process_raw_file inputfile

sfm2hms

Converts time from seconds from midnight (sfm) to hour:minute:second time format

 sfm2hms < infile > outputfile

subset

The subset program is used to create a data file which takes data from another file that follows a certain numerical characteristic. For example, say I want all data from the March 20, 2009 Saudi file that had LWC greater than 1 g/cm^3 or all data with temperatures between -10 and -5 degrees Celsius. I can use the subset program to create a file with that data. The syntax for the program is as follows:

subset [start=n] [end=n] [test=n] [field=n] [fill=MVC's] [mvcfield=n] [newmvc=n] [name='Parameter Names'] [unit='Units Values'] [format='parameter format'] [format_s='string format']  < inputfile > outputfile

Notes: The start and end flags are the starting and ending times of the needed subset, respectively. The field flag sets the parameter number in the data file that is in need of subsetting (found by using listparas). The inputfile is the file with parameter(s) that need subsetting and outputfile is the new file with the subsetted data.

Analysis Scripts

cplot

To start the main visulation program.

cplot data_file

The 'data_file' needs to be in the NASA UND format.

plot2dc

To bring up 2dc probe images, the plot2dc program may be used. To execute simply use this syntax:

plot2dc 2DC_file

Images from the *.2dc file will be brought up for analysis.