.:scripts:home

Hierarchical Processing Scripts

The graphic below shows the structure of the process_all scripts in the ADPAA library. Each script calls the ones beneath it (for example: the process_all_dir script will call the process_all_polcast3 script, which in turn calls the process_all script, and etc.).

Master Processing Scripts

process_all_dir

This simply goes through every PostProcessing directory and processes the *.sea file. Edits are also applied. To view a list of command line arguments that will work with process_all_dir, just enter process_all_dir into the command line.

When running the process_all_dir script it is a requirement to be in the general time period (YYYY) directory. If the script is ran in any other directory nothing will happen.

The following is the list of proper names and directories:

   oracles2016            - Fall 2016 ORACLES Project (UND Directory Structure)
   ophir2016              - Spring 2016 Ophir Project (UND Directory Structure)
   olympex                - Fall 2015 Washington Project (UND Directory Structure)
   CAPE2015               - July/August 2015 Florida Project (UND Directory Structure)
   UTC2015                - Spring 2015 North Dakota Project (UND Directory Structure)
   UTC2014                - Fall 2014 North Dakota Project (UND Directory Structure)
   iphex                  - Summer 2014 North Carolina Project (UND Directory Structure)
   POLCAST4               - Summer 2012 North Dakota Project (RAL Directory Structure)
   gcpex                  - Winter 2012 Georgian Bay Project (UND Directory Structure)
   MC3E                   - Spring 2011 Oklahoma Project (UND Directory Structure)
   Goodrich               - 2010/2011/2012 North Dakota Project (UND Directory Structure)
   POLCAST3               - Summer 2010 North Dakota Project (RAL Directory Structure)
   SaudiArabia_Spring2009 - Spring 2009 Saudi Arabia Project (RAL Directory Structure)
   Mali_Summer2008        - Summer 2008 Mali Project (RAL Directory Structure)
   SaudiArabia_Summer08   - Winter 2007/2008 Saudi Arabia Project (UND Directory Structure)
   POLCAST2               - Summer 2008 North Dakota Project (RAL Directory Structure)
   Mali_Summer2006        - Summer 2006 Mali Project (RAL Directory Structure)
   Mali_Summer2007        - Summer 2007 Mali Project (RAL Directory Structure)
   SaudiArabia_Winter0708 - Winter 2007/2008 Saudi Arabia Project (RAL Directory Structure)
   Saudi                  - Winter 2007/2008 Saudi Arabia Project (UND Directory Structure)
   Mali                   - July/August/September/October 2007 (UND Directory Structure)
   Harvesting             - June/July/August 2007
   Sikorsky3              - September/October 2005
   TAMDAR_Turbulence      - August/September 2005
   Sikorsky2              - January/February 2005
   L3Com                  - November/December 2004
   MPACE                  - September/October 2004
   IOP1                   - June/July 2004
   Sikorsky               - March/April 2004
   WISP4                  - February/March 2004
   THORpex                - November/December 2004
   NACP                   - May/June 2003
   Crystal                - July 2002
   Kwajex                 - August/September 1999

EX: process_all_dir POLCAST3

     This simply goes through every PostProcessing directory and processes the *.sea file. Edits are also applied.
     To view a list of command line arguments that will work with process_all_dir, just enter process_all_dir into the command 
     line.

Applications: In order to change the program so that the cloud bases and temperatures could be determined during the 2010 seeding by the Cessna340_N37360 aircraft, it was necessary to find the time intervals that the aircraft was near a target. This is due to the lack of equipment on the aircraft that would allow it to keep track of cloud bases during flight. The process used to determine this post flight is outlined below:

1. Open the data file for the date in question under /nas/ral/NorthDakota/Summer2010/Aircraft/Cessna340_N37360/FlightData, enter the Post_Processing file, and use cplot to the open the .pol3a or .pol file
2. Plot Latitude versus Longitude in cplot
3. Locate the areas that the aircraft seems to circle on the chart
4. Use a combination of Tools->Select Time Interval and Control->Time Interval in cplot to narrow the flight down to just the circling times of the aircraft. As a rule of thumb, the aircraft must circle a minimum of 15 minutes in order for that region to be considered a target.
5. Now that you have found the time interval for the target, switch your x and y axis on cplot to Pressure_Alt and Air_Temp respectively. These will produce a rather random plot, but will allow you to find the average cloud height and temperature.
6. Use Tools-> Statistics to determine the approximate cloud height and temperature.

Single Field Project Scripts

process_all_*

For example, process_all_saudi calls all programs to process aircraft data for the Saudi Arabia 2007 project. You can test a process_all_* by using TestData files, for example:

 cd /usr/local/ADPAA/src/TestData/FlightData/20140429_152103  && process_all_iphex
 

Platform Scripts

aimmsprocessing_saudi

Handles and converts all of the AIMMs data that was saved on USB drives to UND Modified NASA format.

 aimmsprocessing_saudi
 Note: Needs to be executed from a FlightData directory and processes data within the "AIMMSData" directory
       e.g. SaudiArabia/Spring09/Aircraft/KingAir_N825ST/FlightData20090323_114454

Data Level Processing Scripts

AIMMS USB File (*.a?? and *.r??) Scripts

convert_adptonasa

Converts AIMMS adp.out to UND Modified NASA format.

 convert_adptonasa <inputfile>

Note: The time associated with these files is GPS time. This script currently only converts GPS time to UTC for 2009. For data from any other year make sure that the time offset is correct.

convert_aimmstonasa

Convert AIMMS raw data files (*.aim) created during the Saudi Arabia 2009 project to UND Modified NASA format.

 convert_aimmstonasa file=inputfile
 Note: inputfile must have the *.aim extension.  This indicates the data file created by the concatenation of all of the *.a?? files.
       Files are assumed to be in the form of ????????.aim.

Note: The *.aim files were created by the concatenation of all the *.a?? files which were saved to the USB drive during flights. For example, cat 03231137.a?? > 03231137.aim

convert_aimtonasa

Converts AIMMs (YY_MM_DD_HH_MM_SS.aim.txt) to UND Modified NASA format.

 convert_aimtonasa <inputfile>

M300 Data File (*.sea) Scripts

extract_tables

This script will extract the tables in a given *.sea file. The syntax for this is as follows:

extract_tables <-vm> input_file

getstart_info

This subroutine determines the start time and date from a *.sea file. The syntax for this is:

getstart_info <-sfm> <-end> <-v> <-vm> input_file

process_raw

Process_raw does Level 1 data processing. The process_raw script takes the *.sea binary file produced by the M300/M200 aircraft acquisition system and creates instrument specific ASCII files. The syntax for this subroutine is:

process_raw <-d> <-v> <-vm> input_file

Example Syntax

 process_raw {$ADPAA_DIR}/src/TestData/FlightData/20140429_152103/PostProcessing/14_04_29_15_21_03.sea

process_all

Process_all script does data Level 1-4 processing. The process_all script starts with the *.sea data file produced by the M300/M200 data acquisition system and does all processing possible for that instrument data within the *.sea. The syntax for this subroutine is:

 process_all [options] inputfile

 Options:
   --fast
         Skips long duration processing.
   --final
         The data processing is to produce the final version of the data set
   --L1
         Do not process Level 1 files.
   --L2
         Do not process Level 2 files.
   --L3
         Do not process Level 3 files.
   inputfile - *.sea raw data file or any YY_MM_DD_HH_MM_SS.* based file name.

Note** - No errors should be listed in the outputted YY_MM_DD_HH_MM_SS.log.postrocessing file upon completion of the process_all job. If so there are issues with the results.

NASA/UND ASCII (1001) Input/Output Scripts

Python

Input - ${ADPAA_DIR}/src/python_lib/readfile.py

Note: It is recommended to call this package using the 'ReadFile' method within the ADPAA Python package instead of calling this package directly.

Output - ${ADPAA_DIR}/src/python_lib/writefile.py

Note: It is recommended to call this package using the 'WriteFile' method within the ADPAA Python package instead of calling this package directly.

Constants - ${ADPAA_DIR}/src/python_lib/constants.py

Note: It is recommended to call this package using the 'constants' method within the ADPAA Python package instead of calling this package directly.

NASA/UND ASCII (1001) Conversion Scripts

convert_oid2nasa

Converts a file in the Optical Ice Detector format into the NASA/UND ASCII format Syntax:

 convert_oid2nasa inputfile

Example:

 convert_oid2nasa 150801_182011_OidBackscatterStokes.csv
 produces: 15_08_01_18_20_11.OID.1Hz

convert_undtoICARTT

Converts a file in the NASA/UND ASCII (1001) format to the 2008 ICARTT format.

Syntax:

 convert_undtoICARTT file=inputfile > outputfile

Example:

 convert_undtoICARTT file=14_02_20_00_59_00.KGFK.raw > Surface-KGFK_RWIC_20140220_RA.ict

convert_undtoICARTT2013

Converts a file in the NASA/UND ASCII (1001) format to the 2013 ICARTT format.

Syntax:

 convert_undtoICARTT2013 [pi_contact_info=string1] [platform=string2] [location=string3] [associated_data=string4] 
                         [instrument_info=string5] [data_info=string6] [project_info=string7] [stipulations_on_use=string8] 
                         [revision_number='R#'] [revision_comments=string10]  file=inputfile > outputfile

The input file should be in the NASA/UND 1001 format. The output file will be in the 2013 ICARTT ASCII data format.

Keyword entries in brackets ([]) are optional. Strings following keywords should be in quotes (single or double quotes allowed). The optional revision number entry should be the same in the data file name as in the file header.

Example:

 convert_undtoICARTT2013 platform="UND Citation II (N555DS) - http://airborneresearch.atmos.und.edu/AircraftInfo.aspx" 
   location="N/A" file=15_11_04_19_35_11.applanix.cwc.1Hz > CSI_UNDCitation_20151104193511_R0.ict

UND 'NASA' ASCII (1001) Modification Scripts

addedit

The addedit is a perl program that creates an edit file.

addedit [start=n] [end=n] [id='Editor Name'] [outputfile] name='Parameter' why='Reason for Edit' inputfile

Bracket entries are optional. Single quotation entries only are mandatory. If no start or end times are given the entire time period of the inputfile is used. If no id parameter is given the Flight Scientist name is used.The inputed file has to be a *.raw file. If no outputfile is given the input file is used as the base of the output file.

Ex: addedit start=64100 end=64200 name='Pitot_Nose' why='Iced up' 08_06_13_18_19_05.physical.raw

apply_edits

The apply_edits file is used to create a clean file which can be used with cplot.

apply_edits <csh_error> input_filename edit_filename output_filename

input_filename - File in the standard NASA/UND ASCII format to which edits are applied.

edit_filename - File which contains the edits to apply to the input file.

output_filename - File created by applying the edits to the input file.

Including the csh_error option causes the error to be returned to the csh calling program instead of occurring in the IDL code.

avgfields

Averages fields from a source file into a target file.

 avgfields points=n|frequency=n [avgwithmvc=0|1] [stdfile=?] [exp_notation=0|1] [timeformat=string] < inputfile > outputfile 
    avgwithmvc = 0:  Average is missing value code if any values within the time period are a missing value code.
    avgwithmvc = 1:  Average is missing value code only if all values within the time period are missing value codes.
    exp_notation = 1:  Data is printed in exponential notation (e.g. 9.99999e+01). Default is data is printed in floating point notation.
    timeformat string is in standard notation, for example timeformat='%12.6f

Note: The points flag takes n points and averages them throughout the data set. The frequency flag makes n frequencies out of data by averaging the middle of n with a weight and weights the rest of the data accordingly. USE ONE OR THE OTHER (the points flag is the most often used).

combine_files

This script combines parted (different files on the same flight) files that have the same extension. To use this script call it in any ????????_? (YEARMMDD_#) named folder without specifying any arguments. The syntax for this script is:

combine_files <file_suffix>
file_suffix - Suffix after the first dot of the file to process.
              If file_suffix argument not given, default list of summary files is used.

drift_correction

The drift_correction is a python program that corrects for cloud drift due to wind speeds to show the hypothetical path a plane would follow if clouds did not drift due to wind speed. This can be particully useful when trying to determine when an aircraft is passing through a cloud more than once. The syntax for running the code is:

drift_correction [start=n] [end=n] [coordinates=AIMMS] [speed=n] [direction=n] inputfile

"Start" and "end" represent the start and end times for which you want drift coordinates displayed for. Coordinates are whatever coordinates you want corrected (AIMMS or GPS). Speed and direction are optional parameters input by the user when the user does not want to rely on AIMMS to provide the wind direction and speed for the interval being corrected. If speed and direction are not provided, the program will choose the first valid AIMMS wind speed and direction in the interval selected from the inputfile. The input file should be a *.sau file. The output file will be a *.drift file. this program may only be run on a computer with python 2.5 with numpy installed (as of now esxfedora and buster are the only computers with the program).

fileMath

This script performs mathematical operations on a UND "NASA" formatted file. Syntax for running the script:

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=<parameter index 1> y=<constant (must have decimal), single parameter index, or multiple parameter indices (comma-separated)> Type=<operation> <inputfile> <outputfile>

The available operations are addition (add), subtraction (sub), multiplication (mult), and division (div). Param1 and param2 are the indices of the parameters in the NASA file that will be added, subtracted, multiplied, or divided. The input file can be any UND "NASA" formatted file. To overwrite the input file and add the newly derived data to the original file, use the name of the input file as the name of the output file. The output file does not have to be the original file, but to add the new data to the original file, it must be.

Example syntax:

Single Parameter

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=2 y=1 Type=div 15_08_08_14_57_48.air.ascent.1Hz 15_08_08_14_57_48.air.ascent.1Hz

This will divide the dew point temperature (parameter 2 in this NASA file) by the air temperature (parameter 1 in this NASA file) and add the new data to the original file, 15_08_08_14_57_48.air.ascent.1Hz. This NASA file, 15_08_08_14_57_48.air.ascent.1Hz, is located in /usr/local/ADPAA/src/TestData/FlightData/20150808_145748/Analysis/

Constant

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=2 y=0.5 Type=add 15_08_08_14_57_48.air.ascent.1Hz outfile.txt

This will add 0.5 to the dew point temperature (second parameter in this NASA file) and write the new data to a separate output file called "outfile.txt".

Multiple Parameters

 ${ADPAA_DIR}/src/python_lib/fileMath.py x=1 y=2,3 Type=add 15_08_08_14_57_48.air.ascent.1Hz 15_08_08_14_57_48.air.ascent.1Hz

This will add together the first, second, and third parameters in this aircraft file. For addition and multiplication, the order of the parameter indices does not matter. By setting x=3 and y=1,2 , the resulting data would be exactly the same. However, for subtraction, the parameter from which you will be subtracting must be in the x spot. For example, to subtract parameters 1 and 2 from parameter 3, set x=3 and y=1,2 or y=2,1.

filteroutliers

This script will eliminate outlier data outside of two standard deviations in the UND "NASA" file. filteroutliers.py uses a Z-Score test with rolling windows of x amount of points to mask data as Missing Value Code (MVC) in order to clean up digital noise in the data. The filter can be applied to a single parameter, multiple parameters separated by commas, or all parameters available in the file.

Syntax for running script:

   ${ADPAA_DIR}/src/python_lib/filteroutliers.py freq=<amount of points for average> keys= <NASA/ASCII file in> <output file name>

Examples:

Single Parameter:

   ${ADPAA_DIR}/src/python_lib/filteroutliers.py freq=30 keys=Wind_Z_Nose 15_08_08_14_57_48.REAL.winds.1Hz 15_08_08_14_57_48.REAL.winds.flt

This will take the mean value of every 30 points in the Wind_Z_Nose data of the aircraft file and filter data outside of two standard deviations of that mean by making those points MVCs. The output file (15_08_08_14_57_48.REAL.winds.flt) will have the filtered data in the Wind_Z_Nose parameter.

Multiple Parameters:

   ${ADPAA_DIR}/src/python_lib/filteroutliers.py freq=30 keys=AlphaPreNo,BetaPreNo 15_08_08_14_57_48.physical.raw 15_08_08_14_57_48.physical.flt

This will filter the data from both AlphaPreNo and BetaPreNo.

All Parameters:

   ${ADPAA_DIR}/src/python_lib/filteroutliers.py freq=30 15_08_08_14_57_48.physical.raw 15_08_08_14_57_48.physical.flt

To filter all parameters in an aircraft file, you must leave out the "keys=" command as shown. This will run through all parameters in the file and apply a filter to each one.

NOTE: This program requires a Numpy version of 1.8.0 or later

listparas

This subroutine program can be used to determine what parameters are required by UND "NASA" formatted files. The syntax is:

listparas <-h> <-l> inputfile
-l, --longname                         
Include full variable description

merge_cdp_2ds_hvps3

The purpose is to merge spectrums from specified channels of the CDP, 2DS, and HVPS3 into a single 1 Hz file. These channels are:

CDP: 1 - 30 2DS: 6 - 24 HVPS3: 5 - 28

Data is saved both by channel and as a single, total-channel spectrum.

Syntax

 merge_cdp_2ds_hvps3.py cdp_infile 2ds_infile hvps3_infile

Example

 merge_cdp_2ds_hvps3.py 15_08_01_18_20_11.conc.cdp.1Hz 15_08_01_18_20_11.2DS_H.conc.1Hz 15_08_01_18_20_11.HVPS3_H.=conc.1Hz
 Produces: 15_08_01_18_20_11.merge.cdp.2DS_H.HVPS3_H.1Hz

mergefield

The purpose of the mergefield program is to merge parameters of one data file with another. The syntax of the program is as such:

mergefield field=n [target=n] file=mergefile [MVC[i]=x]+ [tolerance= ] [fill=interpolate|older] < infile > outfile

Field is the parameter(s) being taken from the input file and added to the mergefile. This is expressed as a number and can be found using listparas. Target is the number of the parameter being put into the output file (again, found using listparas).

For example, say you wanted the temperature parameter from the March 20, 2009 Saudi file to be added to the pcasp.raw data file. The temperature is the first parameter (column of data) in the .sau file; therefore, the field flag would be set to 1. If you want all parameters from the .sau file to be added, do not specify a field number. There are 17 parameters in the March 20, 2009 raw pcasp file, so the target flag would be set to the number 18, since you want to add it to the end of the new file (the 18th parameter). Please note that if you do not specify the target, it will automatically be set to the field number(s). The file flag must be set to whatever file the field is being obtained from, in this case it is 09_03_20_06_46_51.sau. The infile is whatever file the field is being added to, in this case it is 09_03_20_06_46_51.conc.pcasp.raw. The outfile is the name of the new file which is being output by this program--in other words, what the file that contains the combined parameters will be named. For example, I could name this new file 09_03_20_06_46_51.temp.pcasp.merge.

Overall, the command would look like this:

mergefield field=1 target=18 file=09_03_20_06_46_51.sau < 09_03_20_06_46_51.conc.pcasp.raw > 09_03_20_06_46_51.temp.pcasp.merge

This would output a file named 09_03_20_06_46_51.temp.pcasp.merge to whatever directory the command is being executed from. Tolerance and fill are optional parameters and are rarely used.

merge_spectra

The purpose of the merge_spectra program is to merge the particle size spectrum produced by one probe with the spectrum produced by another probe. The syntax of the program is as follows:

merge_spectra <final> <csh_error> <verbose_mode> <virtual_machine> <spec_units = #> small_spectrum_file large_spectrum_file

Input:

* small_spectrum_file - any file in the UND or NCAR file formats containing particle size distribution data. This would be 2DC, 2DS, CIP, etc
* large_spectrum file - any file in the UND or NCAR file formats containing particle size distribution data. This would be 2DS, 2DP, HVPS-3, etc.

Options: spec_units - a number corresponding to the desired output unit.

    0 : spectra in [#/cm^3] (default)
    1 : spectra in [#/cm^3/um]
    2 : spectra in [#/m^4]

final - Causes the output data to be tagged Final instead of Preliminary. cash_error - Errors are returned to the cash calling program instead of occurring in the IDL code.

pbp_idl2und.py

pbp_idl2und.py converts the raw particle by particle file outputted from the process_soda2 code into the UND NASA format. The script was written for use with 2DS data, but it can be used for other probe data as well, such as HVPS (as long as it has an option to generate particle by particle files in the soda2 code). To use the script, use the following syntax:

 $ADPAA_DIR/src/scripts_python/pbp_idl2und.py <pbp_file>

where <pbp_file> is the name of the particle by particle file. For example, if this script was run with 073015_175600_2DS_H.pbp, it would generate an output file of the name "15_07_30_17_56_00_2DS_H_pbp.raw".

pcaspscat

The pcaspscat program creates a *.550nm.scat.raw file (from whatever *.conc.pcasp.raw file is in the directory the program is being executed from). This file displays atmospheric scattering, absorbtion, backscattering, and assymetry parameters all based on the pcasp channels. These parameters are all based on a wavelength of 550nm. The program is executed under the following syntax:

pcaspscat [-v] [-vm] input_file

1. v - Verbose mode.
2. vm - Execute code using IDL virtual machine.

The input file must be a *.conc.pcasp.raw file.

process_raw_file

This subset program edits the input file and creates a .1Hz file. This subset utilizes the avgfields function which averages the fields from the source file into a target file. In this case the target file is the newly created .1Hz file. The syntax for this program is as follows:

process_raw_file inputfile

sfm2hms

Converts time from seconds from midnight (sfm) to hour:minute:second time format

 sfm2hms < infile > outputfile

subset

The subset program is used to create a data file which takes data from another file that follows a certain numerical characteristic. For example, say I want all data from the March 20, 2009 Saudi file that had LWC greater than 1 g/cm^3 or all data with temperatures between -10 and -5 degrees Celsius. I can use the subset program to create a file with that data. The syntax for the program is as follows:

subset [start=n] [end=n] [test=n] [field=n] [fill=MVC's] [mvcfield=n] [newmvc=n] [name='Parameter Names'] [unit='Units Values'] [format='parameter format'] [format_s='string format']  < inputfile > outputfile

Notes: The start and end flags are the starting and ending times of the needed subset, respectively. The field flag sets the parameter number in the data file that is in need of subsetting (found by using listparas). The inputfile is the file with parameter(s) that need subsetting and outputfile is the new file with the subsetted data.

NASA/UND ASCII (1001) Analysis Scripts

calculate_b_coef

Used to calculate the the backscatter coefficient using a reorganized version of Equation 1 found in "Analysis of Flight Test Results of the Optical Ice Detector (2015) - Mark Ray and Kaare Anderson. A file is generated containing 1 Hz backscatter coefficient data (*.b_coef.1Hz) using a merged spectrum file generated by merged_cdp_2ds_hvps3.py (*.merge.cdp.2DS.HVPS3.1Hz).

Syntax:

 calculate_b_coef.py infile

Example:

 calculate_b_coef.py 15_08_01_18_20_11.merge.cdp.2DS_H.HVPS3_H.1Hz
 Produces: 15_08_01_18_20_11.b_coef.1Hz

classify_habit

A python script which uses 1Hz data of aircraft temperature and particle aspect ratios to produce a data file containing the most likely ice-habit. Infile1 contains the basic data and infile2 the aspect ratios.

Syntax:

 classify_habit.py infile1 infile2

Example:

 classify_habit.py 14_05_23_18_08_42.basic.1Hz 14_05_23_18_08_42.2DS_H.aspect_ratio.1Hz
 Produces: 14_05_23_18_08_42.habit.1Hz  

cplot

To start the main visualization program.

cplot data_file

cplot2

Cplot2 is similar to Cplot; however, IDL new graphics are used for the display which enables more interactive changes to the plots.

To start the main visualization program.

cplot2 data_file

To calculate statistics over a time period: Run cplot2 on the data file, and click OK. In the top toolbar, select Control and set the Time Interval. Then select the variables to be used for the x and y axes. Then select Tools > Statistics.

plot2dc

To bring up 2dc probe images, the plot2dc program may be used. To execute simply use this syntax:

plot2dc 2DC_file

Images from the *.2dc file will be brought up for analysis.