.:editfiles:home

Automatically Filtered Files (*.flt)

Automatically filtered files (*.flt) are the next step up from *.sync files (3 steps up from *.raw files) because they are automatically filtered for quality assurance. Spikes generated by wrongly encoded serial data are masked with a missing value code.

Creating

The filteroutliers.py script can be used to create a filtered file (*.flt). The command syntax is:

 filteroutliers.py [points=n] [stdev=z] [keys=shortnames] input_file output_file
 n - Number of data points in average (default is 1500 data points, or 1 minute of data at 25 Hz).
 z - The z-score where all outlying points will be converted to missing value codes (default is 3 standard deviations).
 shortnames - Comma-delimited list of short names of parameters (VDESC) to filter (default is all data in file will be filtered).

For example:

 filteroutliers.py points=1500 stdev=3 keys=AlphaPreNo,BetaPreNo,PitotPreNo 15_08_08_14_57_48.physical.raw 15_08_08_14_57_48.physical.flt

Notes:

• The program works best at filtering serial data. Use caution when filtering analog data as too many data points may be filtered.

Human-Edited Files (*.edits)

Human-edited files (*.edits) are the next step up from *.flt files (4 steps up from *.raw files) because they are quality assured by scientists familiar with the instrumentation. Any data that cannot be validated is replaced with a missing value code and is ignored upon further processing. The *.edits files are a summary of the edits made to the data and are a precursor to the *.clean files.

Creating

The addedit script can be used to create or append to an edit file (*.edits). The command syntax is:

 addedit [start=n] [end=n] [id='Editor Name'] [outputfile] name='Parameter' why='Reason for edit' 08_06_13_18_19_05.physical.raw

For Example:

 addedit start=64100 end=64200 name='Pitot_Nose' why='Iced up' inputfile

Notes:

• If start and end times are not given the entire time period of the inputfile is used.
• If no id parameter is given the Flight Scientist name is used.
• The input file needs to be a *.raw file.
• If no outputfile is given the inputfile is used as the base of the output file.

You can use the gvim edit to view/modify an edit file.

Archiving

• To back up all of the *.edits files in a data set first navigate to the project name directory. For example:

 $ cd /nas/ral/Mali

• Rename the current *editfiles* archive file. This way when the create_editstarfile script is run the current archive file will not be overwritten.

 $ mv Summer2008.editfiles.tar.gz Summer2008.editfiles.tar.gz_090819

• Execute the create_editstarfile script from the current directory. The argument passed into the script should be a general time period found within the project name directory.

 $ create_editstarfile Summer2008

• This creates the new *.editfiles.tar.gz archive file.

Extracting and Applying Archived

• To extract and apply edits, first go to the project name directory.

 $ cd /nas/ral/Mali

• Then untar the archived *.edits files.

 $ tar -xvzf Summer2008.editfiles.tar.gz