wiki:AffectedDatasets

Affected Datasets

Project to develop a system which will look into reference files to see if changes to a reference actually affects a specific dataset. For tables, this is on the presumption that there are row changes for only specific modes of an instrument, and that other modes would be unaffected, and therefore not require reprocessing.

The "deep examination" is performed after a pair of reference files has found to be different as per standard CRDS bestrefs operation. These rules are then used to determine whether a set of rows can be determined from the dataset information. If so, those rows are compared between the references. Only if the rows are identical, is the reprocessing flag inverted back to "no reprocess". If the rows are different, or there are no rules for the particular reference type, or any other error or ambiguity is found, the dataset will continue to be flagged for reprocessing.

Reference Table State

The following lists all the tables and whether they have rules or not. If not, the reason is given.

Note: As described above, bestrefs is at this point in checking only because a set of refernces has been found to be different per standard CRDS rules, and the dataset is currently flag for reprocessing.

COS

badttab badt:: Not implemented

Rule: Default

Always reprocess

Currently a dummy table. The rule for row selection is:

If table.START or table.END is in the range dataset.EXPSTART to dataset.EXPEND

Issue: From startview->cos->cos instrument, there is no obvious EXPEND. There is an "exposure length" retrieved from catalog..science.sci_actual_duration. Need to confirm whether this is actually how EXPEND is determined. Otherwise, there appears to be no keyword.

bpixtab bpix:: Implemented

Rule: COS Segment

With anything related to COS Segment, if a dataset has SEGMENT=BOTH, that set has already been processed and there is never a reprocess to be done.

brftab brf:: Not implemented

Rule: Default

Always reprocess

Notes: Rows are not mode-dependent.

brsttab burst:: Implemented

Rule: COS Segment

Match on column SEGMENT

deadtab dead:: Implemented

Rule: COS Segment

Match on column SEGMENT

disptab disp:: Implemented

Rule: COS disptab-specific

Match on OPT_ELEM && CENWAVE

fluxtab phot:: Implemented

Rule: COS full mode:

Match on OPT_ELEM && CENWAVE && APERTURE

gsagtab gsag:: Insufficient information

Rule: Default

Always reprocess

Notes:

Would select on SEGMENT and high voltage. However, high voltage is not in the DADSOPS. Further, the determination process would not result in any other result than reprocess anyways.

hvtab hv:: Not implemented

Rule: Default

Always reprocess

Notes: Rows are not mode-dependent.

lamptab lamp:: Implemented

Rule: COS lamptab-specific

Match on OPT_ELEM && CENWAVE && FPOFFSET

phatab pha:: Implemented

Rule: COS opt_elem

Match on OPT_ELEM

proftab:: Implemented

Rule: COS full mode with segment:

Match on OPT_ELEM, CENWAVE, APERTURE, SEGMENT

spwcstab spwcs:: Implemented

Rule: COS full mode:

Match on OPT_ELEM && CENWAVE && APERTURE

tdstab tds:: Implemented

Rule: COS tdstab-specific

Match on OPT_ELEM && APERTURE

tracetab:: Implemented

Rule: COS full mode with segment:

Match on OPT_ELEM, CENWAVE, APERTURE, SEGMENT

twozxtab:: Implemented

Rule: COS full mode with segment:

Match on OPT_ELEM, CENWAVE, APERTURE, SEGMENT

walktab walk:: Implemented

Rule: COS Segment

Match on column SEGMENT

wcptab wcp:: Implemented

Rule: COS opt_elem

Match on OPT_ELEM

xtractab 1dx:: Implemented

Rule: COS full mode:

Match on OPT_ELEM && CENWAVE && APERTURE

STIS

apdestab apd:: Implemented

Rule: STIS aperture

Match on APERTURE

apertab apt:: Implemented

Rule: STIS aperture

Match on APERTURE

bpixtab bpx:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

ccdtab ccd:: Implemented

Rule: STIS ccdtab-specific

Match on CCDAMP && CCDGAIN && CCDOFFST && BINAXIS1 && BINAXIS2

cdstab cds:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

crrejtab crr:: Not implemented

Rule: Default

Always reprocess

Notes: Complicated algorithm. Three changes that affects only STIS CCD. It is also dependent on the number of images to be combined, which is unknown until calibration is done.

disptab dsp:: Implemented

Rule: STIS cenwave

Match on OPT_ELEM && CENWAVE

echsctab ech:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

exstab exs:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

gactab gac:: Not implemented

Rule: Default

Always reprocess

Notes: Complicated algorithm, though determinable. Currently no reference changes, so this has been deferred until a later build.

halotab hal:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

idctab idc:: Insufficient information

Rule: Default

Always reprocess

Notes: Depends on FILTER and DIRECTION. DIRECTION is not available, as far as can be told.

imphttab imp:: Not implemented

Rule: Default

Always reprocess

Notes: No rules given and columns do not indicate any mode dependencies.

inangtab iac:: Not implemented

Rule: Default

Always reprocess

Notes: No rules given, though columns indicate mode dependencies. Currently no changes have been submitted.

lamptab lmp:: Implemented

Rule: STIS lamptab-specific

Match on OPT_ELEM && LAMPSET && SCLAMP

mlintab lin:: Implemented

Rule: STIS mlintab-specific

Match on DETECTOR

mofftab moc:: Not implemented

Rule: Default

Always reprocess

Notes: No rules given, though columns indicate mode dependencies. However, never has changed. This is no longer used, but old MAMA data might need recalibration. In this case the default action is desired.

pctab pct:: Not implemented

Rule: Default

Always reprocess

Notes: Complicated algorithm depending on other tables. The developer notes that regardless of reference file changes, the pipeline always uses the unity row for throughput. The implication is that this reference should never trigger a reprocessing. Further discussion needed.

phottab pht:: Implemented

Rule: STIS cenwave

Match on OPT_ELEM && CENWAVE

riptab rip:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

sdctab sdc:: Implemented

Rule: STIS full mode

Matches on OPT_ELEM && CENWAVE && APERTURE

sptrctab 1dt:: Implemented

Rule: STIS cenwave

Match on OPT_ELEM && CENWAVE

srwtab srw:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

tdctab tdc:: Not implemented

Rule: Default

Always reprocess

Notes: One change with severe format and purpose change. Can be implemented in a later build.

tdstab tds:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

teltab tel:: Implemented

Rule: STIS optical element

Match on OPT_ELEM

wcptab wcp:: Implemented

Rule: STIS wcptab-specific

Match on OPT_ELEM && DETECTOR

xtractab 1dx:: Implemented

Rule: STIS full mode

Matches on OPT_ELEM && CENWAVE && APERTURE

ACS

atodtab a2d:: Implemented

Rule: ACS CCD parameters

Match on CCDAMP && CCDCHIP && CCDGAIN

bpixtab bpx:: Implemented

Rule: ACS CCD parameters

Match on CCDAMP && CCDCHIP && CCDGAIN

ccdtab ccd:: Implemented

Rule: ACS ccdtab-specific

Match on BINAXIS1 && BINAXIS2

crrejtab crr:: Not implemented

Rule: Default

Always reprocess

Notes: Can be, but complicated. Next delivery. Currently only few table changes to contend with.

idctab idc:: Insufficient information

Rule: Default

Always reprocess

Notes: Cannot be implemented. V2REF and V3REF not in DADSOPS. Also, not clear this table is used any more.

imphttab imp:: Not implemented

Rule: Default

Always reprocess

Notes: Cannot be implemented. Calibration must be run.

mdriztab mdz:: Insufficient information

Rule: Default

Always reprocess

Notes: Depends on FILTER and NUMIMAGES. However, NUMIMAGES is not in any database. Examining the history, there are not changes to this table.

mlintab lin:: Implemented

Rule: ACS detector

Match on DETECTOR

oscntab osc:: Insufficient information

Rule: Default

Always reprocess

Notes: Depends on BINX and BINY which are not in DADSOPS.

pctetab cte:: Not implemented

Rule: Default

Always reprocess

Notes: Calibration must be run.

spottab csp:: Not implemented

Rule: Default

Always reprocess

Notes: Not mode dependent.

WFC3

atodtab a2d:: Implemented

Rule: ACS CCD parameters

Match on CCDAMP && CCDCHIP && CCDGAIN

bpixtab bpx:: Implemented

Rule: WFPC3 bpixtab-specific

Match on CCDCHIP

ccdtab ccd:: Implemented

Rule: ACS ccdtab-specific

Match on BINAXIS1 && BINAXIS2

crrejtab crr:: Not Implemented

Rule: Default

Always reprocess

Notes: Can be, but complicated. Next delivery. Currently only few table changes to contend with.

idctab idc:: Insufficient information

Rule: Default

Always reprocess

Notes: Cannot be implemented. V2REF and V3REF not in DADSOPS. Also, not clear this table is used any more.

imphttab imp:: Insufficient information

Rule: Default

Always reprocess

Notes: Cannot be implemented. Calibration must be run.

mdriztab mdz:: Insufficient information

Rule: Default

Always reprocess

Notes: Depends on FILTER and NUMIMAGES. However, NUMIMAGES is not in any database. Examining the history, there are not changes to this table.

oscntab osc:: Insufficient information

Rule: Default

Always reprocess

Notes: Depends on BINX and BINY which are not in DADSOPS.

Resources

J.Eisenhamer's notes

Complete reference table research

Reference Table Columns: A listing of all columns of all reference tables, by type. This also calls out when columns change within a type of reference.

Reference Table Diffs: A log of all the diffs between all references tables, by type.

Last modified 4 years ago Last modified on 11/28/14 07:27:01