wiki:analyzing_double_matches

Analyzing CRDS ERROR "More than one match clause matched"

At times a CRDS reference file lookup will fail with an error message indicating that more than one match occurred. This can be simulated and debugged as follows:

  1. Create a .json file which contains the required CRDS matching parameters. This should be a copy or near relative (maybe fix single quotes to double) of the matching header visible in CRDS pipeline debug output. For example, put the following into file miri_flat.json:
{ "TEST" : {
    "META.INSTRUMENT.NAME":"MIRI",
    "META.INSTRUMENT.DETECTOR": "MIRIMAGE",
    "META.SUBARRAY.NAME" : "FULL",
    "META.INSTRUMENT.FILTER" : "F1130W",
    "META.INSTRUMENT.BAND" : "N/A",
    "META.EXPOSURE.READPATT" :  "FAST",
    "META.OBSERVATION.DATE" : "2011-06-17",
    "META.OBSERVATION.TIME" : "00:00:00"
    }
}

  1. Run the crds.bestrefs tool using the .json file to define the matching parameters of the test case, e.g. like this:
% python -m crds.bestrefs --load-pickle miri_flat.json --verbosity=70 --types FLAT --new-context jwst_0186.pmap

The --load-pickle switch can be used to load either Python pickles of dataset headers or .json representations.

The form of the pickle or json files is { dataset_id : { parameter_name : parameter_value}, ... }, ... }. Parameter values are strings.

Each file can contain multiple datasets and supersedes (augments?) any similar datasets in preceding files.

Debug level 70 is a fairly verbose mode so there is a lot of largely irrelevant output which will end something like this:

.... tons of output...
CRDS  : DEBUG    Candidates:
 [(-4,
  (('MIRIMAGE', '*', 'N/A', 'FAST', 'FULL'),
   ('MIRIMAGE', 'F1130W', 'N/A', 'FAST', 'FULL'))),
 (-3, (('MIRIMAGE', 'F1130W', 'N/A', 'FAST', 'N/A'),))]
CRDS  : DEBUG    First selection failed: More than one match clause matched.
CRDS  : DEBUG    Fallback selection failed: More than one match clause matched.
CRDS  : DEBUG    No match found and reference is required: More than one match clause matched.
CRDS  : DEBUG    ------------------------------------------------------------------------------------------------------------------------
CRDS  : DEBUG    READONLY CACHE skipping config update.
CRDS  : DEBUG    getrecommendations() results:
 {'flat': 'NOT FOUND More than one match clause matched.'}
CRDS  : DEBUG    ------------------------------------------------------------------------------------------------------------------------
CRDS  : ERROR    instrument=u'MIRI' type='FLAT' data=u'TEST' ::  New: Bestref FAILED:  More than one match clause matched.
CRDS  : DEBUG    Checking updates for bad files.
CRDS  : DEBUG    Total bad files = 0
CRDS  : DEBUG    1 sources processed
CRDS  : DEBUG    0 source updates
CRDS  : INFO     1 errors
CRDS  : INFO     0 warnings
CRDS  : INFO     3 infos

This is where the overlapping match is identified:

 [(-4,
  (('MIRIMAGE', '*', 'N/A', 'FAST', 'FULL'),
   ('MIRIMAGE', 'F1130W', 'N/A', 'FAST', 'FULL'))),

-4 is a better match than -3. -4 actually has two match cases in it where "*" is (most likely) a translation of "ANY".

More than one equal weight match is not allowed for JWST so the rmap has to be fixed to resolve the overlap.

Note that these are higher level matches. Within them, additional matching occurs to resolve USEAFTER, and in the case of HST the USEAFTERs form the basis of merging the two cases. But really, overlap is most likely a problem and so not allowed for JWST where backward compatibilty with another system is not a practical necessity.

Last modified 2 years ago Last modified on 09/02/16 14:52:50