Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#297 closed task (completed)

Explain addition of new reference types

Reported by: jmiller Owned by: stsci_jmiller
Priority: normal Milestone: jwst-build-6
Component: rules Version:
Severity: normal Keywords:
Cc: DMS level 4 requirements:

Description (last modified by jmiller)

The process for adding new types to CRDS consists of:

  1. Defining a "spec" file as described below and adding it to the code base of the CRDS client and CRDS server. This basically defines naming relationships and type identification based on file contents.
  1. Optionally define .tpn and _ld.tpn files which describe reference file and rmap file parameter names and acceptable values. Typically for JWST the data model schema supplant .tpns.
  1. Update the installed CRDS client and server code to reflect the addition of the new type naming and parameter constraints. A related JWST step may be to update the calibration software which defines the JWST data model and any schema for the new reference type.
  1. Submit the spec / first empty rmap to the server to prepare the rules for adding new references.
  1. Submit the first references

The attached file is an example spec. This "spec" file name is not a valid rmap file name. The name of the example spec here is of the form <instrument>_<filekind>.rmap and must appear in that form in the code. Rmap names in contrast look like "jwst_fgs_superbias_0001.rmap". It is OK/normal to submit the spec as-is and let CRDS rename it to a valid rmap name.

For this, FGS FLAT, we'd be replacing superbias and Super Bias in the example with flat and Flat Field.

In addition to naming relationships, we have to update the parkey field to define any FGS-specific matching parameter names in jwst_lib.models data model syntax. (The parameters given are for a superbias, not a flat, so I would guess they're wrong.)

For reference, MIRI FLAT might or might not be a good example for FLAT matching parameters but it's something to consider. Look on the web in the Operational References display if you want to see the current MIRI FLAT matching parameters:

https://jwst-crds-test.stsci.edu/ (TEST)
https://jwst-crds.stsci.edu/ (OPS)

For even newer/stranger types, we'd also be asking "what's the type name?" e.g. is it DARK or is it CTEDARK? Rules for type names: simple, letters and numbers, no punctuation or underscores. i.e. "flat" not "flat_field". They should begin with a letter. They are case insensitive. The basic type name is recorded in the spec as "filekind" and is also formally related to the FITS keyword used to record the bestref.

The spec defines naming relationships permanently, documents initial assumptions about the type which may become stale, and serves as the first rmap. Except for changing say "text_descr" which is a cosmetic name for the web site, generally the specs don't change once the type is added. So if for example a future .rmap updated matching parameters and changed "parkey", we would not change the spec in the code to change "parkey" there as well. And since the specs serve all generations of rules and references, it's important that the naming and type identification relationships never change; don't change them, add new different types and phase the old type out of the current rules.

Here is a description of each field in the spec, look at the attached example as well:

    derived_from                  ---     Generally  "hand made <date>",  replaced in rmaps over time with the preceding name/version of the rmap
    file_ext                      ---     Initial reference file extension
    filekind                      ---     Type name which appears in most file names and inside imaps,  also directly or closely related to bestref FITS keyword
    filetype                      ---     String appearing inside references which identify the type / filekind.
    instrument                    ---
    mapping                       ---     Always REFERENCE since this is related to the role of spec as "initial empty rmap"
    name                          ---     =Name of the spec file,  eventually replaced by the current name of the evolving rmap.
    observatory                   ---     hst or jwst
    parkey                        ---     Matching parameter names,  for HST they are FITS keywords,  for JWST they are jwst_lib.models data model names
    sha1sum                       ---      Should be computed by crds.checksum when spec is added to the code base,  basically the empty rmap's built-in integrity sum.
    suffix                        ---      Reference file uniqname suffix,  also used in .tpn filenames
    text_descr                    ---     This is the "long form" name which appears in the Operational References display on the web site.

Over time some of the fields become "stale" relative the active rmap, but the concept is that the file defines naming and miscellanea, and also documents the initial assumptions of the type and *is* the first empty rmap. So the ideal scenario is that new types are added during development of a build and the spec is built into the code. When references are available, the empty rmap/spec is submitted to the server, and then the first references are submitted as well. Adding this spec and any associated .tpn files are why new types must be defined one OPUS release or JWST build in advance. It's easy to do, but the building test and deployment process overall is slow.

For HST, .tpns are also added to the CRDS code base to enable the certifier to check the type. For JWST, CRDS inherits most type constraints from jwst_lib.models data model schema but can define additional constraints in .tpn format if absolutely required. Since references must pass the data model schema to be reliably usable by calibration code, it's important that CRDS check the schema, and it's counter-productive to duplicate any constraints defined in the DM schema.

For both projects, before the server can work with the type, the spec must be added to the server's code, any .tpn's must be added, and possibly the JWST data model must be updated on the server. Similarly, before the CRDS client used by the pipeline (to sync files) or ReDCaT (to check files on the command line) will work, the version of CRDS including the new spec and .tpns and/or data model must be deployed to where those groups use it.

The way to proceed with this is probably to have the teams define new specs (or at least the *contents* maybe not the Python-like form) and then I will add them to the code. Sometimes there is iteration involved with making the specs consistent with rmaps and consistent with the first references. e.g. FILETYPE and all the naming relationships have to be correct or nothing works. Once the first few references have gone in most problems revolve around adding new parameter values to .tpns or the DM schema or fixing reference files.

Basic checks to do when adding new types are:

  1. Check the empty .rmap using crds.certify --verbose
  2. Check new references using crds.certify --verbose
  3. Install new specs and CRDS and attempt to run crds.refactor to add new references to the empty .rmap as the server will.
  4. Submit the empty rmap to the server carefully reviewing warnings.
  5. Submit first references to the server carefully reviewing warnings.
  6. Sync new rules and references to a personal cache.
  7. Run bestrefs and/or calibration to do tests using new rules and references.

Attachments (3)

fgs_superbias.rmap (502 bytes) - added by jmiller 3 years ago.
acs_dkc.tpn (796 bytes) - added by jmiller 3 years ago.
Example HST ACS DARKCFILE parameter constraints
acs_dkc_ld.tpn (1.1 KB) - added by jmiller 3 years ago.
Example HST ACS DRKCFILE rmap parameter value constraits

Download all attachments as: .zip

Change History (11)

Changed 3 years ago by jmiller

Changed 3 years ago by jmiller

Example HST ACS DARKCFILE parameter constraints

Changed 3 years ago by jmiller

Example HST ACS DRKCFILE rmap parameter value constraits

comment:1 Changed 3 years ago by jmiller

Last edited 3 years ago by jmiller (previous) (diff)

comment:2 Changed 3 years ago by jmiller

Last edited 3 years ago by jmiller (previous) (diff)

comment:3 Changed 3 years ago by jmiller

  • Description modified (diff)

comment:4 Changed 3 years ago by jmiller

  • Description modified (diff)

comment:5 Changed 3 years ago by jmiller

  • Resolution set to completed
  • Status changed from new to closed

comment:6 Changed 3 years ago by jmiller

  • Description modified (diff)

comment:7 Changed 3 years ago by jmiller

  • Description modified (diff)

comment:8 Changed 3 years ago by jmiller

  • Description modified (diff)
Note: See TracTickets for help on using tickets.