wiki:BuildFourWork

DMS Build 4 is due for delivery to I&T on April 1, 2015 (no fooling!). Calibration pipeline development work should be completed on March 1, 2015, which allows for 2 weeks of slack before we deliver to SDP on March 15, with SDP delivering the integrated system to I&T on April 1.

Coronagraphic Processing Documentation

The focus of Build 4 is building the infrastructure (and to a lesser extent preliminary algorithms, if available) for level-3 processing of coronagraphic exposures from the NIRCam, NIRISS and MIRI instruments. The following is a list of presentations and documents that provide information about the processing involved.

JWST Coronagraphic Pipeline Architecture and Data Products (JCCWG May 6, 2014)

Pipeline Workflow Diagram (JCCWG May 6, 2014)

Calibrating NIRISS' Aperture Masking Interferometry (AMI) (JCCWG May 20, 2014)

MIRI and NIRCam Coronagraphic Observations

There are many different types of exposures that will need to be compiled into a PSF library to be used when processing any given science target. The library could take the form of an association table that lists all of the exposures. The library exposures can include:

  1. exposure(s) of a science target and a reference target
  2. exposures of a science target taken at multiple roll angles
  3. a dithered set of exposures of a reference target

Observing templates will exist in the APT for a few basic use cases that include these different types of exposures, so there should be some recognizable structure and meta data passed along from the APT/PPS by which the pipeline can figure out what it's working with. Templates will include all exposures for a given target taken at different roll angles, through different filters, etc. that form an associated set.

High-Level Processing Flow

The high-level processing flow will include things like:

  1. identify all the exposures of the science target and reference PSF exposures, all of which will go into the PSF library to be used for processing a given science target
  2. align all the science target and reference target images
  3. apply an algorithm such as LOCI or KLIP to form a suitable combined reference PSF exposure for each science target exposure
  4. subtract the reference PSF exposure from the science target
  5. combine the subtracted results from multiple science target exposures
  6. generate high-level science products from the subtracted images

Details

  1. All coronagraphic mode exposures will receive the same standard level-2a and level-2b processing as normal image mode exposures (not sure exactly how the absolute flux calibration will be done though).
  2. All of the different types of exposures that can serve a role in doing PSF subtraction will go into the PSF library/association for a given science target, in order to let the pipeline logic figure out which one(s) should be used during the application of any particular subtraction technique. This means the exposures of the science target taken at different roll angles, for example, will all go into the library, as well as contemporaneous (and historical) exposures of a reference PSF target, and all the exposures resulting from the MIRI small-grid dither pattern. The pipeline logic will then select appropriate subsets of all the library exposures to be used when processing a given science target exposure. For cases where the science target appears multiple times, such as with multiple roll angles, each instance of the science target will be processed using the others as references.
  3. When processing a given science target exposure, the PSF library is searched for all candidate exposures that may be useful, hence the PSF library is a bit analogous to an association pool. The pipeline processing then uses some subset of the library exposures that it deems most appropriate to use as references for the particular science target exposure.
  4. In the pipeline built by the Coronagraphic Working Group (CWG), which works on things like HST NICMOS coronagraphic exposures, all of the 2-D images from the PSF library are loaded into a 3-D data cube (one 2-D image per plane) for use within the algorithms. Different subtraction techniques will use different subsets of the planes in the cube. Note that this implies that we'll need to somehow retain and track meta data for each 2-D image that goes into the stack (although we don't necessarily have to construct an identical 3-D data cube - we can use our own data model of some sort that naturally maintains meta data for each of the multiple images).
  5. The CWG envisions several types of additional pre-processing that could be applied to reference images after they're assembled, but before the PSF fitting/subtraction algorithms are applied:
    1. Bad pixel correction: search for and flag bad pixels that aren't already flagged via usual pipeline processing using, for example, more stringent thresholds for things like warm/hot dark current, abnormal QE, etc. They don't currently do this for any HST data, because apparently the standard flagging is sufficient, but think it might be necessary for JWST exposures.
    2. Background correction/subtraction: the background could vary from exposure to exposure and hence some kind of correction may be necessary. They currently have no idea of how to do it, however, because the small subarrays used for coronagraphic exposures don't sample the background at all.
    3. Co-adding: it might be necessary to co-add multiple exposures for faint science targets to improve the SNR to the point where image alignment and PSF fitting/subtraction can be done reliably. This option is low priority, at least for now.
  6. After the reference library has been constructed and any additional corrections have been applied to the images, all of the images need to be aligned with one another, using the primary science target exposure as the alignment reference. Alignment needs to be done to the sub-pixel level. The CWG pipeline currently uses a minimum cross-correlation method, with some sort of masking to exclude regions of the PSF that could bias the results. Not sure what type of image resampling method they use to actually produce the shifted images.
  7. The stack of aligned images is then sent into an algorithm such as LOCI or KLIP, which does some sort of linear combination (via Principle Component Analysis: PCA) to produce a combined reference PSF that is an optimal match to the science target PSF.
  8. The optimal reference PSF is then subtracted from the science target PSF image. I imagine it would be useful to save the combined PSF image and the result of subtraction as data products. These could each be simple ImageModel products.
  9. High-level science products: Once we have a subtracted science target image, we then create high-level products from it, such as:
    1. Contrast curve: take azimuthal average of 2-D subtracted target image and compute contrast vs. radius. Could be stored as table of radius vs. contrast.
    2. SNR image: compute a variance image from the the stack of reference PSF images and then ratio it with the subtracted target image to form a SNR image. Save as normal ImageModel product.

Pipeline Steps

We will assume for now that magic will take place somewhere in SDP and the association table generator, such that the calibration pipeline processing begins at the point where we have something like an association table to be used as input that contains the list of all science and reference exposures that are needed for processing a given science target. The necessary pipeline calibration steps would then consist of:

  1. Bad pixel correction (optional and perhaps not even necessary)
  2. Background subtraction (don't know how to do it yet)
  3. Co-addition of multiple exposures of science target (low priority)
  4. Alignment of all exposures in association
  5. Apply the LOCI or KLIP algorithm to the aligned exposures, which will construct an optimal PSF reference and subtract it from the science target. Output products are the PSF reference and the subtracted science target.
  6. Create high-level science products, including contrast curve and SNR map

NIRISS Aperture Masking Interferometry (AMI) using the Non-Redundant Mask (NRM)

The NIRISS AMI observing mode uses the NRM in the instrument pupil wheel to obtain high-resolution imaging. The NIRISS AMI Operations Concept Document (OCD) indicates that AMI observing scenario can contain up to 4 different types of exposures:

  1. Target acquisition exposures: If the science target is to be observed in the 80x80 (SUB80) subarray, target acquisition exposures will be required, from which the on-board software will center the science target in the field. The TA exposures will require only the normal level-2 imaging mode processing.
  1. Science target NRM exposures: These can be either full-frame or SUB80 exposures, using the NRM in the instrument pupil wheel. These exposures will likely be taken using a dither pattern and the dither pattern may include multiple exposures at each pattern point.
  1. Science target direct exposures: These can also be either full-frame or SUB80 exposures, with the NRM out of the beam. These will usually only be taken if the science target is an extended source, for the purpose of identifying spatial features during analysis. These can be obtained using a dither pattern. They will require only the normal level-2 and, if using a dither pattern, level-3 imaging mode processing.
  1. Reference target NRM exposures: These exposures will usually be taken using the same exposure parameters as the science target (e.g. same subarray and dither pattern), with the NRM in the beam. They can either be included in the observing template for the science target or can come from a separate visit.

High-Level Processing Flow

The processing flow for AMI mode observations will consist of the following:

  1. All exposures - Target Acq, science target NRM, science target direct, reference target NRM - receive normal level-2 processing.
  2. If science target direct exposures were obtained in a dither pattern, construct an association table for those exposures and produce a combined product from the dithered exposures.
  3. Construct an association table that contains ALL of the types of exposures that were included in the observation template (TA, science NRM, science direct, reference NRM). The TA and science direct exposures will not be used in any of the level-3 NRM processing, but are included in the asn table for information only.
  4. If the science NRM exposure dither pattern included multiple exposures at individual dither points, co-add the multiple exposures to produce a combined product at each dither point.
  5. If the reference NRM exposure dither pattern included multiple exposures at individual dither points, co-add the multiple exposures to produce a combined product at each dither point.
  6. Compute the sub-pixel centroid of the PSF in each science and reference NRM exposure. It is not necessary to shift/align the images with one another. Simply pass the centroid values to the subsequent LG algorithm.
  7. Run the LG fringe fitting algorithm on each of the science and reference NRM exposures. Only a subsection of each image should be passed to the algorithm, as there is no useful information beyond a radius of 8-12 pixels from the peak of the PSF and the inclusion of those data only adds noise to the algorithm. The size of the subsection should be a user-settable parameter, with a suitable default (e.g. 19 x 19 pixels) for pipeline use. Ensure that the PSF centroid value passed to the algorithm for each exposure properly reflects the new frame of the subsection. The LG algorithm computes closure phases (CP's) and squared visibilities (SqV's) for each science and reference exposure.
  8. Average the resulting CP's and SqV's across the multiple science exposures; average the CP's and SqV's for the reference exposures.
  9. Subtract the average reference CP's from the average science target CP's; divide the average science target SqV's by the average reference SqV's. This removes residual instrumental signatures.
  10. Apply a binary fitting algorithm to the resulting CP's and SqV's and produce a chi-square map of the fit results

Pipeline Steps

We assume that magic occurs somewhere in SDP and/or the association table generator to produce something like an association table that contains the list of all related exposures for a given science target observation, which serves as input to the calibration pipeline steps. Actual calibration steps would then consist of:

  1. Drizzle-combine science target direct exposures, if present
  2. Co-add multiple science target NRM exposures at each sky pointing, if necessary
  3. Co-add multiple reference target NRM exposures at each sky pointing, if necessary
  4. Compute the PSF centroid in all science and reference exposures. Output is image data model with centroid values added to meta data.
  5. Run the LG fringe fitting algorithm on all science and reference exposures. Output is fringe coeffs for each exposures (multiple tables of coeffs?).
  6. Average the fringe fitting results for multiple science and reference exposures. Output is average fringe coeffs for science and reference exposures (same format as for individual results).
  7. Subtract the average reference results from the science results. Output is subtracted fringe coeffs (same format as above).
  8. Apply binary fitting algorithm to subtracted fringe coeffs (output format is ?)

Associations Design

In a meeting held 18-Dec-2014 we decided on the following high-level design for association tables to be used in the processing of both "traditional" coronagraphy and AMI modes.

  1. The association tables for traditional coronagraphy observations will need to contain lists of all the applicable science target exposures and PSF reference exposures available from the archive. The processing will be done in such a way that each one of the potentially multiple science target exposures will be used as the "primary", with all other remaining science target and PSF target exposures serving as "references" from which to build a suitable reference PSF for subtraction. We decided to build a separate association table for each such combination. Each asn table will define the output product for a given science target exposure, with all the remaining exposures listed as input members of that product. For example, in the simple case of a science target being observed at 2 different roll angles, 2 association tables will be constructed, each one of which will define a single output product. The first such table would define an output product for the first science target exposure, with the second science target exposure and all other archival PSF exposures listed as input members. The second table would define an output product for the second science target exposure, with the first science target exposure and all other archival PSF exposures listed as input members.
  1. AMI observations can consist of up to 4 different types of exposures: target acq, science target AMI, reference target AMI, and science target direct. We don't need to do any special processing for either the TA or science target direct exposures, hence they can use normal associations, when necessary, to define a dither set and produce a level-3 combined image. The sets of science target AMI and reference target AMI exposures each need their own fringe detection processing, and then the results of the two sets need to be processed together, using the reference target results to normalize the science target results. We decided to build a single association table for such observations, with the table defining at least 1 and up to 3 output products. The first product will always be composed of the set of science target exposures. If reference target exposures exist, then there will be a second product composed of the set of reference target exposures and a third product that will be the result of combining the results of products 1 and 2 (i.e. products 1 and 2 will essentially be sub-products of 3). The AMI processing algorithms will need to have logic in place to recognize the different types of products contained in the asn table and know how to apply the necessary processing to each (e.g. first apply fringe detection to all members of products 1 and 2, and then use those results to produce product 3).
Last modified 4 years ago Last modified on 12/18/14 14:04:47