wiki:BuildFiveWork

DMS Build 5 is due for delivery to I&T on Oct 23, 2015. All calibration pipeline development should be completed by Oct. 1, with a target date for delivery to SDP on Oct. 8, so that they have 2 weeks to integrate before delivering the entire system to I&T on Oct. 23.

Functional Sets

The major functional sets that we need to deliver with Build 5 include:

  1. Spectrographic associations infrastructure
  2. Baseline Level-3 Spectrographic Pipeline - MIRI LRS fixed-slit and slitless, MIRI MRS, and NIRISS SOSS
  3. CRDS commit

Level 4 Requirements

The DMS level 4 requirements that are slated for completion in Build 5 include:

ID Requirement Comments
6 DMS shall calibrate WFS image data redundant with 492
471 The Calibration software shall have the capability of measuring the mean count rate and its uncertainty for each pixel from fitting the slopes of ramp data
484 The Calibration software shall have the capability of including information about the calibration history in the Calibrated Science Data Products
489 The Calibration software shall have the capability of calibrating Full Frame data
490 The Calibration software shall have the capability of calibrating Subarray data
491 The Calibration software shall have the capability of calibrating data taken of internal calibration sources
492 The Calibration software shall have the capability of calibrating data taken in support of Wavefront Sensing Operations regular 2a and 2b imaging processing
551 The DMS shall associate exposures taken as part of a Dither sequence
552 The DMS shall associate exposures taken as part of a Mosaic sequence
564 DMS shall initiate Level 2b processing as soon as all exposures necessary for processing the association are available move to workflow manager
565 DMS shall initiate Level 3 processing as soon as all exposures necessary for processing the association are available move to workflow manager
568 The Association Pool used to generate the Association shall be part of the Association name
650 The SI Calibration pipelines shall provide the level-2a calibrated science data set to ingest
651 The SI Calibration pipelines shall provide the level-2b calibrated science data set to ingest
664 The SI Calibration pipelines shall combine associated WFS dithered imaging mode individual exposures into a single level 3 science data product using a WSS specified algorithm

Spectroscopic Associations Infrastructure

This work includes developing the infrastructure necessary for creating association tables (from association pools) for associated spectroscopic exposures. The associated exposures could include multiple exposures of a given source that have been repeated for the purpose of dithering and can also include exposures that are intended to be used for background subtraction from other target exposures. For the latter case, the actual subtraction of target and background exposures will be carried out during level-2b processing. Multiple (e.g. dithered) exposures of the same target will be processed at level-3, for the purpose of combining the data for a given target from all exposures in which that target appears.

MIRI LRS Fixed-Slit Associations

LRS fixed-slit observations of point sources will often (usually?) be nodded to two different positions along the spatial axis of the slit, in order to do background subtraction by subtracting the nodded exposures from one another. The source can also be dithered to different positions along the dispersion direction of the slit.

The identifying parameters for MIRI LRS fixed-slit exposures will be:

  1. Template = "MIRI Low Resolution Spectroscopy"
  2. EXP_TYPE = "MIR_LRS-FIXEDSLIT"
  3. Subarray = "FULL"
  4. Filter = "P750L"
  5. Dither (APT name) = PATTTYPE = "Point Source" or "Extended Target"
  6. Target_name = "xyz"
  7. Target_id = 1,2,3,4,...

Observations using Dither = "Point Source" will have a 2-step nod, resulting in pairs of exposures. The exposures in each pair will have "dither_point_index" values of 1 and 2, and "pointing_sequence_number" values of 2 and 3 (1 corresponds to an associated target-acq exposure). The source will always be located in approximately the same place in the slit for each of the 2 pointings.

Observations using Dither = "Extended Target" can have multiple dither steps in both the spatial and dispersion directions.

Example SODRM programs using LRS fixed-slit observations are 92060 and 95070. Both use the standard point-source two-point nodding dither pattern. 92060 indicates a desire to obtain multiple exposures, but that capability is not implemented yet.

Open questions/issues:

  1. LRS fixed-slit observations can include multiple nodded pairs of exposures, but the APT does not currently have the ability to specify more than 1 pair, so the example programs like 92060 and 95070 show just a single pair per target. It's also not obvious in what order multiple exposures will be obtained: all exposures at nod 1 and then all exposures at nod 2, or nodding back and forth for each pair (1, 2, 1, 2, 1, 2, ...).
  2. Do we need/want to include the target-acq exposure in the associations, simply for informational purposes? Yes.
  3. The LRS processing schemes provided by the IDT suggest that it may be possible to obtain separate background exposures, rather than just nodding the source within the slit. We'll need to confirm whether this will be possible and, if so, account for the dedicated background exposures in LRS associations.

MIRI LRS Single-Object Slitless Spectroscopy Associations

No dithering is possible with this mode, because the source must remain fixed at the location of the slitless aperture in the field of view. Furthermore, slitless exposures will always be used for time-series observations of bright targets and hence there is no desire to produce combined results over multiple exposures (or even over multiple integrations). It will be desired, however, to produce white light curves based on an entire series of integrations/exposures.

The identifying parameters are:

  1. Template = "MIRI Low Resolution Spectroscopy"
  2. EXP_TYPE = "MIR_LRS-SLITLESS"
  3. Subarray = "SUBPRISM"
  4. Filter = "P750L"
  5. Dither (APT name) = PATTTYPE = "None"
  6. Target_name = "xyz"
  7. Target_id = 1,2,3,4,...

An example SODRM program using LRS slitless observations is 93044.

Open questions/issues:

  1. Initial extraction of info from PPS database showed dithered exposure pairs for the slitless observations, which was clearly not correct (no dithering is allowed). As of 6/8/2015 the pointing table exported from APT for 93044 shows the correct single dither point for each slitless observations.

NIRISS Single-Object Slitless Spectroscopy Associations

Associations for this mode should essentially be the same as for the MIRI LRS slitless mode, because the object must remain stationary on the detector and therefore no dithering is possible. All multiple exposures would be for the purpose of either obtaining time-series data for a given source or for combining the data for a given source in a simple stacking/averaging operation.

The identifying parameters for NIRISS SOSS observations will be:

  1. Template = "NIRISS Single-Object Slitless Spectroscopy"
  2. EXP_TYPE = "NIS_SOSS"
  3. Target_name = "abcd"
  4. Target_id = 1,2,3,4,...
  5. Subarray = "FULL", "SUBSTRIP256", "SUBSTRIP80"
  6. Pupil = "GR700XD"

Example SODRM programs using the NIRISS SOSS observations are 93054 and 93055.

Open questions/issues:

  1. SODRM proposals 93054 and 93055 have been entered into the APT, but have errors (total exposure times too long) that currently prevent them from processing through PPS and hence we don't have PPS database information to examine.

NIRSpec Fixed-Slit Associations

Point-sources may be dithered to 2 or 3 different positions in the cross-dispersion direction of the fixed slits, for the purpose of doing image-to-image background subtractions, like the MIRI LRS fixed-slit case. It's also possible to dither point sources in the along-dispersion direction in the slit. It's also possible to move a given source from the A1 to A2 slits in order to cover the wavelength gap between the detectors. So fixed-slit associations will consist of dithered sets of exposures, as well as A1/A2 slit pairings. Observations of a given target taken using different gratings will NOT be associated.

The dithered point-source exposures with have dither_point_index values of 1, 2, ... and the source will always be located in approximately the same place within the slit for each pointing index. Hence we should be able to predict a priori where the spectral trace is located based on the dither_point_index value for a given exposure.

Identifying parameters for NIRSpec fixed slit observations will be:

  1. Template = "NIRSpec Fixed Slit Spectroscopy"
  2. EXP_TYPE = "NRS_FIXEDSLIT"
  3. Target_name =
  4. Target_id =
  5. Filter = CLEAR, F070LP, F100LP, F170LP, F290LP, etc.
  6. Grating = PRISM, G140M, G235M, G395M, etc.
  7. Slit (APT name) = FXD_SLIT = "S200A1", "S200A2", "S200B1", "S400A1", "S1600A1"
  8. Subarray = "ALLSLITS" or individual slit as above
  9. Pointing_type = "SCIENCE", "TARGET_ACQUISITION"

An example SODRM program using the NIRSpec fixed slits is 92070. These observations use a single aperture with a 2-point dither pattern. Another example is 93041, which also uses a single aperture, but no dithers. Program 95110 also uses a single aperture, but uses dithers in both directions. Program 95140 uses ALLSLITS, with no dithers.

Open questions/issues:

  1. If a user chooses a single slit in APT, e.g. 'S200A1', then FXD_SLIT will contain that single slit name, but if they all then choose SUBARRAY='ALLSLITS', they'll get an exposure that contains a readout of all the slits, hence being inconsistent with the single slit named in FXD_SLIT. I guess the value of FXD_SLIT indicates the slit in which the target was placed, while they also then get data for the remaining slits for free, with who knows what sources (if any) in them.

NIRSpec MSA Associations

NIRSpec MSA observations will be the most complex associations.

Backgrounds: Observations may use separate exposures, dithered to include blank sky in the slitlets, to obtain full-frame background exposures. The processing and subtraction of the background data in this case will take place during level-3 processing. Other observations may use 2 or 3 open shutters per slitlet and dither the sources from one shutter to another, resulting in exposures that contain both sources and background. In this scenario, full-frame subtraction of each possible pair of exposures will be done during the early stages of level-2b processing, before the individual slitlet cutouts have been extracted, as well as before flat-fielding and throughput corrections have been applied. Hence associations will need to be constructed that contain the sets of dithered exposures that need to be subtracted from one another to produce multiple background-subtracted products. Each of the resulting background-subtracted products will then proceed through the remaining level-2b calibrations.

Dithers: In addition to dithering for the purpose moving sources within a given slitlet, somewhat larger dithers may also be used in order to fill in the gap in wavelength coverage between the two detectors. Each dither point with use a different MSA configuration, because the sources will have moved a fair distance on the detectors and hence a different set of slitlets is needed to observe them. Associations will need to contain these sets of large dither exposures.

Grating settings: Different gratings will often be used to observe the same (or similar) set of sources, in order to obtain different wavelength coverage. The NIRSpec team indicates, however, that observations using different gratings will NOT be combined and hence they do not need to be grouped together into an association. Each association should contain only those exposures that use a particular filter+grating combination.

Identifying parameters for MSA observations will be:

  1. Template = "NIRSpec MultiObject Spectroscopy"
  2. EXP_TYPE = "NRS_MSASPEC"
  3. Target_name =
  4. Target_id =
  5. Filter = F070LP, F100LP, F170LP, etc.
  6. Grating = G140M, G235M, G395M, etc.

Some example SODGRM programs are 93061, 93120, and 93140.

Open questions/issues:

  1. It appears that many of the APT parameters needed for MSA mode have not yet been implemented. For example, the only choices for "MSA Configuration" are "ALLOPEN" and "ALLCLOSED". There's no obvious indication of a way to specify a particular MSA configuration that involves only certain shutters/slitlets being open. Also, the comment portions of the observations contain parameters like "NTSET=2", "NSLITLET=1", "NNOD=3", "WAVEGAP=YES", which I assume are placeholders for the missing features. We need to know what the missing parameters will look like.
  1. For nodded exposures that move the target to different locations within its slitlet, how will we know where the target is for each exposure? Some information like that could be contained in the MSA configuration data, but if multiple nods/exposures use the same configuration, then that information will be static and not exposure-dependent.

MIRI MRS (IFU) Associations

MRS observing scenarios will include separate off-source exposures for performing background subtraction, background matching and subtraction amongst multiple dithered/mosaiced on-source exposures, mosaics of extended sources, and small-scale dithering for improving sampling. Observations for a given target will also include exposures taken at different wavelength sub-bands that need to be associated, so that they can be combined at the cube building stage.

For observations that only include multiple dither/mosaic exposures, a simple association listing all of the exposures for a given target is sufficient to use as input to level-3 cube building. When separate background (off-source) exposures are present, those exposures will need to be identified as such within the association. A level-2b association could be constructed in order to perform just the background subtraction operation within level-2b processing, or the background exposure(s) could be included in a level-3 association, in which case the level-3 processing will need to perform the background subtraction before doing cube building.

Identifying parameters for MRS observations will be:

  1. Template = "MIRI Medium Resolution Spectroscopy"
  2. EXP_TYPE = "MIR_MRS"
  3. Target_id = 1,2,3,4,...
  4. Target_name = "xyz"
  5. Pointing_type = "SCIENCE", "TARGET_ACQUISITION"
  6. Wavelength (APT name) = BAND = "SHORT", "MEDIUM", "LONG", "ALL"

Example SODRM programs using the MIRI IFU are 93110 and 93130. These programs use the mosaic feature to produce tiled exposures, as well as dithers within each tile.

Open questions/issues:

  1. APT does not currently allow for specifying dithers with MIRI MRS observations, hence we don't have examples of dithers in the PPS spreadsheets.
  2. How will the dedicated background exposures be identified so that we know to include them in an association? As of 6/8/2015 Rob Douglas indicates that they will likely expand the dither patterns to include background CHOP exposures, in which case the background exposure(s) would show up as an additional pointing within each set of dithered exposures and the background exposures would have a pointing_type of CHOP (while the target exposures would have SCIENCE).

NIRSpec IFU Associations

NIRSpec IFU observing scenarios will include mosaics of extended sources and dithers within the mosaic tiles. I assume that separate background exposures will also be employed, but haven't found any information yet that explicitly mentions that. I also assume that background matching will need to be employed for the dithered and mosaiced exposures. IFU observations can also be taken using different gratings, for different wavelength coverage. So far the consensus from the NIRSpec team is that data from different gratings will NOT be combined. Hence filter+grating will need to be part of the rules in the association generator, so that only exposures for a given filter+grating combo get put into each association.

Identifying parameters for NIRSpec IFU observations will be:

  1. Template = "NIRSpec IFU Spectroscopy"
  2. EXP_TYPE = "NRS_IFU"
  3. Target_id = 1,2,3,4,...
  4. Target_name = "xyz"
  5. Pointing_type = "SCIENCE", "TARGET_ACQUISITION"
  6. Filter = "F100LP", F170LP", "F290LP"
  7. Grating = "G140H", "G235H", "G395H"

Example SODRM programs using the NIRSpec IFU are 93110 and 93130. These programs use the mosaic feature to product tiled exposures and also use dithering within the tiles.

Open questions/issues:

  1. How will the dedicated background exposures be identified so that we know to include them in an association? See the comments above for MIRI MRS associations.

APT Issues Summary list of current APT issues/concerns

Baseline Level-3 Spectroscopic Pipeline

This work involves developing the infrastructure necessary for creating combined (level-3) spectroscopic data products for the following JWST spectroscopic modes:

  1. MIRI LRS fixed slit
  2. MIRI LRS single object slitless
  3. NIRISS single object slitless
  4. MIRI MRS (IFU)

The exact procedure for combining data will likely be mode-dependent.

Level-3 Source-Based Products

One of the components of the level-3 spectroscopic pipeline will need to be a task that converts calibrated level-2b exposure-based products into source-based products, for use as input to level-3 processing. At level-2b we have one calibrated product per exposure, which also implies a separate product for each detector on which data for a given target may fall (e.g. for NIRSpec fixed slit and MSA observations, the spectral trace of a given target may fall on both detectors, but that means part of the spectrum will be in one level-2b product and the other part in another level-2b product for the other detector). For build 5, we must create a tool that gathers all of the level-2b products that contain data for a given source and create a single level-3 input product that contains all of the level-2b data for that source. The data from separate level-2b products will be stored as separate extensions in the level-3 source-based product (e.g. equivalent to our MultiSlitModel data model).

NIRSpec Fixed Slit and MSA

Any nodding/dithering that was performed for the purpose of doing background subtraction will need to be handled during level-2b processing. The image-to-image background subtraction should be performed at the beginning of level-2b processing before extractions for each slit/source have been performed. Association tables will therefore be needed to carry out the level-2b processing, so that we know which exposures are involved in performing the background subtractions. So if slitlet nods were used, then level-2b processing consists of:

  1. Full-frame, image-to-image background subtraction, using associated nodded exposures. If more than 2 nods, such that more than 1 background is available, take the average of background exposures before subtracting.
  2. The remaining normal level-2b steps (assign_wcs, extract_2d, flat_field, photom) are applied to each exposure
  3. Aperture flux correction (two modes: point source and extended source)
  4. Save fully calibrated unrectified 2D spectra
  5. Create rectified 2D spectra
  6. 1D extraction

The level-2b output products will have been through the extract_2d step, so that they contain MultiSlitModel data models for one or more sources. The 2-d cutouts in each slit model will be the original unrectified (distorted) image data. These same unrectified 2-d cutouts get mapped to the level-3 input source-based products. Level-3 processing for a given source will then consist of:

  1. Slit-based background subtraction for exposures that did not use slitlet nods.
  2. Drizzle the multiple unrectified (background subtracted) 2D cutouts into a combined, rectified 2D cutout.
  3. Perform 1D extraction on the combined, rectified 2D cutouts.

More information can be found in this presentation: NIRSpec MSA Level 2 and 3 Processing (JCCWG April 14, 2015)

MIRI LRS Slitless

A description of the MIRI LRS calibration pipeline was delivered by the MIRI IDT in CDP-4: file:///grp/jwst/wit/miri/CDP-4/CDP4/PIPELINE_ALGORITHM/MIRI-DD-00006-MPI_issue1.1.pdf

Because the source must always be placed in exactly the same position in field for LRS slitless observations, there will never be dithered exposures, nor are dithers allowed via the APT observation template. Each exposure is likely to consist of many integrations (hundreds or thousands), each with only a few groups per integration, because these will be bright sources. There may be multiple exposures for a given target. The most recent LRS processing document from the IDT LRS working group indicates that 1-d spectra should be extracted from each exposure and each integration at the end of level-2b processing, and then NO level-3 processing (i.e. no combination of either the calibrated 2-d images or 1-d extracted spectra) is to be applied, because observers will only be interested in the individual integration and exposure results, for the purpose of time-series studies. Hence no level-3 processing may be needed for LRS slitless mode.

MIRI LRS Fixed Slit

Point-source observations will usually be taken using nodded pairs of observations, where the source is moved to 2 different positions in the spatial direction of the slit. We will need to form difference images of each nodded pair for a given target, which is done to accomplish background subtraction. The MIRI team indicates that the source will always be located at the same position in the slit for the 1st and 2nd nodded exposures, so we should always know where the source is located just based on the dither point index for each exposure. Once the subtracted images have been formed, the EC team is currently recommending that 1-D extractions be performed on each 2-D difference image and then combine the multiple 1-D extractions into a final spectrum. The STScI team is suggesting, on the other hand, that we might want to take the same approach as NIRSpec, which is to drizzle-combined the 2-D difference images and then do a single 1-D extraction from the combined image.

It's also possible to dither a source along the dispersion direction of the slit, so if a 1-D extraction was performed on each exposure, then we'd either have to resample the 1-D spectra when combining them in order to match up the wavelength binning, or do the resampling by combining the 2-D exposures and then doing a single 1-D extraction from the combined image.

MIRI MRS (IFU)

Details concerning level-2b and level-3 processing are contained in the presentation https://confluence.stsci.edu/download/attachments/25035542/MRSPWG_CALSPEC2nCALIFU3_10March2015.pdf?version=1&modificationDate=1426016223000&api=v2 MRS CALSPEC2 and CALIFU3 (JCCWG Mar 10, 2015)]

Level-2b processing mostly follows the steps we already have in place, namely assign_wcs, flat_field, straylight, fringe, and photom, with the possible insertion of a latents (persistence) correction in the future. Dedicated background exposures receive the same level-2b processing as on-source exposures and are then used during level-3 processing.

Level-3 (CALIFU3) processing consists of:

  1. Wavelength offsets correction (accounts for non-centering of source in slices)
  2. Spectral leak correction (uses channel 1B data to correct channel 3A data)
  3. Background subtraction: for a single exposure, a dedicated background exposure is subtracted. For dithered/mosaiced exposures, background exposure level is first matched to each source exposure and then subtracted.
  4. Diffraction loss correction
  5. Cube building
  6. 1D spectral extraction: both aperture extraction and optimal extraction
Last modified 3 years ago Last modified on 09/18/15 17:02:17