Warren, Justin E. and I started cooking up a solution to this problem early last week. We've run into some interesting (read: unforeseen) consequences associated with version-locking metapackages in Conda. SSBX-like updates are not working as intended. In the future, all "distributions" will be distributed as immutable environment files.

[I've CC'd them on this in case I've missed anything]

For example, if CRDS 8.2-1 is to be associated with a future release of JWST, it might look like this:

  1. Install fresh Anaconda/Miniconda?
  2. Install current Conda packages for JWST's pipeline
  3. Generate an environment file:
(jwst)$ conda list --explicit > jwst-0.7.0-np110py27-Linux-x86_64.dist
  1. Review the resultant environment dump and modify it if necessary to set the "distribution" requirements:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64

Pipelines and/or end-users would download the *.dist file and execute:

$ conda create -n jwst_b7 --file jwst-0.7.0-np110py27-Linux-x86_64.dist

The result is a 1:1 installation of everything we tested against for a particular release. It's far safer than using metapackages alone.

Distribution Map

Conda: 1, 2, 5, 7(?), 8 Manual: 4, 6 Special: 3


Will use the distribution method described above.


4) I am not sure about PyPi?. I assume when we start updating it again we'll base each upload on tagged releases from Git so they match our Conda package recipes. Now that SVN+Ureka builds have finally ceased I can start focusing on rewriting the scripts that handled PyPi? uploads. The black magic involved with "making a release" is pretty hardcore, so I hope to dumb it down by a few hundred orders of magnitude to make it more accessible to the devs.

5) Yes this constitutes a nightly build of spacetelescop/crds.git, but it's considered manual because no *.dist files will exist (unless we're doing a RC release like we did earlier this summer).


3) If you plan to release another version of CRDS into the wild, you can either tell me to update the recipe in astroconda-contrib to reflect the latest CRDS tag, or fork the repository and create a pull request containing all of your proposed changes (i.e. version bump, dependencies, requirements, etc).

Regards, Joe

On 6/27/2016 8:57 AM, Todd Miller wrote:

Hi Joe,

No rush on this but I'm still trying to get a grip on how CRDS makes it out to it's various installations.

Installations / distribution points (they may cross over) seem to be:

  1. HST pipeline OPUS releases (distribution? installation)
  1. JWST pipeline I&T releases (distribution? installation)
  1. Astroconda Contrib (distribution, how?)
  1. PyPi? (legacy / "my fault") (distribution)
  1. DEV installation for SSB JWST (distribution? installation, taken from spacetelescope/crds.git nightly?)
  1. Direct clone from spacetelescope/crds.git (distribution)
  1. Installation for the ReDCaT team
  1. Installation for INS pipelines

Are there summary rules for how these get tagged and/or updated?

7,8 probably boil down to choosing some prior distribution assuming you haven't heard of them.

Last modified 3 years ago Last modified on 07/31/16 09:44:02