wiki:Test/Overview

options for jwst testing

(all references to file locations are on jwcalibdev.stsci.edu unless noted)

Places you can put your tests and test data

source code:

  • installed as packagename.tests , to be called by packagename.test()
  • in the directory more_tests/ in your source tree
  • in the svn repository at file:///data1/svn_rt_jwst/general , a repository for the nightly tests; you can find this checked out at /data1/jwst_rt/general

data:

  • as package data in packagename.tests , assuming it is reasonably small
  • in your souce code under more_tests/ , for larger data, but not huge
  • file:///data1/svn_rt_jwst/general/ , a repository for the nightly tests (this is not distributed); you can find this checked out at /data1/jwst_rt/general
  • /data4/jwst_test_data for data files that are too big to keep in svn

You can find your data by:

  • looking in the tests package, if your data is installed with the test
  • looking in the current directory, if your test source is in the test repository
  • looking in $TEST_BIGDATA for data files too big to keep in svn

You can run your tests by:

  • import packagename; packagename.test()
  • "pdkrun -r ." in the more_tests directory of your souce code
  • log in to jwcalibdev and use "pdkrun -r" in /data1/jwst_rt/general

installed test() function

You can provide package.test() for your package. This is a convenient way to make a reasonable subset of your tests available to the user.

If you do this, we still want your tests to be able to send reports to the pandokia system. You can choose from one or more of these test frameworks:

  • py.test
  • nose
  • pandokia pycode/minipyt

There are instructions and examples at http://ssb.stsci.edu/testing/pandokia/docs_new/helper_testfn.html

Your tests should be implemented by a function named test() in __init__.py in your package. You should have all of your tests in a subsidiary package named package.tests.

If your tests need data, you can install *small* amounts of data in your test package. Try to keep it under 100k bytes total for all your test data.

For each top-level package installed by the nightly builds, the test system will know to run the test() function in that package (if there is one).

If your package contains nested packages that also have test functions, your top level test function should call the test functions in the other packages. I don't have an example for this, but will provide one if anybody needs to do this.

This test function should not run for more than 5 minutes. If you anticipate running longer than that, we should have a group discussion about how to support this capability.

regtest-style tests - big data and reference files

We have a test system that is modeled on the SSB regtest system. There is a repository of tests and a script that runs them regularly. It is in jwcalibdev in /data1/jwst_rt

repository that is set aside for tests. Reference files are not configuration controlled; there is a different reference file for each test machine.

We are not using the regtest XML test format. You write your test in python for any of the supported frameworks (py.test, nose, pandokia pycode). There are helper functions written in python that implement the comparisons to reference files that exist in the old XML test format. Details and examples of how to write a test with reference files are at http://ssb.stsci.edu/testing/pandokia/docs_new/helper_reference.html

You can also place regular py.test or nose style tests in these directories.

not installed, but in the source code

If you have additional tests that you want to keep in your source code, but are not to be run by the end user, you can keep a second set of tests that are not installed.

Create a directory named more_tests/ at the top level of your package.

This directory is not for tests that will be installed for the end user. These tests will be installed in the test tree in /data1/jwst_rt as part of the nightly builds with "cp -r more_tests /data1/jwst_rt/more_tests/XXX" where XXX is taken from the name of the directory where your source code is in.

For example, if jwstlib/stpipe contained a more_tests directory, the files would be copied to /data1/jwst_rt/more_tests/jwstlib/stpipe

After that, the nightly test run will execute those tests with pdkrun -r.

The destination directory is deleted each night before the tests are installed. You can write tests that depend on reference files, but you must maintain them yourself in the source code.

Remember the restrictions about binary files in the repository. (see the note about Binary Data at https://trac.assembla.com/jwst#BinaryData )

You may refer to data files in $TEST_BIGDATA . See /data4/jwst_test_data/README for details.

data distributed separately

If you have data that you will distribute to users separately from the source code, you can write tests that depend on that. Contact Mark to discuss how to get your data into the nightly test runs.

unit tests for C code

Pandokia now has a capability to run unit tests of C code. See http://ssb.stsci.edu/testing/pandokia/docs_new/runner_fctx.html or ask Mark for details.

Last modified 4 years ago Last modified on 12/17/14 16:15:00