Configuration Files

The pipeline is primarily interfaced with using config files. These are .toml, and as such are relatively human-readable. We separate out parameters that control the overall pipeline processing (config files) and ones that are specific to the system directory layout (local files).

At the very least, you should determine a list of targets:

targets = [
   'ic5332',
   'ngc0628',
   'ngc1087',
   'etc',
]

a version:

version = 'v0p8p2'

a list of bands:

bands = ['F300M']

and some steps:

steps = [
    'download',
    'lv1',
    'lv2',
    'single_tile_destripe.nircam',
    'get_wcs_adjust',
    'apply_wcs_adjust',
    'lyot_separate.miri',
    'multi_tile_destripe.nircam',
    'level_match',
    'lv3',
    'astrometric_catalog.miri',
    'astrometric_align',
    'release',
    'regress_against_previous',
]

Note that in the steps here, you can separate things out per-instrument. For instance, destriping only runs on NIRCam images, whilst anything to do with the lyot coronagraph only applies to MIRI.

There is also the option to separately image background observations. For this, append _bgr to a band:

bands = [
    'F2100W',
    'F2100W_bgr',
]

and you can also run steps separately depending on whether the observations are background or not:

steps = [
    'lyot_separate.miri.sci',
    'lyot_mask.miri.bgr',
]

This config file controls the parameters for each step. You can edit things like so:

[parameters.download]

prop_id = '2107'
product_type = [
    'SCIENCE',
]
calib_level = [
    1,
]

which will download data from Program ID 2107 (the PHANGS Cycle 1 Treasury), and only level 1 (uncal) files. For more examples, we include examples at the end of this page. For any parameters passed to the JWST pipeline itself, these should be nested as jwst_parameters, e.g.:

[parameters.lv1]

jwst_parameters.save_results = true
jwst_parameters.ramp_fit.suppress_one_group = false
jwst_parameters.refpix.use_side_ref_pixels = true

The local.toml file simply defines where things will be saved. For example,

crds_path = '/data/beegfs/astro-storage/groups/schinnerer/williams/crds/'
raw_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_raw/archive_20230711/'
reprocess_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_phangs_reprocessed/'
alignment_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_scripts/examples/2107/alignment/'
processors = 20

This should be edited to match your system layout.

Examples

Here we provide some example configuration files, for inspiration

PHANGS Cycle 1

config.toml:

# List of targets. NGC7496 goes first, since IC5332 uses the background from those obs
targets = [
#    'ngc7496',
#    'ic5332',
#    'ngc0628',
#    'ngc1087',
#    'ngc1300',
#    'ngc1365',
#    'ngc1385',
#    'ngc1433',
#    'ngc1512',
#    'ngc1566',
#    'ngc1672',
#    'ngc2835',
#    'ngc3351',
    'ngc3627',
#    'ngc4254',
#    'ngc4303',
#    'ngc4321',
#    'ngc4535',
#    'ngc5068',
]

# Version for the reprocessing
version = 'v1p2'

# Bands to consider
bands = [
    'F200W',
    'F300M',
    'F335M',
    'F360M',
    'F1000W',
    'F770W',
    'F1130W',
    'F2100W',
    'F1000W_bgr',
    'F770W_bgr',
    'F1130W_bgr',
    'F2100W_bgr',
]

# Steps. These can/should be different
# for NIRCam and MIRI, and we can distinguish them here

# We can also distinguish between steps that may be
# different for science/background obs. This is something
# like step.miri.sci/step.miri.bgr
steps = [
    'download',
    'lv1',
    'single_tile_destripe.nircam',
    'lv2',
    'get_wcs_adjust',
    'apply_wcs_adjust',
    'lyot_separate.miri',
    'level_match',
    'multi_tile_destripe.nircam',
#    'psf_model',
    'lv3',
    'astrometric_catalog.miri',
    'astrometric_align',
    'anchoring',
    'psf_matching',
    'release',
    'regress_against_previous',
]

# Parameters for downloading data. This just downloads
# the raw science
[parameters.download]

prop_id = '2107'
product_type = [
    'SCIENCE',
]
calib_level = [
    1,
]

[parameters.move_raw_obs]

[parameters.move_raw_obs.obs_to_skip]

# Failed NGC1566 observation
ngc1566 = 'jw02107007003_02105_00003*'

# New IC5332 MIRI offs, which we need to skip
ic5332.sci = 'jw02107042001_0?2*'

# New NGC4303 MIRI offs, which we need to skip
ngc4303.sci = 'jw02107044001_0?2*'

# New NGC4321 MIRI offs, which we need to skip
ngc4321.sci = 'jw02107043001_0?2*'

# For IC5332, pull the backgrounds from NGC7496 but only for science images
[parameters.move_raw_obs.extra_obs_to_include]
ic5332.sci.ngc7496 = 'jw02107041001_0?2'

[parameters.lv1]

jwst_parameters.save_results = true
jwst_parameters.ramp_fit.suppress_one_group = false
jwst_parameters.refpix.use_side_ref_pixels = true

[parameters.lv2]

bgr_check_type = 'parallel_off'
bgr_background_name = 'off'
bgr_observation_types = [
    'miri',
]

jwst_parameters.save_results = true
jwst_parameters.bkg_subtract.save_combined_background = true
jwst_parameters.bkg_subtract.sigma = 1.5

[parameters.get_wcs_adjust]

bands = [
    'F300M',
    'F1000W',
]

group_dithers = [
    'nircam',
    'miri',
]

[parameters.get_wcs_adjust.tweakreg_parameters]

starfinder = 'iraf'
brightest = 500
snr_threshold = 3
expand_refcat = true
peakmax.nircam = 20
roundlo.nircam = -0.5
roundhi.nircam = 0.5

# Parameters to get wcs_adjust shifts
minobj = 3
sigma = 2
searchrad = 2
separation.miri = 1
separation.nircam = 2
tolerance = 0.3
use2dhist = true
fitgeometry = 'shift'
nclip = 5

# Tweak boxsize, so we detect objects in diffuse emission
bkg_boxsize.nircam_short = 100
bkg_boxsize.nircam_long = 100
bkg_boxsize.miri = 25

[parameters.lyot_mask]
method = 'mask'

[parameters.single_tile_destripe]
quadrants = true
vertical_subtraction = true
filter_diffuse = true
dilate_size = 7
destriping_method = 'median_filter'
vertical_destriping_method = 'row_median'
filter_extend_mode = "reflect"
filter_scales = [3, 7, 15, 31, 63, 127, 255, 511]

[parameters.multi_tile_destripe]

quadrants = true
weight_type = 'ivm'
do_large_scale = true
weight_method = 'mean'

[parameters.level_match]
do_sigma_clip = false
weight_method = 'rms'

[parameters.lv3]

tweakreg_group_dithers = [
    'nircam_short',
    'nircam_long',
    'miri',
]
skymatch_group_dithers = [
    'miri',
    'nircam_short',
    'nircam_long',
]

bgr_check_type = 'parallel_off'

[parameters.lv3.jwst_parameters]
save_results = true

[parameters.lv3.jwst_parameters.tweakreg]

# Skip MIRI wavelengths, since we have already
# solved for this and this can make things worse
skip.F770W = true
skip.F1000W = true
skip.F1130W = true
skip.F2100W = true

starfinder = 'iraf'
brightest.nircam_short = 125
brightest.nircam_long = 500
brightest.miri = 500
snr_threshold = 3
expand_refcat = true
fitgeometry = 'shift'
minobj = 3
peakmax.nircam = 20
roundlo.nircam = -0.5
roundhi.nircam = 0.5

# Take relatively tight tolerances since we've already shifted
# close to the correct solution
separation.miri = 1
separation.nircam = 2
sigma = 2
tolerance.nircam_short = 0.3
tolerance.nircam_long = 0.3
tolerance.miri = 1
use2dhist = false

# Tweak boxsize, so we detect objects in diffuse emission
bkg_boxsize.nircam_short = 100
bkg_boxsize.nircam_long = 100
bkg_boxsize.miri = 25

[parameters.lv3.jwst_parameters.skymatch]

# Turn off matching since we've done it already
skymethod = 'global'
subtract = true
skystat = 'median'
nclip.nircam = 20
nclip.miri = 10
lsigma.nircam = 3
lsigma.miri = 1.5
usigma.nircam = 3
usigma.miri = 1.5

[parameters.lv3.jwst_parameters.outlier_detection]
in_memory = true

[parameters.lv3.jwst_parameters.resample]
rotation = 0.0
in_memory = true

[parameters.lv3.jwst_parameters.source_catalog]
snr_threshold = 2
npixels = 5
bkg_boxsize = 25
deblend = true

[parameters.astrometric_catalog]
snr = 10
starfind_method = "iraf"

[parameters.astrometric_catalog.starfind_parameters]
sharplo = 0.2
sharphi = 1.0
roundlo = -0.5
roundhi = 0.5

[parameters.astrometric_align]

[parameters.astrometric_align.align_mapping]
# Map everything to F1000W, since it has the most point sources
F770W = 'F1000W'
F1130W = 'F1000W'
F2100W = 'F1000W'

F770W_bgr = 'F1000W_bgr'
F1130W_bgr = 'F1000W_bgr'
F2100W_bgr = 'F1000W_bgr'

[parameters.astrometric_align.catalogs]
ic5332 = 'ic5332_agb_cat.fits'
ngc0628 = 'ngc0628_agb_cat.fits'
ngc1087 = 'ngc1087_agb_cat.fits'
ngc1300 = 'ngc1300_agb_cat.fits'
ngc1365 = 'ngc1365_agb_cat.fits'
ngc1385 = 'ngc1385_agb_cat.fits'
ngc1433 = 'ngc1433_agb_cat.fits'
ngc1512 = 'ngc1512_agb_cat.fits'
ngc1566 = 'ngc1566_agb_cat.fits'
ngc1672 = 'ngc1672_agb_cat.fits'
ngc2835 = 'ngc2835_agb_cat.fits'
ngc3351 = 'ngc3351_agb_cat.fits'
ngc3627 = 'ngc3627_agb_cat.fits'
ngc4254 = 'ngc4254_agb_cat.fits'
ngc4303 = 'ngc4303_agb_cat.fits'
ngc4321 = 'ngc4321_agb_cat.fits'
ngc4535 = 'ngc4535_agb_cat.fits'
ngc5068 = 'ngc5068_agb_cat.fits'
ngc7496 = 'ngc7496_agb_cat.fits'

# Initial pass to get decent shifts for absolute astrometry
[parameters.astrometric_align.tweakreg_parameters.iteration1]

# Set quite a large search radius
searchrad.miri = 10
searchrad.nircam_long = 20
searchrad.nircam_short = 40
separation = 1

# This is for NIRCam, NGC3627. May generally be OK, but untested
tolerance = 2
# The default
#tolerance = 1

use2dhist = true
fitgeom = 'shift'
nclip = 5
sigma = 3

# Residual shifts should be small, so use tight parameters here. Allow for rotation
[parameters.astrometric_align.tweakreg_parameters.iteration2]
searchrad = 2
separation = 1
tolerance = 0.5
use2dhist = false
fitgeom = 'rshift'
nclip = 5
sigma = 3

[parameters.anchoring]

internal_conv_band = 'F2100W'

[parameters.anchoring.ref_band]
miri = 'F770W'
nircam = 'F300M'

# List external bands in preference order
[parameters.anchoring.external_bands]
miri = [
    'irac4_atgauss4',
    'w3_atgauss15',
]
nircam = [
    'irac1_atgauss4',
    'w1_atgauss7p5',
    'irac2_atgauss4',
    'w2_atgauss7p5',
]

[parameters.psf_matching]

target_bands = [
    "gauss0p85",
    "gauss0p90",
    "gauss1",
]

[parameters.release]

move_tweakback = true
move_backgrounds = true
move_psf_matched = true
move_diagnostic_plots = true

[parameters.regress_against_previous]
prev_version = 'v1p1p1'

local.toml:

crds_path = '/data/beegfs/astro-storage/groups/schinnerer/williams/crds/'
crds_context = 'jwst_1201.pmap'
raw_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_raw/2107/archive_20240211/'
reprocess_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_phangs_reprocessed/'
alignment_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/pjpipe/config/2107/alignment/'
webb_psf_data = '/data/beegfs/astro-storage/groups/schinnerer/williams/webbpsf-data'
anchor_ref_dir =   '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_data/ref_images/'
kernel_dir = '/data/beegfs/astro-storage/groups/schinnerer/williams/jwst_data/kernels/'
processors = 20

PHANGS Cycle 2

config.toml:

# List of targets
targets = [
    'ngc1511',
    'ngc2283',
    'ngc1637',
    'ngc1792',
    'ngc1808',
    # 'ngc1068', # run separately
    'ic5273',
    'ngc7456',
    'ngc1809',
    'ngc1546',
    'ngc1559',
    'ngc2090',
    'ngc1097',
    'ngc2566',
]

# Version for the reprocessing
version = 'v0p1p6'

# Bands to consider
bands = [
    'F150W',
    'F187N',
    'F200W',
    'F300M',
    'F335M',
    'F770W',
    'F2100W',
    'F770W_bgr',
    'F2100W_bgr',
]

# Steps. These can/should be different
# for NIRCam and MIRI, and we can distinguish them here

steps = [
    'download',
    'gaia_query',
    'lv1',
    'single_tile_destripe.nircam',
    'lv2',
    'get_wcs_adjust',
    'apply_wcs_adjust',
#    'lyot_mask.miri',
    'lyot_separate.miri',
    'level_match',
    'multi_tile_destripe.nircam',
#    'psf_model',  # TODO
    'lv3',
    'astrometric_catalog.miri',
    'astrometric_align',
    'anchoring',
    'psf_matching',
    'release',
    'regress_against_previous',
]

# Parameters for downloading data. This just downloads
# the raw science
[parameters.download]

prop_id = '3707'
product_type = [
    'SCIENCE',
]
calib_level = [
    1,
]

[parameters.gaia_query]

radius = 20

[parameters.lv1]

jwst_parameters.save_results = true
jwst_parameters.ramp_fit.suppress_one_group = false
jwst_parameters.refpix.use_side_ref_pixels = true

[parameters.lv2]

bgr_check_type = 'parallel_off'
bgr_background_name = 'off'
bgr_observation_types = [
    'miri',
]

jwst_parameters.save_results = true
jwst_parameters.bkg_subtract.save_combined_background = true
jwst_parameters.bkg_subtract.sigma = 1.5

[parameters.get_wcs_adjust]

method = "tweakreg"

bands = [
    'F300M',
    'F770W',
]

group_dithers = [
    'nircam',
    'miri',
]

#custom_catalog_function = 'constrained_diffusion'
#custom_catalog_function_kwargs = {roundlo = 0.0, roundhi = 0.1, sharplo = 0.6, sharphi = 1.1}

[parameters.get_wcs_adjust.tweakreg_parameters]

starfinder = 'iraf'
brightest.nircam = 500
brightest.miri = 100
snr_threshold = 5
expand_refcat = true
peakmax.nircam = 20
use_custom_catalogs = true

# Parameters to get wcs_adjust shifts
minobj = 3
sigma = 2
searchrad = 2
separation.miri = 1
separation.nircam = 2
tolerance = 0.3
use2dhist = true
fitgeometry = 'shift'
nclip = 5

# Tweak boxsize, so we detect objects in diffuse emission
bkg_boxsize.nircam_short = 100
bkg_boxsize.nircam_long = 100
bkg_boxsize.miri = 25

[parameters.lyot_mask]
method = 'mask'

[parameters.single_tile_destripe]
quadrants = true
vertical_subtraction = true
filter_diffuse = true
dilate_size = 7
destriping_method = 'median_filter'
vertical_destriping_method = 'row_median'
filter_extend_mode = "reflect"
filter_scales = [3, 7, 15, 31, 63, 127, 255, 511]

[parameters.multi_tile_destripe]
quadrants = true
weight_type = 'ivm'
do_large_scale = true
large_scale_filter_extend_mode = "reflect"

[parameters.level_match]

fit_type_dithers.nircam = "level"
fit_type_mosaic_tiles.nircam = "level"

fit_type_dithers.miri = "level"
fit_type_recombine_lyot = "level"
fit_type_mosaic_tiles.miri = "level"

recombine_lyot = true
combine_nircam_short = true
do_sigma_clip = false
weight_method = 'rms'

[parameters.lv3]

tweakreg_group_dithers = [
    'nircam_short',
    'nircam_long',
    'miri',
]
skymatch_group_dithers = [
    'miri',
    'nircam_short',
    'nircam_long',
]

bgr_check_type = 'parallel_off'

[parameters.lv3.jwst_parameters]
save_results = true

[parameters.lv3.jwst_parameters.tweakreg]

# Skip MIRI wavelengths, since we have already
# solved for this
skip.F770W = true
skip.F2100W = true

# align_to_gaia = false # no longer a tweakreg parameter
starfinder = 'iraf'
brightest.nircam_short = 125
brightest.nircam_long = 500
brightest.miri = 500
snr_threshold = 3
expand_refcat = true
fitgeometry = 'shift'
minobj = 3
peakmax.nircam = 20
roundlo.nircam = -0.5
roundhi.nircam = 0.5
roundlo.miri = -0.5
roundhi.miri = 0.5

# Take relatively tight tolerances since we've already shifted
# close to the correct solution
separation.miri = 1
separation.nircam = 2
tolerance.nircam_short = 0.3
tolerance.nircam_long = 0.3
tolerance.miri = 1
use2dhist = false

# Tweak boxsize, so we detect objects in diffuse emission
bkg_boxsize.nircam_short = 100
bkg_boxsize.nircam_long = 100
bkg_boxsize.miri = 25

[parameters.lv3.jwst_parameters.skymatch]

# Turn off matching since we've done it already
skymethod = 'global'
subtract = true
skystat = 'median'
nclip.nircam = 20
nclip.miri = 10
lsigma.nircam = 3
lsigma.miri = 1.5
usigma.nircam = 3
usigma.miri = 1.5

[parameters.lv3.jwst_parameters.outlier_detection]
in_memory = true

[parameters.lv3.jwst_parameters.resample]
rotation = 0.0
in_memory = true

[parameters.lv3.jwst_parameters.source_catalog]
snr_threshold = 2
npixels = 5
bkg_boxsize = 25
deblend = true

[parameters.astrometric_catalog]
snr = 10
starfind_method = "iraf"
starfind_prefilter = "constrained_diffusion"

[parameters.astrometric_catalog.dao_parameters]
sharplo = 0.2
sharphi = 1.0
roundlo = -0.5
roundhi = 0.5

[parameters.astrometric_align]

[parameters.astrometric_align.align_mapping]
# Map F2100W to F770W
F2100W = 'F770W'
F2100W_bgr = 'F770W_bgr'

# F150W = 'F300M'
# F187N = 'F335M'

[parameters.astrometric_align.catalogs]
ic5273 = 'Gaia_DR3_ic5273.fits'
ngc1559 = 'ngc1559_agb_cat.fits'
ngc7456 = 'Gaia_DR3_ngc7456.fits'
ngc1809 = 'Gaia_DR3_ngc1809.fits'
ngc1546 = 'Gaia_DR3_ngc1546.fits'
ngc1511 = 'Gaia_DR3_ngc1511.fits'
ngc2090 = 'Gaia_DR3_ngc2090.fits'
ngc1097 = 'Gaia_DR3_ngc1097.fits'
ngc1068 = 'Gaia_DR3_ngc1068.fits'
ngc1808 = 'Gaia_DR3_ngc1808.fits'
ngc1792 = 'Gaia_DR3_ngc1792.fits'
ngc1637 = 'Gaia_DR3_ngc1637.fits'
ngc2283 = 'Gaia_DR3_ngc2283.fits'
ngc2566 = 'Gaia_DR3_ngc2566.fits'

# Initial pass to get decent shifts for absolute astrometry
[parameters.astrometric_align.tweakreg_parameters.iteration1]

# Set quite a large search radius
searchrad.miri = 10
searchrad.nircam_long = 20
searchrad.nircam_short = 40
separation = 1
tolerance = 1
use2dhist = false # Changed 13 Feb 24 -- to remove catastrophic fails
fitgeom = 'shift'
nclip = 5
sigma = 3

# Second iteration with tightened up parameters to figure out any residual shifts left
[parameters.astrometric_align.tweakreg_parameters.iteration2]
searchrad = 2
separation = 1
tolerance = 0.5
use2dhist = false
fitgeom = 'rshift'
nclip = 5
sigma = 3


[parameters.anchoring]

internal_conv_band = 'F2100W'

[parameters.anchoring.ref_band]
miri = 'F770W'
# nircam = 'F300M'

# List external bands in preference order
[parameters.anchoring.external_bands]
miri = [
    'irac4_atgauss4',
    'w3_atgauss15',
]
nircam = [
    'irac1_atgauss4',
    'w1_atgauss7p5',
    'irac2_atgauss4',
    'w2_atgauss7p5',
]

[parameters.psf_matching]

target_bands = [
    "gauss0p85",
    "gauss0p90",
    "gauss1",
]


[parameters.release]

move_tweakback = true
move_backgrounds = true
move_psf_matched = true
move_diagnostic_plots = true

[parameters.regress_against_previous]
prev_version = 'v0p1p5'