Reference/API

Pipeline

class pjpipe.PJPipeline(config_file=None, local_file=None)

Overall wrapper for pjpipe.

Parameters:

• config_file – Either string to location of config.toml file, or dict of preloaded toml

• local_file – Either string to location of local.toml file, or dict of preloaded toml

do_pipeline()

Anchoring

class pjpipe.AnchoringStep(target, bands, in_dir, ref_dir, procs, in_step_ext, out_step_ext='i2d_anchor', in_subdir=None, out_subdir=None, kernel_dir=None, ref_band=None, external_bands=None, internal_conv_band=None, reproject_func='interp', simplify_labels=True, overwrite=False, galaxy_table=None, reff_factor=2.0, percentile_range=None)

Anchor aligned data to the external images

Will convolve data to a common resolution given a reference image, bin and fit for an offset between them. Can be used with external images to anchor to known fluxes, or JWST images for internal consistency.

N.B. for external files the naming scheme expects something like [target]_[resolution].fits, where resolution can be something like ‘irac1’, or if pre-convolved, then maybe ‘irac1_atgauss15’. Kernels should be named something like [resolution]_to_[resolution].fits. This is non-negotiable!

Parameters:

• target – Target to consider

• bands – Bands to consider

• in_dir – Input directory

• ref_dir – Directory for reference images

• procs – Number of processes to run in parallel

• in_step_ext – Filename extension for the input files

• out_step_ext – Filename extension for output files. Defaults to “i2d_anchor”

• in_subdir – Where files are located within the target directory (something like ‘lv3’)

• out_subdir – Where to save intermediate files (something like ‘anchored’)

• kernel_dir – Where kernels are located

• ref_band – If internally anchoring, these are the bands to lock to

• external_bands – If externally anchoring, these are a list (in preference order) of resolutions to lock to (e.g. irac1, irac1_atgauss4p5)

• internal_conv_band – For internal anchoring, we use this common resolution.

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• simplify_labels – If True, will strip any convolution info (_atgaussX) from the labels

• overwrite – Whether to overwrite or not

• galaxy_table – Path to galaxy properties table (CSV) for dynamic intensity range calculation. If None, uses hard-coded intensity ranges (original behavior). Defaults to None. CSV must have columns: name, ra, dec, posang, inclination, distance_mpc, size_reff

• reff_factor – Factor to multiply effective radius by for intensity range calculation. Defaults to 2.0

• percentile_range – [min, max] percentiles to calculate intensity range. Defaults to [15, 60] if None is provided. Only used if galaxy_table is provided.

do_step()

Run anchoring step

parallel_anchoring(file, external=True, internal_reference=None)

Parallelize applying anchoring to external images

Does relevant convolutions and fits for the anchoring offsets

Parameters:

• file – File and (potentially) reference to apply anchoring

• external – anchoring to external (True) or internal (False) images

• internal_reference – path to the internal reference image (None if external = True)

run_step(files)

Wrap paralellism around applying anchoring to external images

This will loop over NIRCam/MIRI files (excluding backgrounds) and perform anchoring as set up by the user. Will put out a table of fit values and the anchored images at the end

Parameters:

files – List of files to process

Apply WCS Adjust

class pjpipe.ApplyWCSAdjustStep(wcs_adjust, in_dir, out_dir, step_ext, procs, overwrite=False)

Apply WCS adjustments to images

Parameters:

• wcs_adjust – Dictionary for WCS adjustments

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run applying the WCS adjustments

parallel_wcs_adjust(file)

Parallelise applying WCS adjustments

Parameters:

file – File to apply WCS corrections to

run_step(files, procs=1)

Wrap paralellism around applying WCS adjusts

Parameters:

• files – List of files to mask lyot in

• procs – Number of parallel processes to run. Defaults to 1

Astrometric Align

class pjpipe.AstrometricAlignStep(target, bands, progress_dict, target_dir, catalog_dir, step_ext, procs, step_parameters, catalogs=None, align_mapping_mode='shift', align_mapping=None, tweakreg_parameters=None, reproject_func='interp', overwrite=False)

Perform absolute astrometric alignment

There are a number of modes here. The simplest is by matching a catalog of sources, using tweakreg. Alternatively, we can either apply calculated shifts to other mosaics, or attempt to match via cross-correlation.

Parameters:

• target – Target to consider

• bands – Bands to consider

• catalog_dir – Directory of alignment catalogs

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• catalogs – Dictionary for the external alignment catalogs

• align_mapping_mode – If locking to other JWST image, method to use. Option is “shift” (pull the tweakreg solution from the existing file), or “cross-corr” (do some cross-correlation between the images)

• tweakreg_parameters – Dictionary of parameters to pass to tweakreg for the standard alignment

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• overwrite – Whether to overwrite or not. Defaults to False

align_to_aligned_image(band, in_dir, align_mapping=None, align_mapping_mode='shift', reproject_func='interp')

Align to a pre-aligned image

This will align to a pre-aligned image, either using cross-correlation or by pulling out the shift values and matrix from tweakreg (default)

Parameters:

• band – Band to consider

• in_dir – Input directory

• align_mapping – Mapping to use to align to

• align_mapping_mode – If locking to other JWST image, method to use. Option is “shift” (pull the tweakreg solution from the existing file), or “cross-corr” (do some cross-correlation between the images)

• reproject_func – Which reproject function to use. Defaults to ‘interp’

do_step()

Run absolute astrometric alignment

do_step_band(band)

Run absolute astrometric alignment per-band

Parameters:

band – Band to consider

move_tweakback_files(files, shift=None, matrix=None, ref_tpwcs=None)

Wrapper to parallelise up tweakback

Parameters:

• files – List of files to tweakback

• shift – shift for tweakback. Defaults to None

• matrix – rotation/skew matrix. Defaults to None

• ref_tpwcs – WCS defining the plane in which the shift/matrix was defined. Defaults to None

tweakreg_align(band, in_dir, catalogs=None, cat_suffix='cat.ecsv', tweakreg_parameters=None)

Align using tweakreg

Parameters:

• band – Band to consider

• in_dir – Input directory

• catalogs – Dictionary for the external alignment catalogs

• cat_suffix – extension for the existing catalog. Defaults to “cat.ecsv”, which is the pipeline default

• tweakreg_parameters – Dictionary of parameters to pass to tweakreg for the standard alignment

Astrometric Catalog

class pjpipe.AstrometricCatalogStep(target, band, in_dir, snr=5, starfind_method='dao', starfind_prefilter=None, starfind_parameters=None, dao_parameters=None, overwrite=False)

Generate a catalog for absolute astrometric alignment

Parameters:

• in_dir – Directory to search for files

• snr – SNR to detect sources. Defaults to 5

• starfind_method – Method for detecting sources in image. Options are given be ALLOWED_STARFIND_METHODS

• starfind_parameters – Dictionary of parameters to pass to the starfinder

• starfind_prefilter – Method for prefiltering the image before starfinding

• dao_parameters – Dictionary of parameters to pass to DAOFinder

• overwrite – Overwrite or not. Defaults to False

do_step()

Run astrometric catalog step

generate_astro_cat(file)

Generate an astrometric catalogue using given starfinder

Parameters:

file – File to run starfinder on

Download

class pjpipe.DownloadStep(target=None, prop_id=None, download_dir=None, procs=1, radius=None, telescope='JWST', instrument_name=None, dataproduct_type=None, calib_level=None, extension=None, do_filter=True, breakout_targets=False, product_type=None, filter_gs=True, login=False, api_key=None, overwrite=False)

Query and download data from MAST

Parameters:

• target – Target name. Defaults to None, which will use the proposal ID to figure out names

• prop_id – Proposal identifier. Defaults to None, but at least one of target, prop_id needs to be set

• download_dir – Where to download to. Defaults to current directory if not set

• procs – Number of processes to run in parallel. Defaults to 1

• radius – Radius to search from centre of target in arcmin. Defaults to None, which uses astroquery default

• telescope – Telescope to use. Defaults to ‘JWST’

• instrument_name – Something like ‘NIRCAM/IMAGE’, to filter down what products you want. Defaults to None (no filtering)

• dataproduct_type – These are things like ‘cube’, ‘image’ etc. Defaults to None (no filtering)

• calib_level – Calibration levels to filter. Defaults to None (no filtering)

• extension – File extensions to filter by. Defaults to None (no filtering)

• do_filter – Whether to filter dataproduct_type/calib_level/extension. Defaults to True

• breakout_targets – If targets is None, this will either download the whole proposal to a single directory (False), or breakout each target name to its own directory (True). Defaults to False

• product_type – Product types to download. Defaults to None (no filtering)

• filter_gs – Whether to filter guide stars from the observations. Defaults to True

• login – For proprietary data, set to True. Defaults to False

• api_key – If login is True, supply API key here

• overwrite – If True, overwrites existing data. Defaults to False

do_step()

Run download step

get_products(obs_list)

Get products from an observations list

Parameters:

obs_list – List of observations to get products from

run_archive_query()

Query archive, trimming down as requested on instrument etc.

run_download(max_retries=5)

Download a list of observations

Parameters:

max_retries (int) – Maximum number of retries before failing. Defaults to 5

verify_integrity()

Check files have downloaded successfully

Gaia Query

class pjpipe.GaiaQueryStep(target, out_dir, row_limit=-1, radius=15, overwrite=False)

Query Gaia to get sources for astrometric alignment

Parameters:

• target – Target to get GAIA sources for

• out_dir – Where to save GAIA catalog to

• row_limit – Maximum number of rows to return. Defaults to -1, which will return everything

• radius – Radius in arcmin to query. Defaults to 15

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run Gaia catalog query

make_catalog(out_name)

Create a Gaia catalog

Parameters:

out_name – Name to save catalog to

Get WCS Adjust

class pjpipe.GetWCSAdjustStep(directory, progress_dict, target, alignment_dir, procs=1, bands=None, method='tweakreg', alignment_catalogs=None, group_dithers=None, degroup_nircam_modules=False, tweakreg_parameters=None, reproject_func='interp', overwrite=False, custom_catalog_function=None, custom_catalog_function_kwargs=None)

Gets a table of WCS corrections to apply to visit groups

Experience has shown that the relative JWST guide star uncertainty is very small, but there are significant absolute corrections between guide stars. Thus, we can use the same visit as a correction for all visits, for example using F1000W at F2100W where tweakreg doesn’t work so well.

Here, we take some template bands and loop over with tweakreg, writing out a table of shifts/matrices to apply to other bands. For multiple dithers etc., will take an average correction.

Alternatively, we can take a cross-correlation approach. Here, we instead make a stacked image of the dithers for each mosaic tile, then loop over and cross-correlate to get a shift. This should work better than tweakreg for bands where there aren’t many stars present, e.g. F770W.

Parameters:

• directory – Directory of target

• progress_dict – The progress dictionary the pipeline builds up. This is used to figure out what subdirectories we should be looking in

• target – Target to consider

• alignment_dir – Directory for alignment catalogs

• bands – List of target bands to pull corrections out for

• method – Method to align images together. Can be “tweakreg” (default) or cross_corr. N.B. for “cross_corr”, all dithers will be stacked by default to calculate the cross-correlation. This can also be a dictionary to distinguish between e.g. NIRCam and MIRI, like {‘nircam’: ‘tweakreg’, ‘miri’: ‘cross_corr’}

• alignment_catalogs – Dictionary mapping targets to alignment catalogs

• procs – Number of processes to run in parallel. Defaults to 1

• group_dithers – Which band type (e.g. nircam) to group up dithers for and find a single correction. Defaults to None, which won’t group up anything

• degroup_nircam_modules – Whether to degroup NIRCam A and B modules. Currently, the WCS is inconsistent between the two, so should probably be set to True if you see “ghosting” in the final mosaic. Defaults to False

• tweakreg_parameters – Dictionary of parameters to pass to tweakreg. Defaults to None, which will use observatory defaults

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run the WCS adjust step

get_visit_transforms(in_dir)

Get visit transforms out of the shifted files

Parameters:

in_dir – Directory containing files with shifts

make_stacked_images(dithers, in_dir, out_dir, procs=1)

Function to parallellise up making stacked dither images

Parameters:

• dithers – List of dithers to go

• in_dir – Where to find files

• out_dir – Where to save stacked images to

• procs – Number of simultaneous processes to run. Defaults to 1

parallel_make_stacked_image(dither, in_dir, out_dir)

Light wrapper around parallelising the stacked image

Parameters:

• dither – Dither group to consider

• in_dir – Where to find files

• out_dir – Where to save stacked images to

run_cross_corr(band_full, out_dir)

Get transforms per-visit, using a cross-correlation between stacked dithers

Parameters:

• band_full – Band to consider

• out_dir – Output directory to save files with shifts to

run_tweakreg(band_full, out_dir)

Run tweakreg to get shifts

Parameters:

• band_full – Band to consider

• out_dir – Output directory to save files with shifts to

Level Match

class pjpipe.LevelMatchStep(in_dir, out_dir, step_ext, procs, band, fit_type_dithers='level', fit_type_recombine_lyot='level', fit_type_combine_nircam_short='level', fit_type_mosaic_tiles='level', recombine_lyot=False, combine_nircam_short=False, do_local_subtraction=True, sigma=3, npixels=3, dilate_size=7, max_iters=20, max_points=10000, do_sigma_clip=False, weight_method='equal', min_area_percent=0.002, min_linear_frac=0.2, rms_sig_limit=2, reproject_func='interp', overwrite=False)

Perform background matching between tiles

This step performs background matching by minimizing the per-pixel differences between overlapping tiles. It does this first for dither groups, before creating a stacked image of these (to maximize areal coverage) and minimizing between all stacked images within a mosaic. This is necessary for observations that don’t really have a background, and performs significantly better than the JWST implementation.

N.B. If you use this, skymatch in the level 3 pipeline stage should be global or off, to avoid undoing this work

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of parallel processes to run

• band – JWST band

• fit_type_dithers – What kind of fit to do to match levels between dithers in a single mosaic tile. Options are ‘level’ (the default, just fits a single offset), and ‘level+slope’ (which will fit a plane)

• fit_type_recombine_lyot – What kind of fit to do to match levels between science and lyot in a single mosaic tile. Options are ‘level’ (the default, just fits a single offset), and ‘level+slope’ (which will fit a plane)

• fit_type_combine_nircam_short – What kind of fit to do to match levels between the four NIRCam short chips. Options are ‘level’ (the default, just fits a single offset), and ‘level+slope’ (which will fit a plane)

• fit_type_mosaic_tiles – What kind of fit to do to match levels between mosaic tiles. Options are ‘level’ (the default, just fits a single offset), and ‘level+slope’ (which will fit a plane)

• recombine_lyot – If True, will recombine the lyot coronagraph into the main chip after the initial round of level matching. This will force the main science chip to have a 0 correction, as the lyot seems to be more wobbly. Defaults to False

• combine_nircam_short – Whether to combine the four NIRCam short chips before matching in a mosaic. Defaults to False

• do_local_subtraction – Whether to do a sigma-clipped local median subtraction. Defaults to True

• sigma – Sigma for sigma-clipping. Defaults to 3

• npixels – Pixels to grow for masking. Defaults to 5

• dilate_size – make_source_mask dilation size. Defaults to 7

• max_iters – Maximum sigma-clipping iterations. Defaults to 20

• max_points – Maximum points to include in histogram plots. This step can be slow so this speeds it up. Defaults to 10000

• do_sigma_clip – Whether to do sigma-clipping on data when reprojecting. Defaults to False

• weight_method – How to weight in least-squares minimization. Options are ‘equal’ (equal weighting), ‘npix’ (weight by number of pixels), and ‘rms’ (weight by rms of the delta values). Defaults to ‘equal’

• min_area_percent – Minimum percentage of average areal overlap to remove tiles. Defaults to 0.002 (0.2%)

• min_linear_frac – Minimum linear overlap in any direction to keep tiles. Defaults to 0.2

• rms_sig_limit – Sigma limit for cutting off noisy fits. Defaults to 2

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• overwrite – Whether to overwrite or not. Defaults to False

calculate_delta(files, fit_type='level', stacked_image=False, procs=None)

Match relative offsets between tiles

Parameters:

• files (list) – List of files to match

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

• stacked_image – Whether this is a stacked image or not. Default to False

• procs (int, optional) – Number of processes to run in parallel. Defaults to None, which is series

do_step()

Run level matching

find_optimum_deltas(delta_mat, npix_mat, rms_mat, lin_size_mat, valid_mat, fit_type='level', n_draws=25, n_iter=10000, convergence_abs_tol=1e-08, convergence_rel_tol=1e-05, ref_idx=None)

Get optimum deltas from a delta/weight matrix.

Taken from the JWST skymatch step, with some edits to remove potentially bad fits due to small areal overlaps, or noisy diffs, and various weighting schemes.

If we’re fitting a plane, delta_mat will be an NxNx3 matrix, and we’ll minimize over each of the last axes separately

Parameters:

• delta_mat (np.ndarray) – Matrix of delta values. These may be [a, b, c] coefficients if we’re fitting a plane

• npix_mat (np.ndarray) – Matrix of number of pixel values for calculating delta

• rms_mat (np.ndarray) – Matrix of RMS values

• lin_size_mat (np.ndarray) – 1/0 array for whether overlaps pass minimum linear extent

• valid_mat (np.ndarray) – 1/0 array for whether overlaps are valid

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

• n_draws – When using the iterative method, we need to sample from the fitted plane. This controls how many draws we do. Defaults to 25

• n_iter – Maximum number of iterations before breaking out of the fitting routine. Defaults to 10,000

• convergence_abs_tol – Absolute tolerance to define convergence. Defaults to 1e-8

• convergence_rel_tol – Relative tolerance to define convergence. Defaults to 1e-5

• ref_idx – Index to define the zero level for all the level matching. Defaults to None, which will use the average correction

get_level_match(files1, files2, fit_type, ref_x=0, ref_y=0, plot_name=None, maxiters=10, plane_fit_maxiters=20, plane_fit_abs_tol=1e-08, plane_fit_rel_tol=1e-05)

Calculate relative difference between groups of files on the same pixel grid

Parameters:

• files1 – List of files to get difference from

• files2 – List of files to get relative difference to

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

• ref_x – Reference x-coord to perform fits relative to. Defaults to 0

• ref_y – Reference y-coord to perform fits relative to. Defaults to 0

• plot_name – Output plot name. Defaults to None

• maxiters – Maximum iterations for the sigma-clipping. Defaults to 10

• plane_fit_maxiters – Maximum number of iterations for the plane fitting. Defaults to 20

• plane_fit_abs_tol – Absolute tolerance to define convergence in the plane fit. Defaults to 1e-8

• plane_fit_rel_tol – Relative tolerance to define convergence in the plane fitting. Defaults to 1e-5

get_per_dither_delta(dithers, fit_type='level', procs=1)

Function to parallelise getting the delta for each observation in a dither sequence

Parameters:

• dithers – List of dithers to get deltas for

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

• procs – Number of processes to run simultaneously. Defaults to 1

get_plot_name(files1, files2)

Make a plot name from list of files for level matching

Parameters:

• files1 – First list of files

• files2 – Second list of files

get_reproject(file, optimal_wcs, optimal_shape, stacked_image=False)

Reproject files, maintaining list structure

Parameters:

• file – List or single file to reproject

• optimal_wcs – WCS to reproject to

• optimal_shape – output array shape for the WCS

• stacked_image (bool) – Whether this is a stacked image or not. Defaults to False

make_stacked_images(dithers, stacked_dir, procs=1)

Function to parallellise up making stacked dither images

Parameters:

• dithers – List of dithers to go

• stacked_dir – Where to save stacked images to

• procs – Number of simultaneous processes to run. Defaults to 1

match_dithers(dithers, fit_type='level')

Match levels between the dithers in each mosaic tile

Parameters:

• dithers – List of dither groups

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

match_lyot_science(dithers, fit_type='level')

Match levels between each individual lyot/main science chip, and recombine

Parameters:

• dithers – List of dithers

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

match_mosaic_tiles(dithers, fit_type='level')

Match levels between each mosaic tile

Parameters:

• dithers – List of dither groups

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

match_nircam_short(dithers, fit_type='level')

Match levels between the NIRCam short chips

Parameters:

• dithers – List of dithers

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

parallel_delta_matrix(ij, file_reproj, files, fit_type='level', ref_x=0, ref_y=0)

Function to parallelise up getting delta matrix values

Parameters:

• ij – List of matrix (i, j) values

• file_reproj – Reprojected file

• files – Full list of files

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

• ref_x – Reference x value to subtract to centre the fits. Defaults to 0

• ref_y – Reference y value to subtract to centre the fits. Defaults to 0

parallel_get_reproject(idx, files, optimal_wcs, optimal_shape, stacked_image=False)

Light function to parallelise get_dither_reproject

Parameters:

• idx – File idx to reproject

• files – Full file list

• optimal_wcs – Optimal WCS for input stack of images

• optimal_shape – Optimal shape for input stack of images

• stacked_image – Stacked image or not? Defaults to False

parallel_make_stacked_image(dither, out_dir)

Light wrapper around parallelising the stacked image

Parameters:

• dither – Dither to stack

• out_dir – Directory to save to

parallel_match_lyot_science(dither, stacked_dir, fit_type)

Function to parallelise up combining the lyot back into the main science chip

Because the lyot seems to behave a little weirdly in its backgrounds from time-to-time, we force the main science correction to be 0 and put that all into the lyot

Parameters:

• dither – Dither to level match and combine

• stacked_dir – Directory contained stacked images

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

parallel_match_nircam_short(dither, stacked_dir, fit_type)

Function to parallelise up matching levels between the short NIRCam chips

Parameters:

• dither – Dither to level match and combine

• stacked_dir – Directory contained stacked images

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

parallel_per_dither_delta(dither, fit_type='level')

Function to parallelise up matching dithers

Parameters:

• dither – Input dither

• fit_type – Which type of fit to do. See ALLOWED_FIT_TYPES. Defaults to “level”

Level 1

class pjpipe.Lv1Step(target, band, in_dir, out_dir, dr_version, step_ext, procs, is_bgr, jwst_parameters=None, overwrite=False)

Wrapper around the level 1 JWST pipeline

Parameters:

• target – Target to consider

• band – Band to consider

• in_dir – Input directory

• out_dir – Output directory

• dr_version – Data processing version

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel.

• is_bgr – Whether we’re processing background observations or not

• jwst_parameters – Parameter dictionary to pass to the level 1 pipeline. Defaults to None, which will run the observatory defaults

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run the level 1 pipeline

parallel_lv1(dither)

Parallelise lv1 reprocessing

Parameters:

dither – Name for dither group. This is used because we inherit persistence from previous integration in the set

run_step(dithers, procs=1)

Wrap parallelism around the level 1 pipeline

Parameters:

• dithers – List of dithers to loop over

• procs – Number of processes to run. Defaults to 1

Level 2

class pjpipe.Lv2Step(target, band, in_dir, out_dir, dr_version, step_ext, is_bgr, procs, bgr_check_type='parallel_off', bgr_background_name='off', bgr_observation_types=None, process_bgr_like_science=False, jwst_parameters=None, updated_flats_dir=None, overwrite=False)

Wrapper around the level 2 JWST pipeline

Parameters:

• target – Target to consider

• band – Band to consider

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• is_bgr – Whether we’re processing background observations or not

• procs – Number of processes to run in parallel

• bgr_check_type – Method to check if obs is science or background. Options are given by BGR_CHECK_TYPES. Defaults to ‘parallel_off’

• bgr_background_name – If bgr_check_type is ‘check_in_name’ or ‘filename’, this is the string to match

• bgr_observation_types – List of observation types with dedicated backgrounds. Defaults to None, i.e. no observations have backgrounds

• process_bgr_like_science – If True, will process background images as if they are science images. Defaults to False

• jwst_parameters – Parameter dictionary to pass to the level 2 pipeline. Defaults to None, which will run the observatory defaults

• updated_flats_dir – Directory with the updated flats to use instead of default ones. Defaults to None, which will use the pipeline default flats

• overwrite – Whether to overwrite or not. Defaults to False

create_asn_files(files, procs=1)

Setup asn lv2 files

Parameters:

• files (*) – List of files to include in the asn file

• procs (*) – If greater than 1, will write out an asn file per observation. Defaults to 1

do_step()

Run the level 2 pipeline

get_asn_json(sci_tab, bgr_tab)

Build the JSON file from the science and background tables

parallel_lv2(asn_file)

Parallelise running lv2 processing

Parameters:

asn_file – Association files to run

propagate_metadata(files)

Propagate metadata through to the output files

Parameters:

files – List of files to loop over

run_step(asn_files, procs=1)

Wrap parallelism around the level 2 pipeline

Parameters:

• asn_files – List of association files to loop over

• procs – Number of processes to run. Defaults to 1

Level 3

class pjpipe.Lv3Step(target, band, in_dir, out_dir, dr_version, is_bgr, step_ext, procs, tweakreg_degroup_nircam_modules=False, tweakreg_degroup_nircam_short_chips=False, tweakreg_group_dithers=None, tweakreg_degroup_dithers=None, skymatch_group_dithers=None, skymatch_degroup_dithers=None, bgr_check_type='parallel_off', bgr_background_name='off', process_bgr_like_science=False, jwst_parameters=None, do_drizzle=False, do_blot=False, blot_fillval=nan, overwrite=False)

Wrapper around the level 3 JWST pipeline

Parameters:

• target – Target to consider

• band – Band to consider

• in_dir – Input directory

• out_dir – Output directory

• dr_version – Data processing version

• is_bgr – Whether we’re processing background observations or not

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• tweakreg_degroup_nircam_modules – Whether to degroup NIRCam A and B modules. Currently, the WCS is inconsistent between the two, so should probably be set to True if you see “ghosting” in the final mosaic. Defaults to False

• tweakreg_degroup_nircam_short_chips – Whether to degroup NIRCam short 1/2/3/4 chips. There may be some shifts between these, so should ideally find a shift for each chip. Defaults to False

• tweakreg_group_dithers – List of ‘miri’, ‘nircam_long’, ‘nircam_short’ of whether to group up dithers for tweakreg. Defaults to None, which will keep at default

• tweakreg_degroup_dithers – List of ‘miri’, ‘nircam_long’, ‘nircam_short’ of whether to degroup dithers for tweakreg. Defaults to None, which will keep at default.

• skymatch_group_dithers – List of ‘miri’, ‘nircam_long’, ‘nircam_short’ of whether to group up dithers for skymatch. Defaults to None, which will keep at default

• skymatch_degroup_dithers – List of ‘miri’, ‘nircam_long’, ‘nircam_short’ of whether to degroup dithers for skymatch. Defaults to None, which will keep at default.

• bgr_check_type – Method to check if obs is science or background. Options are given by BGR_CHECK_TYPES. Defaults to ‘parallel_off’

• bgr_background_name – If bgr_check_type is ‘check_in_name’ or ‘filename’, this is the string to match

• process_bgr_like_science – If True, will process background images as if they are science images. Defaults to False

• jwst_parameters – Parameter dictionary to pass to the level 2 pipeline. Defaults to None, which will run the observatory defaults

• do_drizzle – If True, drizzle individual frames to the i2d mosaic WCS after the main pipeline run. Note: creates a lot of files. Defaults to False.

• do_blot – If True, blot the final i2d mosaic to the detector frame of each exposure, producing one *_i2d_blot.fits file per exposure. Independent of do_drizzle. Defaults to False.

• blot_fillval – Fill value for pixels outside the blotted footprint. Defaults to np.nan

• overwrite – Whether to overwrite or not. Defaults to False

create_asn_file(files)

Setup asn lv3 file

do_step()

Run the level 3 pipeline

propagate_metadata(files)

Propagate metadata through to the output files

Parameters:

files – List of files to loop over

run_step(asn_file)

Run the level 3 step

Parameters:

asn_file – Path to association JSON file

Lyot Masking

class pjpipe.LyotMaskStep(in_dir, out_dir, step_ext, procs, method='mask', overwrite=False)

Mask the lyot coronagraph in MIRI observations

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• method – Whether to just mask the coronagraph (mask), or only parts that overlap the main science chip in other observations (mask_overlap). Defaults to ‘mask’

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run lyot masking

parallel_lyot_mask(file, mask_value=513)

Parallelise lyot masking

Parameters:

• file – File to mask lyot in

• mask_value – DQ bit value for masked values. Defaults to 513 (DO_NOT_USE+NON_SCIENCE)

parallel_lyot_mask_overlap(file, all_files, mask_value=513)

Parallelise lyot overlap masking

Parameters:

• file – File to mask lyot in

• all_files – Complete list of files

• mask_value – DQ bit value for masked values. Defaults to 513 (DO_NOT_USE+NON_SCIENCE)

run_step(files, procs=1)

Wrap paralellism around the lyot masking

Parameters:

• files – List of files to mask lyot in

• procs – Number of parallel processes to run. Defaults to 1

Lyot Separation

class pjpipe.LyotSeparateStep(in_dir, out_dir, step_ext, procs, miri_ext='mirimage', overwrite=False)

Separate each MIRI file out into main science chip and lyot coronagraph

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• miri_ext – MIRI filename extension. Defaults to “mirimage”

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run lyot separation

parallel_lyot_separate(file, mask_value=513)

Parallelise lyot separation

Parameters:

• file – File to separate lyot in

• mask_value – DQ bit value for masked values. Defaults to 513 (DO_NOT_USE+NON_SCIENCE)

run_step(files, procs=1)

Wrap paralellism around the lyot separation

Parameters:

• files – List of files to separate lyot in

• procs – Number of parallel processes to run. Defaults to 1

Individual Field Mosaicking

class pjpipe.MosaicIndividualFieldsStep(target, band, in_dir, out_dir, procs, crf_ext='crf_tweakback', resample_parameters=None, parallel=True, overwrite=False)

Mosaic each individual field in an observation set

N.B. This should be run on _crf_tweakback files, so this step assumes you’ve already run level 3 and any astrometric alignment.

Parameters:

• target – Target to consider

• band – Band to consider

• in_dir – Input directory for the crf files

• out_dir – Where to save the mosaicked files to

• procs – Number of processes to run in parallel

• crf_ext – Extension for files to mosaic. Defaults to crf_tweakback

• resample_parameters – Parameters to pass to the JWST resample step

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run mosaicking individual fields

parallel_mosaic_fields(visit)

Parallellise mosaicking

Parameters:

visit – Visit to produce mosaic for

run_step(individual_visits, procs=1)

Wrap parallelism around mosaicking

Parameters:

• individual_visits – List of individual visits

• procs – Number of parallel processes to run. Defaults to 1

Move Raw Observations

class pjpipe.MoveRawObsStep(target, band, step_ext, in_dir, out_dir, dr_version, is_bgr, obs_to_skip=None, extra_obs_to_include=None, overwrite=False)

Move raw observations from the MAST folder into a specific target/band folder

Because we may want to pull files for science images but not for backgrounds, they can be distinguished at the config level, as:

[parameters.move_raw_obs.extra_obs_to_include]
ic5332.sci.ngc7496 = 'jw02107041001_0?2'

where you can use ‘sci’ or ‘bgr’ to distinguish. To include in either case, just omit this part.

Parameters:

• target – Target to consider

• band – Band to consider

• step_ext – .fits extension to look for (e.g. uncal, rate etc.)

• in_dir – Input directory to search for files (should be some kind of mastDownload-esque folder)

• out_dir – Where to move files to

• dr_version – Data reprocessing version

• is_bgr – Whether we’re processing background observations or not

• obs_to_skip – List of failed or otherwise observations that shouldn’t be moved. Defaults to None, which skips nothing

• extra_obs_to_include – List of extra observations to include, for example MIRI flats from elsewhere. Defaults to None, which includes nothing extra

• overwrite (bool) – Whether to overwrite or not. Defaults to False

do_step()

Move raw observation files

get_raw_files()

Build a list of raw files

move_raw_files(raw_files)

Actually move the raw files

Multi-Tile Destriping

class pjpipe.MultiTileDestripeStep(in_dir, out_dir, step_ext, procs, apply_to_unflat=False, do_convergence=False, convergence_sigma=1, convergence_max_iterations=5, weight_method='mean', weight_type='ivm', do_level_match=False, quadrants=True, min_mask_frac=0.2, do_vertical_subtraction=False, do_large_scale=True, large_scale_filter_scale=None, large_scale_filter_extend_mode='reflect', sigma=3, dilate_size=7, maxiters=None, reproject_func='interp', overwrite=False)

Subtracts large-scale stripes using dither information

Create a weighted average image, then do a sigma-clipped median along (optionally) columns and rows (optionally by quadrants), after optionally smoothing the stacked image to attempt to remove persistent large-scale ripples.

If you see clear oversubtraction in the data, you should set do_large_scale to False. In most cases, it appears to work well but there may be some edge cases where it doesn’t work well.

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• apply_to_unflat – If True, will undo the flat-fielding before applying the stripe model, and then reapply it. Defaults to False

• do_convergence – Whether to loop this iteratively until convergence, or just do a single run. Defaults to False

• convergence_sigma – Maximum sigma difference to decide if the iterative loop has converged. Defaults to 1

• convergence_max_iterations – Maximum number of iterations to run. Defaults to 5

• weight_type – Weighting method for stacking the image. Should be one of ‘mean’, ‘median’, ‘sigma_clip’. Defaults to ‘mean’

• weight_type – How to weight the stacked image. Defaults to ‘ivm’, inverse readnoise

• do_level_match – Whether to do a simple match between tiles. Should be set to False if this is run after level_match_step. Defaults to False

• quadrants – Whether to split up stripes per-amplifier. Defaults to True

• min_mask_frac – Minimum fraction of unmasked data in quadrants to calculate a median. Defaults to 0.2 (i.e. 20% unmasked)

• do_vertical_subtraction – Whether to also do a step of vertical stripe subtraction. Defaults to False

• do_large_scale – Whether to do filtering to try and remove large, consistent ripples between data. Defaults to True

• large_scale_filter_scale – Factor by which we smooth for large scale persistent ripple removal. Defaults to None, which will use a scale ~10% of the data shape

• large_scale_filter_extend_mode – How to extend values in the filter beyond array edge. Default is “reflect”. See the specific docs for more info

• sigma – sigma value for sigma-clipped statistics. Defaults to 3

• dilate_size – Dilation size for mask creation. Defaults to 7

• maxiters – Maximum number of sigma-clipping iterations. Defaults to None

• overwrite – Whether to overwrite or not. Defaults to False

create_weighted_avg_image(data, weights)

Create an average image from a bunch of reprojected ones

Parameters:

• data – List of data arrays

• weights – List of weights. Should be same length as data

do_step()

Run multi-tile destriping

get_data_avg_smooth(direction=None)

Filter data with a large scale filter

Will either perform a large-scale median filter over a specific axis, or a mean filter over all axes. Also creates a mask

Parameters:

direction – Direction to smooth over, either “horizontal”, “vertical”, or None. Defaults to None

get_mask(data)

Create positive/negative mask

multi_tile_destripe(file, iteration=1, do_large_scale=False)

Do a row-by-row, column-by-column data subtraction using other dither information

Reproject average image, optionally remove persistent large-scale stripes, then do a sigma-clipped median along columns and rows (optionally by quadrants), and finally a smoothed clip along rows after boxcar filtering to remove persistent large-scale ripples in the data

Parameters:

• file (str) – File to correct

• iteration – What iteration are we on? Defaults to 1

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

parallel_multi_tile_destripe(idx, iteration=1, do_large_scale=False)

Function to parallelise up multi-tile destriping

Parameters:

• idx – Index of file to be destriped

• iteration – What iteration are we on? Defaults to 1

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

run_multi_tile_destripe(procs=1, iteration=1, do_large_scale=False)

Wrap parallelism around the multi-tile destriping

Parameters:

• procs – Number of parallel processes. Defaults to 1

• iteration – What iteration are we on? Defaults to 1

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

weighted_reproject_image(files, procs=1, do_large_scale=False)

Get reprojected images (and weights)

Parameters:

• files (list) – Files to reproject

• procs (int) – Number of processes to use. Defaults to 1.

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

PSF Matching

class pjpipe.PSFMatchingStep(target, in_dir, out_dir, kernel_dir, procs, in_step_ext, band=None, target_bands=None, reproject_func='interp', overwrite=False, skip_if_missing_kernels=False)

Match PSF for all images

Taking a list of target resolutions and kernels, will convolve to those resolutions. If an existing file already exists (e.g. a F2100W image from processing), will also regrid to that pixel grid

Parameters:

• target – Target to consider

• in_dir – Input directory

• out_dir – Output directory

• kernel_dir – Kernel directory

• procs – Number of processes to run in parallel

• in_step_ext – Filename extension for the input files

• band – Bands to consider

• target_bands – Bands to convolve to

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• overwrite – Whether to overwrite or not

• skip_if_missing_kernels – Whether to skip if kernels are missing

do_step()

Run psf_matching step

parallel_psf_match(current_task, current_band=None)

Parallelize psf matching to target resolution

Parameters:

• current_task – tuple (file, target_band), where file is the File to apply psf matching, and target_band is the band of target resolution

• current_band – band of the current image

Returns:

True or False

run_step(files, procs=1)

Wrap paralellism around applying psf matching

Parameters:

• files – List of files to process

• procs – Number of parallel processes to run. Defaults to 1

PSF Modelling

class pjpipe.PSFModelStep(in_dir, out_dir, step_ext, procs, method='replace', npixels=9, separation=0.1, psf_fov_pixels=511, psf_thresh=1e-05, dilate_size=7, nsigma=5, overwrite=False)

Step to model the PSF in saturated sources

In the centres of galaxies, saturation and PSF wings can blow out the image in an unpleasant way. This step attempts to alleviate that by finding saturated sources and either subtracting the PSF, or painting in the saturated regions

N.B. This is still highly preliminary, and should be seen as alpha. It hasn’t been thoroughly tested across the whole sample yet, so weird errors may arise. You have been warned!

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel. Currently, does nothing

• method – Whether to “replace” saturated cores, or “subtract” the PSF. Defaults to replace

• npixels – Minimum number of pixels to define a saturated source. Defaults to 9

• separation – When creating catalogues for the saturated sources, this is the minimum distance (in arcsec) to identify a distinct source. Defaults to 0.1

• psf_fov_pixels – Size of the simulated PSF. Should be odd so it has a centre. Defaults to 511

• psf_thresh – Minimum threshold to define where we consider the PSF to be significant (and thus used in the fit). Defaults to 1e-5, i.e. 0.001% of the PSF peak

• dilate_size – Dilate size for creating source mask before fitting, since we don’t want to fit in very bright areas. Defaults to 7

• nsigma – Sigma-clipping limit for creating source mask, since we don’t want to fit in very bright areas. Defaults to 5

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run PSF modelling

get_initial_amp(data, psf, x_cen, y_cen, psf_x_cen, psf_y_cen)

Get initial amplitude guess for PSF

This calculates an average ratio between the image and the PSF at the initial guess of the PSF centre. Our bounds for the amplitude are quite broad, so as long as this is order-of-magnitude right, we should be OK

Parameters:

• data – Input data

• psf – Input PSF

• x_cen – Guess for x centre of saturated source

• y_cen – Guess for y centre of saturated source

• psf_x_cen – x centre of the PSF

• psf_y_cen – y centre of the PSF

get_psf(file)

Get PSF for given observation

Parameters:

file – Input file to get PSF for

get_sat_coords(files)

Get RA/Dec for the centres of saturated sources in each image

Will look for saturated pixels in each image, and then merge these given a separation to a minimum catalogue

Parameters:

files – List of input files to loop over

make_diagnostic_plot(data, psf_model, plot_name, mask=None)

Create a diagnostic plot to show the fit

If subtracting, will create data/fit PSF/subtracted data, otherwise will show data/fit PSF

Parameters:

• data – Input data

• psf_model – Final PSF model

• plot_name – Name to save plot to

• mask – If not None, will NaN out pixels. Can make the visualisation clearer. Defaults to None

run_step(files, sat_coords)

Run the step, fitting PSFs to catalogue positions of saturated coordinates

Parameters:

• files – List of files to fit PSF for

• sat_coords – List of coordinates corresponding to saturated positions

Regress Against Previous Version

class pjpipe.RegressAgainstPreviousStep(target, in_dir, curr_version, prev_version=None, file_exts=None, reproject_func='interp', overwrite=False)

Create diagnostic plots to regress against previous versions

Parameters:

• target – Target to consider

• in_dir – Input directory

• curr_version – Current version to compare to…

• prev_version – Previous version

• file_exts – File extensions (in priority order) to search for

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run previous version regression

regress_plot(file_dict, key)

Plot per-instrument comparison

Parameters:

• file_dict – Dictionary of files, separated by instrument

• key – Instrument key

Release

class pjpipe.ReleaseStep(in_dir, out_dir, target, bands, file_exts=None, remove_bloat=False, move_tweakback=False, move_backgrounds=False, move_individual_fields=False, move_psf_matched=False, move_diagnostic_plots=False, compress_diagnostic_plots=True, lv3_dir='lv3', tweakback_dir='lv3', tweakback_ext='tweakback', background_dir='lv2', background_ext='combinedbackground', individual_fields_dir='mosaic_individual_fields', psf_matched_dir='psf_matching', diagnostic_plot_dir='plots', overwrite=False)

Tidies up files, moves to a single directory for release

This step will move the final useful files, plus optionally any tweakback’d crf files and background files, into a neat directory structure for release

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• target – Target to consider

• bands – Bands to consider

• file_exts – List of filetypes to move. Defaults to moving fits files, plus any generated catalogues and segmentation maps

• remove_bloat – Will remove generally un-needed extensions from fits files. Defaults to False

• move_tweakback – Whether to move tweakback’d crf files or not. Defaults to False

• move_backgrounds – Whether to move combined background files or not. Defaults to False

• move_individual_fields – Whether to move individual field mosaics or not. Defaults to False

• move_psf_matched – Whether to move PSF matched images or not. Defaults to False

• move_diagnostic_plots – Whether to move various diagnostic plots or not. Defaults to False

• compress_diagnostic_plots – Whether to compress the diagnostic plot folder to limit file number. Defaults to True

• lv3_dir – Where level 3 files are located, relative to the target directory structure. Defaults to “lv3”

• background_dir – Where tweakback files are located, relative to the target directory structure. Defaults to “lv3”

• tweakback_ext – Filename extension for tweakback files. Defaults to “tweakback”

• background_dir – Where combined background files are located, relative to the target directory structure. Defaults to “lv2”

• background_ext – Filename extension for combined background files. Defaults to “combinedbackground”

• individual_fields_dir – Where individual field mosaics are located, relative to the target directory structure. Defaults to “mosaic_individual_fields”

• overwrite – Whether to overwrite or not. Defaults to False

do_compress_diagnostic_plots()

Compress diagnostic plot directories

do_move_backgrounds(band)

Move combined background files

Parameters:

band – Band to consider

do_move_diagnostic_plots(band)

Move various diagnostic plots

Parameters:

band – Band to consider

do_move_individual_fields(band)

Move individual field mosaics

Parameters:

band – Band to consider

do_move_psf_matched(band)

Move PSF matched images

Parameters:

band – Band to consider

do_move_tweakback(band)

Move tweakback crf files

Parameters:

band – Band to consider

do_step()

Run the release step

move_files(band, file_ext)

Move files

Parameters:

• band – Band to consider

• file_ext – File extension to move

Single-Tile Destriping

class pjpipe.SingleTileDestripeStep(in_dir, out_dir, step_ext, procs, quadrants=True, vertical_subtraction=True, destriping_method='median_filter', vertical_destriping_method='row_median', filter_diffuse=False, min_mask_frac=0.2, sigma=3, npixels=3, dilate_size=11, max_iters=20, filter_scales=None, filter_extend_mode='reflect', pca_components=50, pca_reconstruct_components=10, overwrite=False)

NIRCAM/NIRISS Destriping routines

Contains a number of routines to destripe NIRCAM or NIRISS data – median filtering, PCA, and an equivalent of remstripe from the CEERS team

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits file extension to run step on

• procs – Number of processes to run in parallel

• quadrants – Whether to split the chip into 512 pixel segments, and destripe each (mostly) separately. Defaults to True

• vertical_subtraction – Perform sigma-clipped median column subtraction? Defaults to True

• destriping_method – Method to use for destriping. Allowed options are given by DESTRIPING_METHODS. Defaults to ‘median_filter’

• vertical_destriping_method – Method to use for vertical destriping. Allowed options are given by DESTRIPING_METHODS. Defaults to ‘row_median’

• filter_diffuse – Whether to perform high-pass filter on data, to remove diffuse, extended emission. Defaults to False, but should be set True for observations where emission fills the FOV

• min_mask_frac – Minimum fraction of unmasked data in quadrants to calculate a median. Defaults to 0.2 (i.e. 20% unmasked)

• sigma – Sigma for sigma-clipping. Defaults to 3

• npixels – Pixels to grow for masking. Defaults to 5

• dilate_size – make_source_mask dilation size. Defaults to 11

• max_iters – Maximum sigma-clipping iterations. Defaults to 20

• filter_scales – Scales for filtering. Used in median filtering and smooth

• filter_extend_mode – How to extend values in the filter beyond array edge. Default is “reflect”. See the specific docs for more info

• pca_components – Number of PCA components to model. Defaults to 50

• pca_reconstruct_components – Number of PCA components to use in reconstruction. Defaults to 10

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run single-tile destriping

fit_robust_pca(data, err, mask, mask_column_frac=0.25, min_column_frac=0.5)

Fits the robust PCA algorithm

Parameters:

• data – Input data

• err – Input errors

• mask – Where data is masked

• mask_column_frac – In low masked cases, take the data where less than mask_column_frac is masked. Defaults to 0.25

• min_column_frac – In highly masked cases, take min_column_frac of data to ensure we have enough to fit. Defaults to 0.5

get_filter_diffuse(data, mask, dq_mask)

Filter out diffuse emission using Butterworth filter

Parameters:

• data – Input data

• mask – Pre-calculated mask

• dq_mask – Calculated data quality mask

make_destripe_plot(in_im, noise_model, out_name)

Create diagnostic plot for the destriping

Parameters:

• in_im – Input datamodel

• noise_model – Model for the stripes

• out_name – Output filename

make_mask_plot(data, mask, out_name, filter_diffuse=False)

Create mask diagnostic plot

Parameters:

• data – Input data

• mask – Calculated mask

• out_name – Output filename

• filter_diffuse – Whether to filter diffuse emission. Defaults to False

parallel_destripe(file)

Parallel destriping function

Parameters:

file – input file

reconstruct_pca(eigen_system_dict, data, err, mask)

Reconstruct PCA from the fit

Parameters:

• eigen_system_dict – Dictionary of the outputs from the PCA fit

• data – Input data

• err – Input error

• mask – Input mask

run_median_filter(im, prev_noise_model, out_name, is_subarray=False, quadrants=True, transpose=False)

Run a series of filters over the row medians. From Mederic Boquien.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_pca_denoise(im, prev_noise_model, pca_file, out_name, is_subarray=False, quadrants=True, transpose=False)

PCA-based de-noising

Build a PCA model for the noise using the robust PCA implementation from Tamas Budavari and Vivienne Wild. We mask the data, optionally high-pass filter (Butterworth) to remove extended diffuse emission, and build the PCA model from there. pca_final_med_row_subtraction is on, it will do a final row-by-row median subtraction, to catch large-scale noise that might get filtered out.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• pca_file – Where to save PCA model to

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_remstriping(im, prev_noise_model, out_name, is_subarray=False, quadrants=True, transpose=False)

Destriping based on the CEERS remstripe routine

Mask out sources, then collapse a median along x and y to remove stripes.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_row_median(im, out_name, prev_noise_model, quadrants=True, transpose=False)

Calculate sigma-clipped median for each row. From Tom Williams.

Parameters:

• im – Input datamodel

• out_name – Output filename

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_smooth(im, prev_noise_model, out_name, is_subarray=False, quadrants=False, transpose=False)

Smoothing-based de-noising

Calculate the sigma-clipped median over the rows, smooth these over a range of scales and use this to subtract. Should have the benefit of maintaining flux without necessarily having to filter away the large-scale structure. This is based on Dan Coe’s algorithm, except we do a number of scales here to effectively remove noise at multiple levels

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to False

• transpose – Whether data has been transposed. Defaults to False

run_step(files, procs=1)

Wrap paralellism around the destriping

Parameters:

• files – List of files to destripe

• procs – Number of parallel processes to run. Defaults to 1

run_vertical_subtraction(im, prev_noise_model, is_subarray=False, transpose=False)

Median filter subtraction of columns (optional diffuse emission filtering)

Parameters:

• im – Input datamodel

• prev_noise_model – Already calculated noise model, to subtract before doing vertical subtraction

• is_subarray – Whether the image is a subarray. Defaults to False

• transpose – Whether to transpose the data. Defaults to False

Utilities

pjpipe.utils.attribute_setter(pipeobj, parameters, band, target)

Set attributes for a function

Parameters:

• pipeobj – Function/class to set parameters for

• parameters – Dictionary of parameters to set

• band – Band to pull band-specific parameters for

• target – Target to pull target-specific parameters for

pjpipe.utils.do_jwst_convolution(file_in, file_out, file_kernel, blank_zeros=True, output_grid=None, reproject_func='interp')

Convolves input image with an input kernel, and writes to disk.

Will also process errors and do reprojection, if specified

Parameters:

• file_in – Path to image file

• file_out – Path to output file

• file_kernel – Path to kernel for convolution

• blank_zeros – If True, then all zero values will be set to NaNs. Defaults to True

• output_grid – None (no reprojection to be done) or tuple (wcs, shape) defining the grid for reprojection. Defaults to None

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

pjpipe.utils.get_band_ext(band)

Get the specific extension (e.g. mirimage) for a band

pjpipe.utils.get_band_type(band, short_long_nircam=False)

Get the instrument type from the band name

Parameters:

• band (str) – Name of band

• short_long_nircam (bool) – Whether to distinguish between short/long NIRCam bands. Defaults to False

pjpipe.utils.get_default_args(func)

Pull the default arguments from a function

pjpipe.utils.get_dq_bit_mask(dq, bit_flags='~DO_NOT_USE+NON_SCIENCE')

Get a DQ bit mask from an input image

Parameters:

• dq – DQ array

• bit_flags – Bit flags to get mask for. Defaults to only get science pixels

pjpipe.utils.get_kws(parameters, func, band, target, max_level=None)

Set up kwarg dict for a function, looping over band and target

Parameters:

• parameters – Dictionary of parameters

• func – Function to set the parameters for

• band – Band to pull band-specific parameters for

• target – Target to pull target-specific parameters for

• max_level – How far to recurse down the dictionary. Defaults to None, which will recurse all the way down

pjpipe.utils.get_obs_table(files, check_bgr=False, check_type='parallel_off', background_name='off')

Pull necessary info out of fits headers

pjpipe.utils.get_pixscale(hdu)

Get pixel scale from header.

Checks HDU header and returns a pixel scale

Parameters:

hdu – hdu to get pixel scale for

pjpipe.utils.get_short_band_name(band)

Get a stripped down short name for a band

pjpipe.utils.level_data(im, transpose=False)

Level overlaps in NIRCAM amplifiers

Parameters:

• im – Input datamodel

• transpose – Whether to transpose the data. Defaults to False

pjpipe.utils.load_toml(filename)

Open a .toml file

Parameters:

filename (str) – Path to toml file

pjpipe.utils.make_source_mask(data, mask=None, nsigma=3, npixels=3, dilate_size=11, sigclip_iters=5)

Make a source mask from segmentation image

pjpipe.utils.make_stacked_image(files, out_name, additional_hdus=None, auto_rotate=True, reproject_func='interp', match_background=False)

Create a quick stacked image from a series of input images

Parameters:

• files – List of input files

• out_name – Output stacked file

• additional_hdus – Can also append some additional data beyond the science extension by specifying the fits extension here. Defaults to None, which will not add anything extra

• auto_rotate – Whether to rotate the WCS to make a minimum sized image. Defaults to True

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

• match_background – Whether to match backgrounds when making the stack. Defaults to False

pjpipe.utils.parse_fits_to_table(file, check_bgr=False, check_type='parallel_off', background_name='off')

Pull necessary info out of fits headers

Parameters:

• file (str) – File to get info for

• check_bgr (bool) – Whether to check if this is a science or background observation (in the MIRI case)

• check_type (str) – How to check if background observation. Options are - ‘parallel_off’, which will use the filename to see if it’s a parallel observation with NIRCAM - ‘check_in_name’, which will use the observation name to check, matching against ‘background_name’. - ‘filename’, which will use the filename Defaults to ‘parallel_off’

• background_name (str) – Name to indicate background observation. Defaults to ‘off’.

pjpipe.utils.parse_parameter_dict(parameters, key, band, target, max_level=None)

Pull values out of a parameter dictionary

Parameters:

• parameters (dict) – Dictionary of parameters and associated values

• key (str) – Particular key in parameter_dict to consider

• band (str) – JWST band, to parse out band type and potentially per-band values

• target (str) – JWST target, for very specific values

• max_level – Maximum level to recurse down. Defaults to None, which will go until it finds something that’s not a dictionary

pjpipe.utils.recursive_setattr(f, attribute, value, protected=False)

Set potentially recursive function attributes.

This is needed for the JWST pipeline steps, which have levels to them

Parameters:

• f – Function to consider

• attribute – Attribute to consider

• value – Value to set

• protected – If a function is protected, this won’t strip out the leading underscore

pjpipe.utils.reproject_image(file, optimal_wcs, optimal_shape, hdu_type='data', do_sigma_clip=False, stacked_image=False, do_level_data=False, reproject_func='interp')

Reproject an image to an optimal WCS

Parameters:

• file – File to reproject

• optimal_wcs – Optimal WCS for input image stack

• optimal_shape – Optimal shape for input image stack

• hdu_type – Type of HDU. Can either be ‘data’, ‘err’, or ‘var_rnoise’

• do_sigma_clip – Whether to perform sigma-clipping or not. Defaults to False

• stacked_image – Stacked image or not? Defaults to False

• do_level_data – Whether to level between amplifiers or not. Defaults to False

• reproject_func – Which reproject function to use. Defaults to ‘interp’, but can also be ‘exact’ or ‘adaptive’

pjpipe.utils.save_file(im, out_name, dr_version)

Save out an image, adding in useful metadata

Parameters:

• im – Input JWST datamodel

• out_name – File to save output to

• dr_version – Data processing version

pjpipe.utils.sigma_clip(data, dq_mask=None, sigma=1.5, n_pixels=5, max_iterations=20)

Get sigma-clipped statistics for data