SingleTileDestripeStep

This step should be run on NIRCam only!

Provides a number of methods to attempt to remove 1/f noise in NIRCam data just using the information in each tile (as opposed to the MultiTileDestripeStep, which runs on everything). If you have extended emission in your data, set filter_diffuse=True. We’ve seen good results with the default settings, but they can be tuned to your liking. Can be run on both the output from level 1 (i.e. rate.fits files) which we recommend, or from level 2 and beyond (i.e. cal.fits files). We recommend running straight after level 1 as the stripes should ideally be corrected before flat-fielding.

If you have subarray observations, quadrants will automatically be set to False, as the readout is different in these modes.

The PCA implementation for NIRCam destriping was adapted from work by Elizabeth Watkins (University of Manchester), and up-to-date code is available at this GitHub repository.

API

class pjpipe.SingleTileDestripeStep(in_dir, out_dir, step_ext, procs, quadrants=True, vertical_subtraction=True, destriping_method='median_filter', vertical_destriping_method='row_median', filter_diffuse=False, min_mask_frac=0.2, sigma=3, npixels=3, dilate_size=11, max_iters=20, filter_scales=None, filter_extend_mode='reflect', pca_components=50, pca_reconstruct_components=10, overwrite=False)

NIRCAM/NIRISS Destriping routines

Contains a number of routines to destripe NIRCAM or NIRISS data – median filtering, PCA, and an equivalent of remstripe from the CEERS team

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits file extension to run step on

• procs – Number of processes to run in parallel

• quadrants – Whether to split the chip into 512 pixel segments, and destripe each (mostly) separately. Defaults to True

• vertical_subtraction – Perform sigma-clipped median column subtraction? Defaults to True

• destriping_method – Method to use for destriping. Allowed options are given by DESTRIPING_METHODS. Defaults to ‘median_filter’

• vertical_destriping_method – Method to use for vertical destriping. Allowed options are given by DESTRIPING_METHODS. Defaults to ‘row_median’

• filter_diffuse – Whether to perform high-pass filter on data, to remove diffuse, extended emission. Defaults to False, but should be set True for observations where emission fills the FOV

• min_mask_frac – Minimum fraction of unmasked data in quadrants to calculate a median. Defaults to 0.2 (i.e. 20% unmasked)

• sigma – Sigma for sigma-clipping. Defaults to 3

• npixels – Pixels to grow for masking. Defaults to 5

• dilate_size – make_source_mask dilation size. Defaults to 11

• max_iters – Maximum sigma-clipping iterations. Defaults to 20

• filter_scales – Scales for filtering. Used in median filtering and smooth

• filter_extend_mode – How to extend values in the filter beyond array edge. Default is “reflect”. See the specific docs for more info

• pca_components – Number of PCA components to model. Defaults to 50

• pca_reconstruct_components – Number of PCA components to use in reconstruction. Defaults to 10

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run single-tile destriping

fit_robust_pca(data, err, mask, mask_column_frac=0.25, min_column_frac=0.5)

Fits the robust PCA algorithm

Parameters:

• data – Input data

• err – Input errors

• mask – Where data is masked

• mask_column_frac – In low masked cases, take the data where less than mask_column_frac is masked. Defaults to 0.25

• min_column_frac – In highly masked cases, take min_column_frac of data to ensure we have enough to fit. Defaults to 0.5

get_filter_diffuse(data, mask, dq_mask)

Filter out diffuse emission using Butterworth filter

Parameters:

• data – Input data

• mask – Pre-calculated mask

• dq_mask – Calculated data quality mask

make_destripe_plot(in_im, noise_model, out_name)

Create diagnostic plot for the destriping

Parameters:

• in_im – Input datamodel

• noise_model – Model for the stripes

• out_name – Output filename

make_mask_plot(data, mask, out_name, filter_diffuse=False)

Create mask diagnostic plot

Parameters:

• data – Input data

• mask – Calculated mask

• out_name – Output filename

• filter_diffuse – Whether to filter diffuse emission. Defaults to False

parallel_destripe(file)

Parallel destriping function

Parameters:

file – input file

reconstruct_pca(eigen_system_dict, data, err, mask)

Reconstruct PCA from the fit

Parameters:

• eigen_system_dict – Dictionary of the outputs from the PCA fit

• data – Input data

• err – Input error

• mask – Input mask

run_median_filter(im, prev_noise_model, out_name, is_subarray=False, quadrants=True, transpose=False)

Run a series of filters over the row medians. From Mederic Boquien.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_pca_denoise(im, prev_noise_model, pca_file, out_name, is_subarray=False, quadrants=True, transpose=False)

PCA-based de-noising

Build a PCA model for the noise using the robust PCA implementation from Tamas Budavari and Vivienne Wild. We mask the data, optionally high-pass filter (Butterworth) to remove extended diffuse emission, and build the PCA model from there. pca_final_med_row_subtraction is on, it will do a final row-by-row median subtraction, to catch large-scale noise that might get filtered out.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• pca_file – Where to save PCA model to

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_remstriping(im, prev_noise_model, out_name, is_subarray=False, quadrants=True, transpose=False)

Destriping based on the CEERS remstripe routine

Mask out sources, then collapse a median along x and y to remove stripes.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_row_median(im, out_name, prev_noise_model, quadrants=True, transpose=False)

Calculate sigma-clipped median for each row. From Tom Williams.

Parameters:

• im – Input datamodel

• out_name – Output filename

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• quadrants – Whether to break out by quadrants. Defaults to True

• transpose – Whether data has been transposed. Defaults to False

run_smooth(im, prev_noise_model, out_name, is_subarray=False, quadrants=False, transpose=False)

Smoothing-based de-noising

Calculate the sigma-clipped median over the rows, smooth these over a range of scales and use this to subtract. Should have the benefit of maintaining flux without necessarily having to filter away the large-scale structure. This is based on Dan Coe’s algorithm, except we do a number of scales here to effectively remove noise at multiple levels

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to False

• transpose – Whether data has been transposed. Defaults to False

run_step(files, procs=1)

Wrap paralellism around the destriping

Parameters:

• files – List of files to destripe

• procs – Number of parallel processes to run. Defaults to 1

run_vertical_subtraction(im, prev_noise_model, is_subarray=False, transpose=False)

Median filter subtraction of columns (optional diffuse emission filtering)

Parameters:

• im – Input datamodel

• prev_noise_model – Already calculated noise model, to subtract before doing vertical subtraction

• is_subarray – Whether the image is a subarray. Defaults to False

• transpose – Whether to transpose the data. Defaults to False