SingleTileDestripeStep

This step should be run on NIRCam only!

Provides a number of methods to attempt to remove 1/f noise in NIRCam data just using the information in each tile (as opposed to the MultiTileDestripeStep, which runs on everything). If you have extended emission in your data, set filter_diffuse=True. We’ve seen good results with the default settings, but they can be tuned to your liking. Can be run on both the output from level 1 (i.e. rate.fits files) which we recommend, or from level 2 and beyond (i.e. cal.fits files). We recommend running straight after level 1 as the stripes should ideally be corrected before flat-fielding.

If you have subarray observations, quadrants will automatically be set to False, as the readout is different in these modes.

API

class pjpipe.SingleTileDestripeStep(in_dir, out_dir, step_ext, procs, quadrants=True, vertical_subtraction=True, destriping_method='median_filter', vertical_destriping_method='row_median', filter_diffuse=False, min_mask_frac=0.2, sigma=3, npixels=3, dilate_size=11, max_iters=20, filter_scales=None, filter_extend_mode='reflect', pca_components=50, pca_reconstruct_components=10, overwrite=False)

NIRCAM Destriping routines

Contains a number of routines to destripe NIRCAM data – median filtering, PCA, and an equivalent of remstripe from the CEERS team

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits file extension to run step on

• procs – Number of processes to run in parallel

• quadrants – Whether to split the chip into 512 pixel segments, and destripe each (mostly) separately. Defaults to True

• vertical_subtraction – Perform sigma-clipped median column subtraction? Defaults to True

• destriping_method – Method to use for destriping. Allowed options are given by DESTRIPING_METHODS. Defaults to ‘median_filter’

• vertical_destriping_method – Method to use for vertical destriping. Allowed options are given by DESTRIPING_METHODS. Defaults to ‘row_median’

• filter_diffuse – Whether to perform high-pass filter on data, to remove diffuse, extended emission. Defaults to False, but should be set True for observations where emission fills the FOV

• min_mask_frac – Minimum fraction of unmasked data in quadrants to calculate a median. Defaults to 0.2 (i.e. 20% unmasked)

• sigma – Sigma for sigma-clipping. Defaults to 3

• npixels – Pixels to grow for masking. Defaults to 5

• dilate_size – make_source_mask dilation size. Defaults to 11

• max_iters – Maximum sigma-clipping iterations. Defaults to 20

• filter_scales – Scales for filtering. Used in median filtering and smooth

• filter_extend_mode – How to extend values in the filter beyond array edge. Default is “reflect”. See the specific docs for more info

• pca_components – Number of PCA components to model. Defaults to 50

• pca_reconstruct_components – Number of PCA components to use in reconstruction. Defaults to 10

• overwrite – Whether to overwrite or not. Defaults to False

do_step()

Run single-tile destriping

fit_robust_pca(data, err, mask, mask_column_frac=0.25, min_column_frac=0.5)

Fits the robust PCA algorithm

Parameters:

• data – Input data

• err – Input errors

• mask – Where data is masked

• mask_column_frac – In low masked cases, take the data where less than mask_column_frac is masked. Defaults to 0.25

• min_column_frac – In highly masked cases, take min_column_frac of data to ensure we have enough to fit. Defaults to 0.5

get_filter_diffuse(data, mask, dq_mask)

Filter out diffuse emission using Butterworth filter

Parameters:

• data – Input data

• mask – Pre-calculated mask

• dq_mask – Calculated data quality mask

make_destripe_plot(in_im, noise_model, out_name)

Create diagnostic plot for the destriping

Parameters:

• in_im – Input datamodel

• noise_model – Model for the stripes

• out_name – Output filename

make_mask_plot(data, mask, out_name, filter_diffuse=False)

Create mask diagnostic plot

Parameters:

• data – Input data

• mask – Calculated mask

• out_name – Output filename

• filter_diffuse – Whether to filter diffuse emission. Defaults to False

parallel_destripe(file)

Parallel destriping function

Parameters:

file – input file

reconstruct_pca(eigen_system_dict, data, err, mask)

Reconstruct PCA from the fit

Parameters:

• eigen_system_dict – Dictionary of the outputs from the PCA fit

• data – Input data

• err – Input error

• mask – Input mask

run_median_filter(im, prev_noise_model, out_name, quadrants=True)

Run a series of filters over the row medians. From Mederic Boquien.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• quadrants – Whether to break out by quadrants. Defaults to True

run_pca_denoise(im, prev_noise_model, pca_file, out_name, is_subarray=False, quadrants=True)

PCA-based de-noising

Build a PCA model for the noise using the robust PCA implementation from Tamas Budavari and Vivienne Wild. We mask the data, optionally high-pass filter (Butterworth) to remove extended diffuse emission, and build the PCA model from there. pca_final_med_row_subtraction is on, it will do a final row-by-row median subtraction, to catch large-scale noise that might get filtered out.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• pca_file – Where to save PCA model to

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

run_remstriping(im, prev_noise_model, out_name, is_subarray=False, quadrants=True)

Destriping based on the CEERS remstripe routine

Mask out sources, then collapse a median along x and y to remove stripes.

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to True

run_row_median(im, out_name, prev_noise_model, quadrants=True)

Calculate sigma-clipped median for each row. From Tom Williams.

Parameters:

• im – Input datamodel

• out_name – Output filename

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• quadrants – Whether to break out by quadrants. Defaults to True

run_smooth(im, prev_noise_model, out_name, is_subarray=False, quadrants=False)

Smoothing-based de-noising

Calculate the sigma-clipped median over the rows, smooth these over a range of scales and use this to subtract. Should have the benefit of maintaining flux without necessarily having to filter away the large-scale structure. This is based on Dan Coe’s algorithm, except we do a number of scales here to effectively remove noise at multiple levels

Parameters:

• im – Input datamodel

• prev_noise_model – Previously calculated noise model, to subtract before destriping

• out_name – Output filename

• is_subarray – Whether image is subarray or not. Defaults to False

• quadrants – Whether to break out by quadrants. Defaults to False

run_step(files, procs=1)

Wrap paralellism around the destriping

Parameters:

• files – List of files to destripe

• procs – Number of parallel processes to run. Defaults to 1

run_vertical_subtraction(im, prev_noise_model, is_subarray=False)

Median filter subtraction of columns (optional diffuse emission filtering)

Parameters:

• im – Input datamodel

• prev_noise_model – Already calculated noise model, to subtract before doing vertical subtraction

• is_subarray – Whether the image is a subarray. Defaults to False