MultiTileDestripeStep

This step should only be run on NIRCam data!

This step uses information from all available tiles to correct any remaining 1/f noise. This is particularly important if you’ve filtered out diffuse emission in SingleTileDestripeStep, as this will leave large ripples untouched. This works by creating a mean image of the input tiles, and comparing each individual tile to this mean. The hope is that any remaining 1/f noise will be averaged out, so we can use this to cleanly separate source from instrumental noise.

If you have a small number of dithers, or at shorter NIRCam wavelengths, we suggest turning do_large_scale = True with a large_scale_filter_scale of ~200. This may need to be tweaked per-dataset. In this case, after the first pass we perform the same average imaging, but smooth it perpendicular to the stripe direction, to hopefully remove any remaining ripples. We then compare each tile to this smoothed, stacked image.

API

class pjpipe.MultiTileDestripeStep(in_dir, out_dir, step_ext, procs, apply_to_unflat=False, do_convergence=False, convergence_sigma=1, convergence_max_iterations=5, weight_method='mean', weight_type='ivm', do_level_match=False, quadrants=True, min_mask_frac=0.2, do_vertical_subtraction=False, do_large_scale=True, large_scale_filter_scale=None, large_scale_filter_extend_mode='reflect', sigma=3, dilate_size=7, maxiters=None, reproject_func='interp', overwrite=False)

Subtracts large-scale stripes using dither information

Create a weighted average image, then do a sigma-clipped median along (optionally) columns and rows (optionally by quadrants), after optionally smoothing the stacked image to attempt to remove persistent large-scale ripples.

If you see clear oversubtraction in the data, you should set do_large_scale to False. In most cases, it appears to work well but there may be some edge cases where it doesn’t work well.

Parameters:

• in_dir – Input directory

• out_dir – Output directory

• step_ext – .fits extension for the files going into the step

• procs – Number of processes to run in parallel

• apply_to_unflat – If True, will undo the flat-fielding before applying the stripe model, and then reapply it. Defaults to False

• do_convergence – Whether to loop this iteratively until convergence, or just do a single run. Defaults to False

• convergence_sigma – Maximum sigma difference to decide if the iterative loop has converged. Defaults to 1

• convergence_max_iterations – Maximum number of iterations to run. Defaults to 5

• weight_type – Weighting method for stacking the image. Should be one of ‘mean’, ‘median’, ‘sigma_clip’. Defaults to ‘mean’

• weight_type – How to weight the stacked image. Defaults to ‘ivm’, inverse readnoise

• do_level_match – Whether to do a simple match between tiles. Should be set to False if this is run after level_match_step. Defaults to False

• quadrants – Whether to split up stripes per-amplifier. Defaults to True

• min_mask_frac – Minimum fraction of unmasked data in quadrants to calculate a median. Defaults to 0.2 (i.e. 20% unmasked)

• do_vertical_subtraction – Whether to also do a step of vertical stripe subtraction. Defaults to False

• do_large_scale – Whether to do filtering to try and remove large, consistent ripples between data. Defaults to True

• large_scale_filter_scale – Factor by which we smooth for large scale persistent ripple removal. Defaults to None, which will use a scale ~10% of the data shape

• large_scale_filter_extend_mode – How to extend values in the filter beyond array edge. Default is “reflect”. See the specific docs for more info

• sigma – sigma value for sigma-clipped statistics. Defaults to 3

• dilate_size – Dilation size for mask creation. Defaults to 7

• maxiters – Maximum number of sigma-clipping iterations. Defaults to None

• overwrite – Whether to overwrite or not. Defaults to False

create_weighted_avg_image(data, weights)

Create an average image from a bunch of reprojected ones

Parameters:

• data – List of data arrays

• weights – List of weights. Should be same length as data

do_step()

Run multi-tile destriping

get_data_avg_smooth(direction=None)

Filter data with a large scale filter

Will either perform a large-scale median filter over a specific axis, or a mean filter over all axes. Also creates a mask

Parameters:

direction – Direction to smooth over, either “horizontal”, “vertical”, or None. Defaults to None

get_mask(data)

Create positive/negative mask

multi_tile_destripe(file, iteration=1, do_large_scale=False)

Do a row-by-row, column-by-column data subtraction using other dither information

Reproject average image, optionally remove persistent large-scale stripes, then do a sigma-clipped median along columns and rows (optionally by quadrants), and finally a smoothed clip along rows after boxcar filtering to remove persistent large-scale ripples in the data

Parameters:

• file (str) – File to correct

• iteration – What iteration are we on? Defaults to 1

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

parallel_multi_tile_destripe(idx, iteration=1, do_large_scale=False)

Function to parallelise up multi-tile destriping

Parameters:

• idx – Index of file to be destriped

• iteration – What iteration are we on? Defaults to 1

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

run_multi_tile_destripe(procs=1, iteration=1, do_large_scale=False)

Wrap parallelism around the multi-tile destriping

Parameters:

• procs – Number of parallel processes. Defaults to 1

• iteration – What iteration are we on? Defaults to 1

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False

weighted_reproject_image(files, procs=1, do_large_scale=False)

Get reprojected images (and weights)

Parameters:

• files (list) – Files to reproject

• procs (int) – Number of processes to use. Defaults to 1.

• do_large_scale – Is this a large-scale smoothed subtraction? Defaults to False