2. MUSEreduce¶

MUSEreduce.musereduce is an easy-to-use python Class used as wrapper for the VLT/MUSE data reduction pipeline and does not replace the core functionalities of the pipeline provided by ESO. In order to function properly with this version of MUSEreduce.musereduce we recommend to install the pipeline version v2.8.1 found under: https://www.eso.org/sci/software/pipelines/muse/. This version of MUSEreduce.musereduce has not been tested with other version of the data reduction pipeline.

To run MUSEreduce.musereduce all of the raw data must be stored in a folder named user_path/raw/OB_ID, where OB_ID must be a unique name for each individual OB. It is not mandatory (but recommended) to use the same nomenclature as as in the fits header. The script has to point to user_path by setting the keyword rootpath of the json config file to the parent directory.

class MUSEreduce.musereduce(configfile, debug=False)[source]¶

Kwargs:

configfile : str: A json configfile for musereduce, where all the parameters are set.
debug : bool, (optional), default: None: True: MUSEreduce.musereduce runs in debug mode and ESORex will not be executed. All products must be available. It can be used for testing the creation of folder, and creating the .sof files

execute()[source]¶: This method executes wrapper and starts the data reduction process set in the json config file.

2.1. The basic folder structure¶

If auto_sort_data = True in the config file and MUSEreduce.musereduce is executed, the basic folder structure is created in the rootpath. As example, we consider that three OBs were observed: OB1a, OB1b, and OB2. OB1a and OB1b are two dither positions (with rotation angles of 5 and 95 deg and an exposure time of 2800s) of the same pointing OB1, while OB2 consists of three dither postions observed in one OB (with rotation angles of 10, 100, and 190 deg and an exposure time of 220s). The folder structure looks the following:

Each master OB (OB1, OB2) has its own folder in the reduced folder. Pointing OB1 consists of the two OBs OB1a and OB1b. Each OB has the following folders:

Each individual exposure (e.g., 091315-402023_2800_005_00) following the structure: RADEC_EXPTIME_ROTANGLE_COUNTER. RADEC are the coordinates in sexagesimal, EXPTIME is the exposure time in seconds and the ROTANGLE is the rotation angle in degrees. The COUNTER is needed if there are two exposures with the same configuration.

Each individual pointing (e.g., 091315-402023_2800) following the structure: RADEC_EXPTIME.

The calibrations contain DARK, TWILIGHT, and SCIENCE, in case the calibration files are created from the raw calibration files provided by ESO Therefore, the calibration steps must be executed.

The ESO_calibrations is the folder, into which the reduced calibration files delivered by ESO (if available) are copied.

The static_calibration_files, is the folder, into which the statics (part of the MUSE data reduction pipeline installation) are copied.

The folder std will contain the reduced data of the standard star.

Note

The calibration files and the standard star are the same per individual OB and are only needed once.

2.2. The config file¶

The json configuration file is needed to run MUSEreduce.musereduce. This file contains all of the data reduction setup. The json configuration file can be found in the main directly of MUSEpack and can be downloaded here.

{
  "global": {
    "pipeline_path": "path_to_pipeline/muse/2.8.1/",
    "mode": "WFM-NOAO",
    "withrvcorr": true,
    "auto_sort_data": true,
    "using_specific_exposure_time": false,
    "dither_multiple_OBs": false,
    "n_CPU": -1,
    "rootpath": "path_to_data_folder",
    "OB": "",
    "OB_list": []
  },
  "calibration":{
    "execute": true,
    "using_ESO_calibration": true,
    "renew_statics": false,
    "dark": false,
    "create_sof": true
  },
  "sci_basic":{
    "execute": true,
    "execute_std": true,
    "skyreject": "15.,15.,1",
    "skylines": "5577.339,6300.304",
    "create_sof": true
  },
  "std_flux":{
    "execute": true,
    "create_sof": true
  },
  "sky":{
    "execute": true,
    "modified": false,
    "sky_field": "auto",
    "fraction": 0.05,
    "ignore": 0.05,
    "method": "model",
    "create_sof": true
  },
  "sci_post":{
    "execute": true,
    "autocalib": "none",
    "subtract_sky": true,
    "raman": false,
    "create_sof": true
  },
  "dither_collect":{
    "execute": true,
    "user_list": []
  },
  "exp_align":{
    "execute": true,
    "srcmin": 5,
    "srcmax": 80,
    "create_sof": true
  },
  "exp_combine":{
    "execute": true,
    "weight": "exptime",
    "create_sof": true
  }

}

The file config file is structured the following. If keywords are directly controlling toggles of the MUSE data reduction pipeline their naming is identical.

config¶

Object Properties:

global (global) – global parameters affecting the data reduction process
calibration (calibration) – parameters that affect the calibration steps of the MUSE data reduction pipeline
sci_basic (sci_basic) – the preprocessing of the science exposures
std_flux (std_flux) – the flux calibration
sky (sky) – the sky subtraction
sci_post (sci_post) – the postprocessing step of the data reduction
dither_collect (dither_collect) – collecting the individual data cubes and pixel tables to combine them to one pointing
exp_align (exp_align) – aligning the individual data cubes and pixel tables to one common WCS
exp_combine (exp_combine) – combining the individual data cubes and pixel tables to one final science product

global¶

Object Properties:

pipeline_path (string) – The absolut path to the MUSE data reduction pipeline installation folder.
mode (string) – The observation mode the data was obtained with. (WFM-NOAO, WFM-AO, NFM-AO)
withrvcorr (bool) – bariocentric correction. Needs to be turned off, if one wants run an own wavelength calibration (true, false, default=’true’)
auto_sort_data (bool) – The raw data is sorted and the calibration files are assigned based on their header information. If True, the file lists ID_DAR.list, ID_SCI.list, and ID_TWI.list are created. If these have to be altered manually (e.g., using different calibration files), we recommend to run it first with auto_sort_data = True, then make the changes accordingly and from this point on set auto_sort_data = False. (false, true, default=true)
using_specific_exposure_time (float) – The user can choose to only reduce a specific exposure time, if the same OB contains multiple exposures with different exposure times (e.g., long and short exposures)
dither_multiple_OBs (bool) – Each OB is normally limited to a total exposure time of one hour. Therefore, one pointing may be distributed via multiple OBs. If dither_multiple_OBs = True it is possible to dither exposures from multiple OBs. In this case one must provide an OB_list. (false, true, default=false)
n_CPU (int) – The number of CPUs used to reduce the data. If set to -1 all available cores are used. (default=-1)
rootpath (string) – The absolut path in which the raw folder is located and in which the processed folder will be created.
OB (string) – The name of the OB that shall be reduce. It must identical to the OB_ID given in the raw folder.
OB_list (list) – If dither_multiple_OBs = True one must give a string list of OB_IDs, which will be dithered in the end. The calibration and data reduction runs on each individual OB to take into account different calibration files. (default=[])

calibration¶

Object Properties:

execute (bool) – Must be set True if this step should be executed (false, true, default=true)
using_ESO_Calibration (bool) – If the user wants to use the ESO calibration files (recommended if available) instead of running the calibration themselves (BIAS, DARK, FLAT, WAVECAL, LSF, TWILIGHT) it must be set to True (false, true, default=true)
renew_statics (bool) – Set to True if the static calibration files should be copied again from the MUSE data reduction pipeline folder. This is necessary if the installed version of the pipline changes and one wants to obtain the latest static calibrations (false, true, default=false)
dark (bool) – Set to True if one wants to also use the DARK files from the calibration. (false, true, default=false)
create_sof (bool) – Must be set True if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually, than it must be set to False since it will be overwritten otherwise (false, true, default=true)

sci_basic¶

Object Properties:

execute (bool) – Must be set True if this step should be executed (false, true, default=true)
execute_std (bool) – Must be set True if the STD star should be reduced (false, true, default=true)
sky_reject (string) – The sigma clipping parameters for the Gaussian fit to each sky emission line: high sigma clipping limit (float), low sigma clipping limit (float), number of iterations (int). For a more detailed description we refer to the MUSE data reduction pipeline manual. (default=’15., 15., 1’)
skylines (string) – The sky lines used to calibrate the wavelength offset in each IFU. the default are two lines but more strong skylines may be provided. For a more detailed description we refer to the MUSE data reduction pipeline manual. (default=’5577.339, 6300.304’)
create_sof (bool) – Must be set True if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually, than it must be set to False since it will be overwritten otherwise (false, true, default=true)

std_flux¶

Object Properties:
	execute (bool) – Must be set `True` if this step should be executed (false, true, default=true) create_sof (bool) – Must be set `True` if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually, than it must be set to `False` since it will be overwritten otherwise (false, true, default=true)

sky¶

Object Properties:

execute (bool) – Must be set True if this step should be executed (false, true, default=true)
modified (bool) – If set True the modified sky subtraction will be executed. This method will prevent the over subtraction of emission lines that are both emitted from the Earth’s atmosphere (tellurics) and the target (e.g., HII regions). For a detailed description of this method we refer to Zeidler et al. 2019. If modified = True the method should be subtract_model. Other methods will not lead to an error but they may lead to wrong results. (false, true, default=false)
sky_field (string) – Determines if a sky observation (if available) or the science observation itself is used to determine the background contamination (tellurics). If set to auto the pipeline will check if there are sky observations available and use the closest one in time to the science exposures. If the skyfield = object, the science exposure will be used. (‘auto’, ‘object’, default=’auto’)
fraction (float) – The fraction of the image used to be considered as sky. Must be between 0. and 1. For more details we refer to the MUSE data reduction pipeline handbook. ([0., 1.[, default=0.05)
ignore (float) – The fraction of the image ignored for sky consideration. Must be between 0. and 1. For more details we refer to the MUSE data reduction pipeline handbook. ([0., 1.[, default=0.05)
method (string) – The method how the determined sky spectrum is fitted to each spaxel to subtract the sky background. Should be set to ‘subtract-model’ in case of modified = True. For more details we refer to the MUSE data reduction pipeline handbook. (‘model’, ‘subtract-model’, ‘simple’, ‘none’, default=’model’)
create_sof (bool) – Must be set True if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually, than it must be set to False since it will be overwritten otherwise (false, true, default=true)

sci_post¶

Object Properties:

execute (bool) – Must be set True if this step should be executed (false, true, default=true)
autocalib (string) – This may execute the flux autocalibration between the different IFUs for empty fields. If user a AUTOCAL_FACTORS.fits table has to be provided by the user. (‘none’, ‘deepfield’, ‘user’, default=none)
subtract_sky (bool) – The user can decide to not run any sky subtraction. All telluric lines will be included in the final datacube. this may be useful if one wants to do their own wavelength calibration. (false, true, default=true)
raman (bool) – If laser guide stars are used raman scattering in the atmosphere may be visible in the final data cubes. If set to True the Raman lines are removed. For more details we refer to the MUSE data reduction pipeline handbook. (false, true, default=false)
create_sof (bool) – Must be set True if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually it must be set to False since it will be overwritten otherwise (false, true, default=true)

dither_collect¶

Object Properties:
	execute (bool) – Must be set `True` if this step should be executed (false, true, default=true) user_list (list) – A user list with the identifiers of the dither positions, in case only a subset shall be collected and copied to the final folder to be used for the combination. If left empty all dither positions are used to create the final data cube. (default=[])

exp_align¶

Object Properties:
	execute (bool) – Must be set `True` if this step should be executed (false, true, default=true) create_sof (bool) – Must be set `True` if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually, than it must be set to `False` since it will be overwritten otherwise (false, true, default=true)

exp_combine¶

Object Properties:

execute (bool) – Must be set True if this step should be executed (false, true, default=true)
weight (string) – The method how the fluxes in each dither position sre weighted. For more details we refer to the MUSE data reduction pipeline handbook. (‘exptime’, ‘fwhm’, ‘none’, ‘header’, default=exptime)
create_sof (bool) – Must be set True if one wants to create a new .sof file. In case the user wants to change or create the .sof file manually, than it must be set to False since it will be overwritten otherwise (false, true, default=true)

2.3. Utility methods¶

The following methods are routines to support the MUSEreduce.musereduce, needed to execute the various data reduction steps as it is suggested in the MUSE data reduction pipeline. They are documented here for completness.

MUSEreduce._get_filelist(self, data_dir, filename_wildcard)[source]¶

This module collects the necessary file lists from folders.

Args:

data_dir : str: The directory where the files are located.
filename_wildcard : str: The filenames which should be collected

MUSEreduce._call_esorex(self, exec_dir, esorex_cmd)[source]¶

This module calls the various ESOrex commands and gives it to the terminal for execution.

Args:

exec_dir : str: The directory where ESOrex should be executed.
esorex_cmd : str: The ESOrex command that needs to be executed

MUSEreduce._sort_data(self)[source]¶: This module reads the headers of all of the raw data and sorts it accordingly. It also assigns the correct calibration files to the exposures

MUSEreduce._bias(self, exp_list_SCI, exp_list_DAR, exp_list_TWI, create_sof)[source]¶

This module calls ESORex’ muse_bias

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

exp_list_DAR : list:

The list of all associated dark files including their category read from the fits header

exp_list_TWI : list:

The list of all associated twilight files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._dark(self, exp_list_SCI, exp_list_DAR, create_sof)[source]¶

This module calls ESORex’ muse_dark

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

exp_list_DAR : list:

The list of all associated dark files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._flat(self, exp_list_SCI, exp_list_TWI, create_sof)[source]¶

This module calls ESORex’ muse_flat

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

exp_list_TWI : list:

The list of all associated twilight files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._wavecal(self, exp_list_SCI, exp_list_TWI, create_sof)[source]¶

This module calls ESORex’ muse_wavecal

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

exp_list_TWI : list:

The list of all associated twilight files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._lsf(self, exp_list_SCI, exp_list_TWI, create_sof)[source]¶

This module calls ESORex’ muse_lsf

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

exp_list_TWI : list:

The list of all associated twilight files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._twilight(self, exp_list_SCI, exp_list_TWI, create_sof)[source]¶

This module calls ESORex’ muse_twilight

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

exp_list_TWI : list:

The list of all associated twilight files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._science_pre(self, exp_list_SCI, create_sof)[source]¶

This module calls ESORex’ muse_scibasic

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._std_flux(self, exp_list_SCI, create_sof)[source]¶

This module calls ESORex’ muse_standard

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._sky(self, exp_list_SCI, create_sof)[source]¶

This module calls ESORex’ muse_sky

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._modified_sky(self, exp_list_SCI, create_sof)[source]¶

This module calls ESORex’ muse_sky with the modified continuum and line subtraction as described in Zeidler et al. 2019.

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

MUSEreduce._scipost(self, exp_list_SCI, create_sof, OB)[source]¶

This module calls ESORex’ muse_scipost

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

OB : str:

The specific OB to be reduced.

MUSEreduce._dither_collect(self, exp_list_SCI, OB)[source]¶

This module collects the individual dither exposures for one OB to be combined in the steps: MUSEreduce._exp_align and MUSEreduce._exp_combine

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

OB : str:

The specific OB to be reduced.

MUSEreduce._exp_align(self, exp_list_SCI, create_sof, OB)[source]¶

This module calls ESORex’ muse_exp_align

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

OB : str:

The specific OB to be reduced.

MUSEreduce._exp_combine(self, exp_list_SCI, create_sof)[source]¶

This module calls ESORex’ muse_exp_combine

Args:

exp_list_SCI : list

The list of all associated science files including their category read from the fits header

create_sof : bool:

True: bias.sof is created and populated

False: bias.sof is not created and needs to be provided by the user

2.4. History¶

New in version 0.1.0: Executes the MUSE reduction pileline in the correct order

New in version 0.1.1: introducing only one calibration file folder per OB

New in version 0.1.2: choosing the illumination file closest to the observation

New in version 0.1.3: selecting the files for the different master file creations

New in version 0.1.4: minor corrections

New in version 0.1.5: looping order changed. each module loops by itself

New in version 0.1.6: always checking if calibration files already exist

New in version 0.1.7: Choosing between ESO calibrations and own calibrations

New in version 0.1.8: User can choose specific exposure time to reduce

New in version 0.2.0: exposures spread via different OBs for one pointing is supported. To do so: run the script normally for each OB including muse_scipost. After all are processed then run muse_exp_align and muse_exp_combine. AO observations are supported. To reduce multiple OBs in one run, set rootpath to manual.

New in version 0.2.1: user can choose the number of CPUs used

New in version 0.2.2: sky subtraction can be modified, so that individual elements can be excluded

New in version 0.2.3: use json file as input

New in version 0.2.4: additional parameters added: skyreject and skysubtraction. The set parameters and modules are shown in the output

New in version 0.2.5: bug fixes

New in version 0.3.0: using the correct ILLUM file for the STD reduction in the muse_sci_basic routine, muse_sci_basic now separated for STD and OBJECT reduction.

New in version 0.3.1: one can select if the sof file are created automatically or provided by the user

New in version 0.4.0: supports now pipeline v2.4.2 and the NFM-AO added: pipeline_path choosing if darks may be used only reduces STD once per OB general use of external SKY fields collecting the files for muse_exp_combine in an independent step muse_exp_align is an independent step now

New in version 0.4.1: new file names to correct a problem where data gets replaced in the muse_scipost routine if you reduce the data with and without sky

New in version 0.4.2: one can now change the ignore and fraction parameters in the json file

New in version 0.4.3: one can auto remove and rewrite the statics

New in version 0.4.4: changed the sky subtraction keyword the user can give now individual names for the different dither exposures: Does currently not work with multiple OBs or multiple pointings per OB

New in version 0.5.0: rewriting MUSEreduce.musereduce to a class and pep-8 style. DEBUG keyword added. Wrapper can be executed without running esorex but needs to be used with already existing reduced data.

New in version 0.5.1: added skymethod

New in version 0.5.2: MUSEreduce.musereduce can now handle if the exposures for one pointing are distributed via multiple OBs with multiple exposures in each OB.

New in version 1.0: The release version as originally published in Zeidler et al. 2019.

New in version 1.0.2: It is possible to omit the data reduction of the standard star in muse_scibasic by setting the execute_std to False. More skylines can be added in the muse_scibasic module but adding additional wavelengths in Angstrom to skylines.

New in version 1.1.0: MUSEreduce.musereduce supports now pipeline v2.8.1. A legacy version for pipeline v2.4.2 was created in a separate release. The keywords autocalib and in sci_oost was added. The naming convention in dither_collect was changed so that it matches the convention RADEC_EXPTIME_ROTANGLE_COUNTER.

New in version 1.1.1: The key words srcmin and srcmax were added to config.json to be used in exp_align.