INFO [6328] ______________________________________________________________________ _____ ____ __ ____ __ _ _____ _ | ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | | | _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| | | |___ ___) | | | | \ V / (_| | | | | (_) | (_) | | |_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_| ______________________________________________________________________ ESMValTool - Earth System Model Evaluation Tool. http://www.esmvaltool.org CORE DEVELOPMENT TEAM AND CONTACTS: Birgit Hassler (Co-PI; DLR, Germany - birgit.hassler@dlr.de) Alistair Sellar (Co-PI; Met Office, UK - alistair.sellar@metoffice.gov.uk) Bouwe Andela (Netherlands eScience Center, The Netherlands - b.andela@esciencecenter.nl) Lee de Mora (PML, UK - ledm@pml.ac.uk) Niels Drost (Netherlands eScience Center, The Netherlands - n.drost@esciencecenter.nl) Veronika Eyring (DLR, Germany - veronika.eyring@dlr.de) Bettina Gier (UBremen, Germany - gier@uni-bremen.de) Remi Kazeroni (DLR, Germany - remi.kazeroni@dlr.de) Nikolay Koldunov (AWI, Germany - nikolay.koldunov@awi.de) Axel Lauer (DLR, Germany - axel.lauer@dlr.de) Saskia Loosveldt-Tomas (BSC, Spain - saskia.loosveldt@bsc.es) Ruth Lorenz (ETH Zurich, Switzerland - ruth.lorenz@env.ethz.ch) Benjamin Mueller (LMU, Germany - b.mueller@iggf.geo.uni-muenchen.de) Valeriu Predoi (URead, UK - valeriu.predoi@ncas.ac.uk) Mattia Righi (DLR, Germany - mattia.righi@dlr.de) Manuel Schlund (DLR, Germany - manuel.schlund@dlr.de) Breixo Solino Fernandez (DLR, Germany - breixo.solinofernandez@dlr.de) Javier Vegas-Regidor (BSC, Spain - javier.vegas@bsc.es) Klaus Zimmermann (SMHI, Sweden - klaus.zimmermann@smhi.se) For further help, please read the documentation at http://docs.esmvaltool.org. Have fun! INFO [6328] Package versions INFO [6328] ---------------- INFO [6328] ESMValCore: 2.7.1 INFO [6328] ESMValTool: 2.7.1.dev35+gb1dbdd2b1 INFO [6328] ---------------- INFO [6328] Using config file /projects/NS2980K/home/tomast/ESMValTool/rundir_ipcc_node/config-ipcc_node_sigex.yml INFO [6328] Writing program log files to: /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/main_log.txt /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/main_log_debug.txt INFO [6328] Starting the Earth System Model Evaluation Tool at time: 2023-01-05 16:13:56 UTC INFO [6328] ---------------------------------------------------------------------- INFO [6328] RECIPE = /projects/NS2980K/home/tomast/ESMValTool/rundir_ipcc_node/recipe_ocean_Landschuetzer2016_NorESM2-sigex.yml INFO [6328] RUNDIR = /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run INFO [6328] WORKDIR = /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/work INFO [6328] PREPROCDIR = /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/preproc INFO [6328] PLOTDIR = /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/plots INFO [6328] ---------------------------------------------------------------------- INFO [6328] Running tasks using at most 8 processes INFO [6328] If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory. INFO [6328] If you experience memory problems, try reducing 'max_parallel_tasks' in your user configuration file. INFO [6328] Creating tasks from recipe INFO [6328] Creating tasks for diagnostic diag_timeseries_surface_average_vs_OBS INFO [6328] Creating diagnostic task diag_timeseries_surface_average_vs_OBS/Global_surface_timeseries INFO [6328] Creating preprocessor task diag_timeseries_surface_average_vs_OBS/dpco2 INFO [6328] Creating preprocessor 'prep_global_Surface_average_timeseries_2D' task for variable 'dpco2' INFO [6328] Found input files for CMIP6 r1i1p0f1 INFO [6328] Found input files for CMIP6 r1i1p100f1 WARNING [6328] Missing data for fx variable 'areacella' of dataset OBS WARNING [6328] Missing data for fx variable 'areacello' of dataset OBS INFO [6328] Found input files for OBS INFO [6328] PreprocessingTask diag_timeseries_surface_average_vs_OBS/dpco2 created. INFO [6328] Creating preprocessor task diag_timeseries_surface_average_vs_OBS/spco2 INFO [6328] Creating preprocessor 'prep_global_Surface_average_timeseries_2D' task for variable 'spco2' INFO [6328] Found input files for CMIP6 r1i1p0f1 INFO [6328] Found input files for CMIP6 r1i1p100f1 WARNING [6328] Missing data for fx variable 'areacella' of dataset OBS WARNING [6328] Missing data for fx variable 'areacello' of dataset OBS INFO [6328] Found input files for OBS INFO [6328] PreprocessingTask diag_timeseries_surface_average_vs_OBS/spco2 created. INFO [6328] Creating preprocessor task diag_timeseries_surface_average_vs_OBS/fgco2 INFO [6328] Creating preprocessor 'prep_global_Surface_average_timeseries_2D' task for variable 'fgco2' INFO [6328] Found input files for CMIP6 r1i1p0f1 INFO [6328] Found input files for CMIP6 r1i1p100f1 WARNING [6328] Missing data for fx variable 'areacella' of dataset OBS WARNING [6328] Missing data for fx variable 'areacello' of dataset OBS INFO [6328] Found input files for OBS INFO [6328] PreprocessingTask diag_timeseries_surface_average_vs_OBS/fgco2 created. INFO [6328] Creating tasks for diagnostic diag_surface_maps_vs_OBS INFO [6328] Creating diagnostic task diag_surface_maps_vs_OBS/Global_Ocean_surface_map INFO [6328] Creating diagnostic task diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs INFO [6328] Creating preprocessor task diag_surface_maps_vs_OBS/dpco2 INFO [6328] Creating preprocessor 'prep_surface_map_2D' task for variable 'dpco2' INFO [6328] Found input files for CMIP6 r1i1p0f1 INFO [6328] Found input files for CMIP6 r1i1p100f1 INFO [6328] Found input files for OBS INFO [6328] PreprocessingTask diag_surface_maps_vs_OBS/dpco2 created. INFO [6328] Creating preprocessor task diag_surface_maps_vs_OBS/spco2 INFO [6328] Creating preprocessor 'prep_surface_map_2D' task for variable 'spco2' INFO [6328] Found input files for CMIP6 r1i1p0f1 INFO [6328] Found input files for CMIP6 r1i1p100f1 INFO [6328] Found input files for OBS INFO [6328] PreprocessingTask diag_surface_maps_vs_OBS/spco2 created. INFO [6328] Creating preprocessor task diag_surface_maps_vs_OBS/fgco2 INFO [6328] Creating preprocessor 'prep_surface_map_2D' task for variable 'fgco2' INFO [6328] Found input files for CMIP6 r1i1p0f1 INFO [6328] Found input files for CMIP6 r1i1p100f1 INFO [6328] Found input files for OBS INFO [6328] PreprocessingTask diag_surface_maps_vs_OBS/fgco2 created. INFO [6328] These tasks will be executed: diag_surface_maps_vs_OBS/spco2, diag_timeseries_surface_average_vs_OBS/fgco2, diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs, diag_surface_maps_vs_OBS/Global_Ocean_surface_map, diag_surface_maps_vs_OBS/fgco2, diag_surface_maps_vs_OBS/dpco2, diag_timeseries_surface_average_vs_OBS/dpco2, diag_timeseries_surface_average_vs_OBS/Global_surface_timeseries, diag_timeseries_surface_average_vs_OBS/spco2 INFO [6328] Running 9 tasks using 8 processes INFO [6483] Starting task diag_timeseries_surface_average_vs_OBS/dpco2 in process [6483] INFO [6486] Starting task diag_timeseries_surface_average_vs_OBS/spco2 in process [6486] INFO [6487] Starting task diag_timeseries_surface_average_vs_OBS/fgco2 in process [6487] INFO [6488] Starting task diag_surface_maps_vs_OBS/dpco2 in process [6488] INFO [6489] Starting task diag_surface_maps_vs_OBS/spco2 in process [6489] INFO [6490] Starting task diag_surface_maps_vs_OBS/fgco2 in process [6490] INFO [6328] Progress: 6 tasks running, 3 tasks waiting for ancestors, 0/9 done WARNING [6487] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file /projects/NS2980K/data/modeldata/NorESM/cmor/NorESM2-LM/generic/sigex-L0/fgco2_Omon_NorESM2-LM_generic_r1i1p0f1_gn_200001-200912.nc WARNING [6487] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file WARNING [6490] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file /projects/NS2980K/data/modeldata/NorESM/cmor/NorESM2-LM/generic/sigex-L100/fgco2_Omon_NorESM2-LM_generic_r1i1p100f1_gn_200001-200912.nc WARNING [6490] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file WARNING [6490] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file /projects/NS2980K/data/modeldata/NorESM/cmor/NorESM2-LM/generic/sigex-L0/fgco2_Omon_NorESM2-LM_generic_r1i1p0f1_gn_200001-200912.nc WARNING [6490] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file WARNING [6490] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file /projects/NS2980K/data/obsdata/cmor/Tier2/Landschuetzer2016/OBS_Landschuetzer2016_clim_v2016_Omon_fgco2_198201-201512.nc WARNING [6490] There were warnings in variable fgco2: fgco2: attribute positive not present loaded from file INFO [6490] Successfully completed task diag_surface_maps_vs_OBS/fgco2 (priority 8) in 0:00:18.013521 INFO [6328] Progress: 5 tasks running, 3 tasks waiting for ancestors, 1/9 done INFO [6489] Successfully completed task diag_surface_maps_vs_OBS/spco2 (priority 7) in 0:00:18.371917 INFO [6488] Successfully completed task diag_surface_maps_vs_OBS/dpco2 (priority 6) in 0:00:18.445734 INFO [6328] Progress: 4 tasks running, 3 tasks waiting for ancestors, 2/9 done INFO [6328] Progress: 3 tasks running, 3 tasks waiting for ancestors, 3/9 done INFO [6491] Starting task diag_surface_maps_vs_OBS/Global_Ocean_surface_map in process [6491] INFO [6492] Starting task diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs in process [6492] INFO [6491] Running command ['/projects/NS2980K/home/tomast/conda/esmvaltool/bin/python', '/projects/NS2980K/home/tomast/ESMValTool/ESMValTool_dev/esmvaltool/diag_scripts/ocean/diagnostic_maps.py', '/projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/settings.yml'] INFO [6491] Writing output to /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/work/diag_surface_maps_vs_OBS/Global_Ocean_surface_map INFO [6491] Writing plots to /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/plots/diag_surface_maps_vs_OBS/Global_Ocean_surface_map INFO [6491] Writing log to /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/log.txt INFO [6491] To re-run this diagnostic script, run: cd /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_surface_map; MPLBACKEND="Agg" /projects/NS2980K/home/tomast/conda/esmvaltool/bin/python /projects/NS2980K/home/tomast/ESMValTool/ESMValTool_dev/esmvaltool/diag_scripts/ocean/diagnostic_maps.py /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_surface_map/settings.yml INFO [6492] Running command ['/projects/NS2980K/home/tomast/conda/esmvaltool/bin/python', '/projects/NS2980K/home/tomast/ESMValTool/ESMValTool_dev/esmvaltool/diag_scripts/ocean/diagnostic_model_vs_obs.py', '/projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs/settings.yml'] INFO [6492] Writing output to /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/work/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs INFO [6492] Writing plots to /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/plots/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs INFO [6492] Writing log to /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs/log.txt INFO [6492] To re-run this diagnostic script, run: cd /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs; MPLBACKEND="Agg" /projects/NS2980K/home/tomast/conda/esmvaltool/bin/python /projects/NS2980K/home/tomast/ESMValTool/ESMValTool_dev/esmvaltool/diag_scripts/ocean/diagnostic_model_vs_obs.py /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs/settings.yml INFO [6328] Progress: 5 tasks running, 1 tasks waiting for ancestors, 3/9 done INFO [6492] Maximum memory used (estimate): 0.3 GB INFO [6492] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. INFO [6328] Maximum memory used (estimate): 5.1 GB INFO [6328] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. ERROR [6328] Program terminated abnormally, see stack trace below for more information: multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 796, in _run_task output_files = task.run() File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 263, in run self.output_files = self._run(input_files) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 594, in _run raise DiagnosticError( esmvalcore._task.DiagnosticError: Diagnostic script ocean/diagnostic_model_vs_obs.py failed with return code 1. See the log in /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs/log.txt """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 499, in run fire.Fire(ESMValTool()) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 443, in run process_recipe(recipe_file=recipe, config_user=cfg) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 124, in process_recipe recipe.run() File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_recipe.py", line 1921, in run self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks']) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 725, in run self._run_parallel(max_parallel_tasks) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 768, in _run_parallel _copy_results(task, running[task]) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 791, in _copy_results task.output_files, task.products = future.get() File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/multiprocessing/pool.py", line 774, in get raise self._value File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 796, in _run_task output_files = task.run() File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 263, in run self.output_files = self._run(input_files) File "/projects/NS2980K/home/tomast/conda/esmvaltool/lib/python3.10/site-packages/esmvalcore/_task.py", line 594, in _run raise DiagnosticError( esmvalcore._task.DiagnosticError: Diagnostic script ocean/diagnostic_model_vs_obs.py failed with return code 1. See the log in /projects/NS2980K/www/tomast/esmvaltool2.7.0/recipe_ocean_Landschuetzer2016_NorESM2-sigex_20230105_161354/run/diag_surface_maps_vs_OBS/Global_Ocean_model_vs_obs/log.txt INFO [6328] If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.