INFO [2783287] ______________________________________________________________________ _____ ____ __ ____ __ _ _____ _ | ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | | | _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| | | |___ ___) | | | | \ V / (_| | | | | (_) | (_) | | |_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_| ______________________________________________________________________ ESMValTool - Earth System Model Evaluation Tool. http://www.esmvaltool.org CORE DEVELOPMENT TEAM AND CONTACTS: Birgit Hassler (Co-PI; DLR, Germany - birgit.hassler@dlr.de) Alistair Sellar (Co-PI; Met Office, UK - alistair.sellar@metoffice.gov.uk) Bouwe Andela (Netherlands eScience Center, The Netherlands - b.andela@esciencecenter.nl) Lee de Mora (PML, UK - ledm@pml.ac.uk) Niels Drost (Netherlands eScience Center, The Netherlands - n.drost@esciencecenter.nl) Veronika Eyring (DLR, Germany - veronika.eyring@dlr.de) Bettina Gier (UBremen, Germany - gier@uni-bremen.de) Remi Kazeroni (DLR, Germany - remi.kazeroni@dlr.de) Nikolay Koldunov (AWI, Germany - nikolay.koldunov@awi.de) Axel Lauer (DLR, Germany - axel.lauer@dlr.de) Saskia Loosveldt-Tomas (BSC, Spain - saskia.loosveldt@bsc.es) Ruth Lorenz (ETH Zurich, Switzerland - ruth.lorenz@env.ethz.ch) Benjamin Mueller (LMU, Germany - b.mueller@iggf.geo.uni-muenchen.de) Valeriu Predoi (URead, UK - valeriu.predoi@ncas.ac.uk) Mattia Righi (DLR, Germany - mattia.righi@dlr.de) Manuel Schlund (DLR, Germany - manuel.schlund@dlr.de) Breixo Solino Fernandez (DLR, Germany - breixo.solinofernandez@dlr.de) Javier Vegas-Regidor (BSC, Spain - javier.vegas@bsc.es) Klaus Zimmermann (SMHI, Sweden - klaus.zimmermann@smhi.se) For further help, please read the documentation at http://docs.esmvaltool.org. Have fun! INFO [2783287] Package versions INFO [2783287] ---------------- INFO [2783287] ESMValCore: 2.12.0.dev20+gd6eaac24e INFO [2783287] ESMValTool: 2.11.0.dev41+g1d5382aed INFO [2783287] ---------------- INFO [2783287] Using config file /nird/home/tomast/.esmvaltool/config-user.yml INFO [2783287] Writing program log files to: /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/run/main_log.txt /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/run/main_log_debug.txt /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/run/cmor_log.txt INFO [2783287] Using default ESGF configuration, configuration file /nird/home/tomast/.esmvaltool/esgf-pyclient.yml not present. INFO [2783287] Starting the Earth System Model Evaluation Tool at time: 2024-06-13 16:50:22 UTC INFO [2783287] ---------------------------------------------------------------------- INFO [2783287] RECIPE = /projects/NS2980K/home/tomast/ESMValTool/recipes-2.10.0/standard_recipes_NorESM/seaice/recipe_seaice_feedback_NorESM2.yml INFO [2783287] RUNDIR = /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/run INFO [2783287] WORKDIR = /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/work INFO [2783287] PREPROCDIR = /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/preproc INFO [2783287] PLOTDIR = /projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/plots INFO [2783287] ---------------------------------------------------------------------- INFO [2783287] Running tasks using at most 16 processes INFO [2783287] If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory. INFO [2783287] If you experience memory problems, try reducing 'max_parallel_tasks' in your user configuration file. WARNING [2783287] Using the Dask basic scheduler. This may lead to slow computations and out-of-memory errors. Note that the basic scheduler may still be the best choice for preprocessor functions that are not lazy. In that case, you can safely ignore this warning. See https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#dask-distributed-configuration for more information. INFO [2783287] Creating tasks from recipe INFO [2783287] Creating tasks for diagnostic NegativeSeaIceFeedback INFO [2783287] Creating diagnostic task NegativeSeaIceFeedback/negative_seaice_feedback INFO [2783287] Creating preprocessor task NegativeSeaIceFeedback/sithick INFO [2783287] Creating preprocessor 'default' task for variable 'sithick' INFO [2783287] Found input files for Dataset: sithick, SImon, CMIP6, NorESM2-LM, CMIP, historical, r1i1p1f1, gn, v20190815 INFO [2783287] Found input files for Dataset: sithick, SImon, CMIP6, NorESM2-MM, CMIP, historical, r1i1p1f1, gn, v20191108 INFO [2783287] PreprocessingTask NegativeSeaIceFeedback/sithick created. INFO [2783287] Creating preprocessor task NegativeSeaIceFeedback/areacello INFO [2783287] Creating preprocessor 'default' task for variable 'areacello' INFO [2783287] Found input files for Dataset: areacello, Ofx, CMIP6, NorESM2-LM, CMIP, historical, r1i1p1f1, gn, v20190815 INFO [2783287] Found input files for Dataset: areacello, Ofx, CMIP6, NorESM2-MM, CMIP, historical, r1i1p1f1, gn, v20191108 INFO [2783287] PreprocessingTask NegativeSeaIceFeedback/areacello created. INFO [2783287] These tasks will be executed: NegativeSeaIceFeedback/areacello, NegativeSeaIceFeedback/negative_seaice_feedback, NegativeSeaIceFeedback/sithick INFO [2783287] Wrote recipe with version numbers and wildcards to: file:///projects/NS2980K/www/tomast/esmvaltool2.10.0/output/recipe_seaice_feedback_NorESM2_20240613_165016/run/recipe_seaice_feedback_NorESM2_filled.yml INFO [2783287] Using the Dask basic scheduler. INFO [2783287] Running 3 tasks using 3 processes INFO [2783497] Starting task NegativeSeaIceFeedback/sithick in process [2783497] INFO [2783499] Starting task NegativeSeaIceFeedback/areacello in process [2783499] INFO [2783287] Progress: 2 tasks running, 1 tasks waiting for ancestors, 0/3 done INFO [2783499] Successfully completed task NegativeSeaIceFeedback/areacello (priority 2) in 0:00:01.447818 INFO [2783287] Progress: 1 tasks running, 1 tasks waiting for ancestors, 1/3 done ERROR [2783497] Failed to run preprocessor function 'fix_metadata' on the data [, , , ] loaded from original input file(s) [LocalFile('/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/sithick/gn/v20190815/sithick_SImon_NorESM2-LM_historical_r1i1p1f1_gn_197001-197912.nc'), LocalFile('/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/sithick/gn/v20190815/sithick_SImon_NorESM2-LM_historical_r1i1p1f1_gn_198001-198912.nc'), LocalFile('/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/sithick/gn/v20190815/sithick_SImon_NorESM2-LM_historical_r1i1p1f1_gn_199001-199912.nc'), LocalFile('/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/sithick/gn/v20190815/sithick_SImon_NorESM2-LM_historical_r1i1p1f1_gn_200001-200912.nc')] with function argument(s) check_level = , session = Session({'auxiliary_data_dir': PosixPath('/projects/NS9560K-datalake/ESGF/auxiliary_data'), 'check_level': , 'compress_netcdf': False, 'config_developer_file': PosixPath('/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/config-developer.yml'), 'config_file': PosixPath('/nird/home/tomast/.esmvaltool/config-user.yml'), 'diagnostics': None, 'download_dir': PosixPath('/scratch/tomast/ESGF'), 'drs': {'CMIP3': 'DKRZ', 'CMIP5': 'DKRZ', 'CMIP6': 'DKRZ', 'CORDEX': 'DKRZ', 'OBS': 'default', 'OBS6': 'default', 'ana4mips': 'default', 'native6': 'default', 'obs4MIPs': 'default'}, 'exit_on_warning': False, 'extra_facets_dir': (), 'log_level': 'info', 'max_datasets': None, 'max_parallel_tasks': 16, 'max_years': None, 'output_dir': PosixPath('/projects/NS2980K/www/tomast/esmvaltool2.10.0/output'), 'output_file_type': 'png', 'profile_diagnostic': False, 'remove_preproc_dir': False, 'resume_from': [], 'rootpath': {'CMIP3': [PosixPath('/projects/NS9560K-datalake/ESGF/cmip3')], 'CMIP5': [PosixPath('/projects/NS9560K-datalake/ESGF/cmip5/output1'), PosixPath('/projects/NS9034K/CMIP5/output1'), PosixPath('/scratch/tomast/ESGF/cmip5/output1')], 'CMIP6': [PosixPath('/projects/NS9560K-datalake/ESGF/CMIP6'), PosixPath('/projects/NS9034K/CMIP6'), PosixPath('/projects/NS2980K-datalake/shared/external/CMIP6'), PosixPath('/projects/NS9252K/KeyCLIM_CMOR'), PosixPath('/scratch/tomast/ESGF/CMIP6')], 'OBS': [PosixPath('/projects/NS9560K-datalake/ESGF/obsdata'), PosixPath('/scratch/tomast/ESGF/obsdata')], 'OBS6': [PosixPath('/projects/NS9560K-datalake/ESGF/obsdata'), PosixPath('/scratch/tomast/ESGF/obsdata')], 'RAWOBS': [PosixPath('/projects/NS9560K-datalake/ESGF/rawdata/obs'), PosixPath('/scratch/tomast/ESGF/rawdata/obs')], 'ana4mips': [PosixPath('/projects/NS9560K-datalake/ESGF/ana4mips'), PosixPath('/scratch/tomast/ESGF/ana4mips')], 'default': [PosixPath('/scratch/tomast/ESGF/obsdata')], 'native6': [PosixPath('/projects/NS9560K-datalake/ESGF/obsdata/native6'), PosixPath('/scratch/tomast/ESGF/obsdata/native6')], 'obs4MIPs': [PosixPath('/projects/NS9560K-datalake/ESGF/obs4MIPs'), PosixPath('/projects/NS9560K-datalake/ESGF/obsdata'), PosixPath('/scratch/tomast/ESGF/obs4MIPs')]}, 'run_diagnostic': True, 'save_intermediary_cubes': False, 'search_esgf': 'when_missing', 'skip_nonexistent': False, 'write_ncl_interface': False}), mip = 'SImon', grid = 'gn', project = 'CMIP6', exp = 'historical', dataset = 'NorESM2-LM', ensemble = 'r1i1p1f1', short_name = 'sithick', timerange = '1979/2004', variable_group = 'sithick', diagnostic = 'NegativeSeaIceFeedback', recipe_dataset_index = 0, alias = 'NorESM2-LM', institute = ['NCC'], activity = 'CMIP', original_short_name = 'sithick', standard_name = 'sea_ice_thickness', long_name = 'Sea Ice Thickness', units = 'm', modeling_realm = ['seaIce', 'ocean'], frequency = 'mon', version = 'v20190815' INFO [2783287] Maximum memory used (estimate): 1.8 GB INFO [2783287] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. ERROR [2783287] Program terminated abnormally, see stack trace below for more information: multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) ^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_task.py", line 816, in _run_task output_files = task.run() ^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_task.py", line 264, in run self.output_files = self._run(input_files) ^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/preprocessor/__init__.py", line 692, in _run product.cubes # pylint: disable=pointless-statement ^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/preprocessor/__init__.py", line 502, in cubes self._cubes = [ds.load() for ds in self.datasets] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/preprocessor/__init__.py", line 502, in self._cubes = [ds.load() for ds in self.datasets] ^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/dataset.py", line 685, in load cube = self._load() ^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/dataset.py", line 771, in _load result = preprocess( ^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/preprocessor/__init__.py", line 397, in preprocess result.append(_run_preproc_function(function, items, settings, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/preprocessor/__init__.py", line 346, in _run_preproc_function return function(items, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/cmor/fix.py", line 211, in fix_metadata cube = checker(cube).check_metadata() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/cmor/check.py", line 188, in check_metadata self.report_errors() File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/cmor/check.py", line 250, in report_errors raise CMORCheckError(msg) esmvalcore.cmor.check.CMORCheckError: There were errors in variable sithick: lon: has values > valid_max = 360.0 lat: has values > valid_max = 90.0 in cube: sea_ice_thickness / (m) (time: 120; cell index along second dimension: 384; cell index along first dimension: 360) Dimension coordinates: time x - - cell index along second dimension - x - cell index along first dimension - - x Auxiliary coordinates: latitude - x x longitude - x x Cell methods: 0 area: time: mean where sea_ice (mask=siconc mask=siconc) Attributes: Conventions 'CF-1.7 CMIP-6.2' activity_id 'CMIP' branch_method 'Hybrid-restart from year 1600-01-01 of piControl' branch_time 0.0 branch_time_in_child -674885 branch_time_in_parent -91250 cmor_version '3.5.0' contact 'Please send any requests or bug reports to noresm-ncc@met.no.' data_specs_version '01.00.31' experiment 'all-forcing simulation of the recent past' experiment_id 'historical' external_variables 'areacello' forcing_index 1 frequency 'mon' further_info_url 'https://furtherinfo.es-doc.org/CMIP6.NCC.NorESM2-LM.historical.none.r1 ...' grid 'tripolar grid with 1deg nominal resolution' grid_label 'gn' initialization_index 1 institution 'NorESM Climate modeling Consortium consisting of CICERO (Center for International ...' institution_id 'NCC' license 'CMIP6 model data produced by NCC is licensed under a Creative Commons Attribution ...' mip_era 'CMIP6' model_id 'NorESM2-LM' nominal_resolution '100 km' original_name 'sithick' parent_activity_id 'CMIP' parent_experiment_id 'piControl' parent_mip_era 'CMIP6' parent_source_id 'NorESM2-LM' parent_sub_experiment_id 'none' parent_time_units 'days since 1850-1-1 00:00:00' parent_variant_label 'r1i1p1f1' physics_index 1 product 'model-output' realization_index 1 realm 'seaIce' run_variant 'N/A' source 'NorESM2-LM (2017): \naerosol: OsloAero\natmos: CAM-OSLO (2 degree resolution; ...' source_file '/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/ ...' source_id 'NorESM2-LM' source_type 'AOGCM' sub_experiment 'none' sub_experiment_id 'none' table_id 'SImon' table_info 'Creation Date:(24 July 2019) MD5:08e314340b9dd440c36c1b8e484514f4' title 'NorESM2-LM output prepared for CMIP6' variable_id 'sithick' variant_label 'r1i1p1f1' loaded from file /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/sithick/gn/v20190815/sithick_SImon_NorESM2-LM_historical_r1i1p1f1_gn_197001-197912.nc """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_main.py", line 533, in run fire.Fire(ESMValTool()) File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_main.py", line 413, in run self._run(recipe, session) File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_main.py", line 455, in _run process_recipe(recipe_file=recipe, session=session) File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_main.py", line 130, in process_recipe recipe.run() File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_recipe/recipe.py", line 1096, in run self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks']) File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_task.py", line 738, in run self._run_parallel(address, max_parallel_tasks) File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_task.py", line 782, in _run_parallel _copy_results(task, running[task]) File "/projects/NS2980K/home/tomast/ESMValTool/ESMValCore/esmvalcore/_task.py", line 805, in _copy_results task.output_files, task.products = future.get() ^^^^^^^^^^^^ File "/projects/NS2980K/home/tomast/conda/esmvaltool-latest/lib/python3.11/multiprocessing/pool.py", line 774, in get raise self._value esmvalcore.cmor.check.CMORCheckError: There were errors in variable sithick: lon: has values > valid_max = 360.0 lat: has values > valid_max = 90.0 in cube: sea_ice_thickness / (m) (time: 120; cell index along second dimension: 384; cell index along first dimension: 360) Dimension coordinates: time x - - cell index along second dimension - x - cell index along first dimension - - x Auxiliary coordinates: latitude - x x longitude - x x Cell methods: 0 area: time: mean where sea_ice (mask=siconc mask=siconc) Attributes: Conventions 'CF-1.7 CMIP-6.2' activity_id 'CMIP' branch_method 'Hybrid-restart from year 1600-01-01 of piControl' branch_time 0.0 branch_time_in_child -674885 branch_time_in_parent -91250 cmor_version '3.5.0' contact 'Please send any requests or bug reports to noresm-ncc@met.no.' data_specs_version '01.00.31' experiment 'all-forcing simulation of the recent past' experiment_id 'historical' external_variables 'areacello' forcing_index 1 frequency 'mon' further_info_url 'https://furtherinfo.es-doc.org/CMIP6.NCC.NorESM2-LM.historical.none.r1 ...' grid 'tripolar grid with 1deg nominal resolution' grid_label 'gn' initialization_index 1 institution 'NorESM Climate modeling Consortium consisting of CICERO (Center for International ...' institution_id 'NCC' license 'CMIP6 model data produced by NCC is licensed under a Creative Commons Attribution ...' mip_era 'CMIP6' model_id 'NorESM2-LM' nominal_resolution '100 km' original_name 'sithick' parent_activity_id 'CMIP' parent_experiment_id 'piControl' parent_mip_era 'CMIP6' parent_source_id 'NorESM2-LM' parent_sub_experiment_id 'none' parent_time_units 'days since 1850-1-1 00:00:00' parent_variant_label 'r1i1p1f1' physics_index 1 product 'model-output' realization_index 1 realm 'seaIce' run_variant 'N/A' source 'NorESM2-LM (2017): \naerosol: OsloAero\natmos: CAM-OSLO (2 degree resolution; ...' source_file '/projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/ ...' source_id 'NorESM2-LM' source_type 'AOGCM' sub_experiment 'none' sub_experiment_id 'none' table_id 'SImon' table_info 'Creation Date:(24 July 2019) MD5:08e314340b9dd440c36c1b8e484514f4' title 'NorESM2-LM output prepared for CMIP6' variable_id 'sithick' variant_label 'r1i1p1f1' loaded from file /projects/NS9034K/CMIP6/CMIP/NCC/NorESM2-LM/historical/r1i1p1f1/SImon/sithick/gn/v20190815/sithick_SImon_NorESM2-LM_historical_r1i1p1f1_gn_197001-197912.nc INFO [2783287] If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.