comp_clim_miss_3d

Authors

Pascal Terray (LOCEAN/IPSL)

Latest revision

28/05/2024

Purpose

Compute a climatology (e.g., means and standard-deviations) from a tridimensional variable with missing values extracted from a NetCDF dataset and, optionally, the mask and scale factors of the 2-D grid-mesh associated with the input NetCDF variable.

It is assumed that the input NetCDF variable has a scalar missing_value or _FillValue attribute, which allows a proper identification of the missing values in the dataset, otherwise the procedure will exit with an error message.

Mean and standard-deviation are computed for each point in the time series of the 2-D grid-mesh associated with the input NetCDF variable. These statistics are computed by using all available observations for each grid-point time series. Since missing values are present, the number of observations used to compute the means and standard-deviations may vary from one grid-point to another in the 2-D grid-mesh associated with the NetCDF variable.

These means and standard-deviations may be computed by taking into account the periodicity of the data. These statistics are stored in an output NetCDF dataset.

Refer to comp_clim_3d, for details on how these univariate statistics are calculated in comp_clim_miss_3d. Moreover, if your data does not contain missing values in addition to those associated with a constant land-sea mask, use comp_clim_3d instead of comp_clim_miss_3d to estimate means and standard deviations from your dataset.

This procedure is parallelized if OpenMP is used and the NCSTAT software has been built with the _PARALLEL_READ CPP macro. Moreover, this procedure computes the means and standard-deviations with only one pass through the data and an out-of-core strategy which is highly efficient on huge datasets.

Optionally, a mesh-mask NetCDF dataset may also be created. This dataset will contain a presence-absence mask and scale factor variables which may be used to compute the surface of each cell in the 2-D grid-mesh associated with the input NetCDF variable. This mesh-mask NetCDF dataset will be used by other NCSTAT procedures such as comp_serie_miss_3d, comp_eof_miss_3d, etc. When computing the scale factors, it is assumed that the 2-D grid-mesh associated with the input NetCDF variable is regular or that this grid is a regular gaussian grid (the grid of the input NetCDF variable is assumed to be gaussian if this NetCDF variable has a grid_type character attribute with the string “gaussian” as a value).

Further Details

Usage

$ comp_clim_miss_3d \
  -f=input_netcdf_file \
  -v=netcdf_variable \
  -p=periodicity                    (optional) \
  -x=lon1,lon2                      (optional) \
  -y=lat1,lat2                      (optional) \
  -t=time1,time2                    (optional) \
  -c=output_climatology_netcdf_file (optional) \
  -m=output_mesh_mask_netcdf_file   (optional) \
  -np=nobs_limit_by_period          (optional) \
  -yl=latl1,latl2                   (optional) \
  -mi=missing_value                 (optional) \
  -ntr=number_of_time_records       (optional) \
  -double                           (optional) \
  -bigfile                          (optional) \
  -hdf5                             (optional) \
  -tlimited                         (optional)

By default

-p=
the periodicity is equal to 1
-x=
the whole longitude domain associated with the netcdf_variable
-y=
the whole latitude domain associated with the netcdf_variable
-t=
the whole time period associated with the netcdf_variable
-c=
the output_climatology_netcdf_file is named clim_netcdf_variable.nc
-m=
the output_mesh_mask_netcdf_file is not created
-np=
this argument is equal to 0
-yl=
it is assumed that the grid is gaussian and the domain is the whole globe or that the grid is regular when computing the scale factors
-mi=
the missing_value for the STD variable is equal to 1.e+20
-ntr=
the number_of_time_records read in each iteration of the procedure is set to the periodicity
-double
the standard-deviations are stored as single-precision floating point numbers in the output NetCDF file. If -double is activated, the standard-deviations are stored as double-precision floating point numbers in the output_climatology_netcdf_file
-bigfile
a NetCDF classical format file is created. If -bigfile is activated, the output NetCDF file is a 64-bit offset format file
-hdf5
a NetCDF classical format file is created. If -hdf5 is activated, the output NetCDF file is a NetCDF-4/HDF5 format file
-tlimited
the time dimension is defined as unlimited in the output NetCDF file. However, if -tlimited is activated, the time dimension is defined as limited in the output NetCDF file

Remarks

  1. The -v=netcdf_variable argument specifies the NetCDF variable for which statistics must be computed and the -f=input_netcdf_file argument specifies that this NetCDF variable must be extracted from the NetCDF file, input_netcdf_file.

  2. The -p=periodicity argument gives the periodicity of the input data. For example, with monthly data -p=12 should be specified, with yearly data -p=1 may be used, etc. By default, the periodicity is set to 1. Note that the output NetCDF file will have periodicity time observations.

  3. If the -x=lon1,lon2 and -y=lat1,lat2 arguments are missing, the whole geographical domain associated with the netcdf_variable is used to construct the climatology.

    The longitude or latitude range must be a vector of two integers specifying the first and last selected indices along each dimension. The indices are relative to 1. Negative values for lon1 are not allowed.

    Refer to comp_mask_3d for transforming geographical coordinates as indices before using comp_clim_miss_3d.

  4. If the -t=time1,time2 argument is missing, the whole time period associated with the netcdf_variable is used to construct the climatology. The selected time period is a vector of two integers specifying the first and last time observations. The indices are relative to 1.

  5. It is assumed that the specified netcdf_variable has a scalar missing_value or _FillValue attribute and that missing values in the data are identified by the value of this missing attribute.

  6. If the -m=output_mesh_mask_netcdf_file argument is present and if some scale factors can not be computed, these scale factors are set to 1.

  7. If the -m=output_mesh_mask_netcdf_file argument is present and the -yl= argument is missing, it is assumed when computing the scale factors that the 2-D grid-mesh associated with the input NetCDF variable is gaussian and covers the whole globe or that this 2-D grid-mesh is regular.

    If the grid is a regular gaussian grid, but the domain does not cover the whole globe, the -yl= argument must be used to specify the latitude boundaries of the domain, otherwise the first and last columns (elements) of the first two scale factors are wrong.

    The -yl= argument specifies the latitude limits of the domain in degrees (latl1 and latl2 must be real numbers) and the ordering of these two values must be coherent with the ordering of the latitudinal coordinate variable in the input_netcdf_file file.

  8. If the -np=nobs_limit_by_period and -m=output_mesh_mask_netcdf_file arguments are present, the mask in the output_mesh_mask_netcdf_file is constructed as follow:

    • if the number of observations by period (as determined by the -p= argument) is less than nobs_limit_by_period, the corresponding mask value is set to 0 (e.g., missing) otherwise, the mask value is set to 1.

    If the -np=nobs_limit_by_period argument is not specified and the -m=output_mesh_mask_netcdf_file argument is present, the mask is constructed as follow:

    • if the total number of non-missing observations is 0, the corresponding mask value is set to 0 (e.g., missing), otherwise the mask value is set to 1.

  9. The -mi=missing_value argument specifies the missing value indicator for the standard-deviation (STD) variable in the output_climatology_netcdf_file.

    If the -mi= argument is not specified and the NetCDF variable has a missing_value or _FillValue attribute, the missing_value is set to 1.e+20. This argument is not used if the NetCDF variable specified in the -v= argument has no missing_value or _FillValue attribute.

  10. The -ntr=number_of_time_records argument specifies how many time records are read in each iteration of the loop for reading the input NetCDF variable.

    By default, the number_of_time_records read in each iteration of the procedure is set to the periodicity (as specified by the -p= argument). On very large dataset, it may be useful to reduce the number_of_time_records in order to decrease the memory used by the procedure.

  11. The -double argument specifies that the standard-deviation variable must be stored as double-precision floating point numbers instead of single-precision floating point numbers in the output_climatology_netcdf_file.

  12. The -bigfile argument is allowed only if the NCSTAT software has been compiled with the _USE_NETCDF36 or _USE_NETCDF4 CPP macros (e.g., -D_USE_NETCDF36 or -D_USE_NETCDF4) and linked to the NetCDF 3.6 library or higher.

    If this argument is specified, the output_netcdf_file will be a 64-bit offset format file instead of a NetCDF classic format file. However, this argument is recognized in the procedure only if the NCSTAT software has been built with the _USE_NETCDF36 or _USE_NETCDF4 CPP macros.

  13. The -hdf5 argument is allowed only if the NCSTAT software has been compiled with the _USE_NETCDF4 CPP macro (e.g., -D_USE_NETCDF4) and linked to the NetCDF 4 library or higher.

    If this argument is specified, the output_netcdf_file will be a NetCDF-4/HDF5 format file instead of a NetCDF classic format file. However, this argument is recognized in the procedure only if the NCSTAT software has been built with the _USE_NETCDF4 CPP macro.

  14. Duplicate parameters are allowed, but this is always the last occurrence of a parameter which will be used for the computations. Moreover, the number of specified parameters must not be greater than the total number of allowed parameters.

  15. For more details on the use of univariate statistics in the climate literature, see

    • “Statistical Analysis in Climate Research”, by von Storch, H., and Zwiers, F.W., Cambridge University press, Cambridge, UK, Chapter 2, 484 pp., 2002. ISBN: 9780521012300

Outputs

comp_clim_miss_3d creates an output NetCDF file that contains the means, standard-deviations and number of observations of the input NetCDF variable, taking into account eventually the periodicity of the data as determined by the -p=periodicity argument. The output NetCDF dataset contains the following NetCDF variables (in the description below, nlat and nlon are the lengths of the spatial dimensions of the input NetCDF variable) :

  1. netcdf_variable_mean(periodicity,nlat,nlon) : the means for each point in the time series of the 2-D grid-mesh associated with the input NetCDF variable.
  2. netcdf_variable_std(periodicity,nlat,nlon) : the standard-deviations for each point in the time series of the 2-D grid-mesh associated with the input NetCDF variable.
  3. netcdf_variable_nobs(periodicity,nlat,nlon) : the number of observations used to compute the statistics for each point in the time series of the 2-D grid-mesh associated with the input NetCDF variable.

The means, standard-deviations and numbers of observations are packed in tridimensional variables whose first and second dimensions are exactly the same as those associated with the input NetCDF variable netcdf_variable even if you restrict the geographical domain with the -x= and -y= arguments. However, outside the selected domain, these output NetCDF variables are filled with missing values (zero values for the numbers of observations).

Optionally, comp_clim_miss_3d can also create an output mesh-mask NetCDF file that contains the following NetCDF variables :

  1. netcdf_variable_nmask(nlat,nlon) : a presence-absence or land-sea 2-D mask associated with the input NetCDF variable.
  2. netcdf_variable_e1n(nlat,nlon) : the first scale factor associated with the 2-D grid-mesh of the input NetCDF variable.
  3. netcdf_variable_e2n(nlat,nlon) : the second scale factor associated with the 2-D grid-mesh of the input NetCDF variable.

Multiplying the two scale factors together gives the surface of each cell in the 2-D grid-mesh associated with the input NetCDF variable.

Examples

  1. For computing monthly means and standard-deviations from the NetCDF file ST7_1m_00101_20012_grid_T_sosstsst.nc which includes a NetCDF variable sosstsst, which may possibly contain missing values and store the results in a NetCDF file named clim_ST7_1m_00101_20012_grid_T_sosstsst.nc, use the following command :

    $ comp_clim_miss_3d \
  -f=ST7_1m_0101_20012_grid_T_sosstsst.nc \
  -v=sosstsst \
  -p=12 \
  -c=clim_ST7_1m_00101_20012_grid_T_sosstsst.nc

  2. For computing monthly means and standard-deviations from the NetCDF file precip.mon.mean.nc, which includes a NetCDF variable precip and store the results in a NetCDF file named clim_precip_cmapstd.nc` and, in addition, generate an associated *mesh_mask_netcdf_file* named ``meshmask.cmapstd.3d.nc, use the following command (here the -np= argument is used to specify how to construct the mesh_mask_netcdf_file, see remarks above for further details) :

    $ comp_clim_miss_3d \
  -f=precip.mon.mean.nc \
  -v=precip \
  -p=12 \
  -np=10 \
  -c=clim_precip_cmapstd.nc \
  -m=meshmask.cmapstd.3d.nc
Flag Counter