OPH_IMPORTNC2

Description

Type

Data Process.

Behaviour

It imports a NetCDF file into a datacube (both measure and dimensions). Optimized version of IMPORTNC. WARNING: It imports only mono-dimensional coordinate variables.

Parameters

  • container: name of the input container; by default it will be automatically set to file name.
  • cwd: absolute path corresponding to the current working directory, used to select the folder where the container is located.
  • host_partition: name of I/O host partition used to store data. If default value “auto” is specified, the first host partition available will be used.
  • filesystem: type of filesystem used to store data. Possible values are “local”, “global” or “auto” (default). In the last case the first filesystem available will be used.
  • ioserver: type of I/O server used to store data. Only possible value is “ophidiaio_memory”(default).
  • import_metadata: with “yes” (default), it will import also metadata; with “no”, it will import only data.
  • check_compliance: checks if all the metadata registered for reference vocabulary are available. No check is done by default.
  • schedule: scheduling algorithm. The only possible value is 0, for a static linear block distribution of resources.
  • nhost: number of output hosts. With default value (‘0’), all host available in the host partition are used.
  • ndbms: number of the output DBMS per host. With default value (‘0’), all DBMS instance available per host are used.
  • ndb: number of output database per host. Default value is ‘1’.
  • nfrag: number of fragments per database. With default value (‘0’), the number of fragments will be the ratio of the product of sizes of the n-1 most outer explicit dimensions to the product of the other arguments.
  • measure: name of the measure related to the NetCDF file.
  • run: if set to ‘no’, the operator simulates the import and computes the fragmentation parameters that would be used else, if set to ‘yes’, the actual import operation is executed.
  • src_path: path or OPeNDAP URL of the NetCDF file. Local files have to be stored in folder BASE_SRC_PATH or its sub-folders (see oph_configuration for additional information).
  • cdd: absolute path corresponding to the current directory on data repository. It is appended to BASE_SRC_PATH to build the effective path to files (see configuration notes for further details).
  • exp_dim: names of explicit dimensions. Multiple-value field: list of dimensions separated by “|” can be provided. It implicitly defines explicit dimension number and level (order). If default value “auto” is specified, then the first n-1 dimension of the measure in the NetCDF file will be used as explicit dimensions.
  • imp_dim: names of implicit dimensions. Multiple-value field: list of dimensions separated by “|” can be provided. It implicitly defines implicit dimension number and level (order) and must be the same number of “imp_concept_level”. If default value “auto” is specified, then the last dimension of the measure in the NetCDF file will be used as implicit dimension.
  • subset_dims: dimension names used for the subsetting. Multiple-value field: list of dimensions separated by “|” can be provided and must be the same number of “subset_filter”.
  • subset_filter: enumeration of comma-separated elementary filters (1 series of filters for each dimension). Multiple-value field: list of filters separated by “|” can be provided and must be the same number of “subset_dims”. Values should be numbers. Example: subset_dims=lat|lon;subset_filter=35:45|15:20. Possible forms are:
    • start_value: single value specifying the start index of the subset
    • start_value:stop_value: select elements from start_index to stop_index.
  • subset_type: if set to “index” (default), the subset_filter is considered on dimension index. With “coord”, filter is considered on dimension values. In case of single value, that value is used for all the dimensions.
  • time_filter: enable filters using dates for time dimensions; enabled by default.
  • offset: it is added to the bounds of subset intervals defined with “subset_filter” in case of “coord” filter type is used.
  • exp_concept_level: concept level short name (must be a single char) of explicit dimensions. Default value is “c”. Multiple-value field: list of concept levels separated by “|” can be provided and must be the same number of “exp_dim”.
  • imp_concept_level: concept level short name (must be a single char) of implicit dimensions. Default value is “c”. Multiple-value field: list of concept levels separated by “|” can be provided.
  • compressed: with “yes”, it will save compressed data; with “no” (default), it will save original data.
  • grid: optional argument used to identify the grid of dimensions to be used (if the grid already exists) or the one to be created (if the grid has a new name). If it isn’t specified, no grid will be used.
  • hierarchy: concept hierarchy name of the dimensions. Default value is “oph_base”. Multiple-value field: list of concept hierarchies separated by “|” can be provided.
  • vocabulary: optional argument used to indicate a vocabulary (name of set of keys) to be used to associate metadata to the container.
  • base_time: in case of time hierarchy, it indicates the base time of the dimension. Default value is 1900-01-01.
  • units: in case of time hierarchy, it indicates the units of the dimension. Possible values are: s, m, h, 3, 6, d.
  • calendar: in case of time hierarchy, it indicates the calendar type. Possible values are:
    • standard (default)
    • gregorian
    • proleptic_gregorian
    • julian
    • 360_day
    • no_leap
    • all_leap
    • user_defined
  • month_lengths: in case of time dimension and user-defined calendar, it indicates the sizes of each month in days. There must be 12 positive integers separated by commas. Default is ‘31,28,31,30,31,30,31,31,30,31,30,31’.
  • leap_year: in case of time dimension and user-defined calendar, it indicates the first leap year. By default it is set to 0.
  • leap_month: in case of time dimension and user-defined calendar, it indicates the leap month. By default it is set to 2 (i.e. February).
  • check_grid: optional flag to be enabled in case the values of grid have to be checked (valid only if the grid already exists).
  • description: additional description to be associated with the output cube.

System parameters

  • exec_mode: operator execution mode. Possible values are async (default) for asynchronous mode, sync for synchronous mode with json-compliant output.
  • ncores: number of parallel processes to be used (min. 1).
  • nthreads: number of parallel threads per process to be used (min. 1).
  • sessionid: session identifier used server-side to manage sessions and jobs. Usually, users don’t need to use/modify it, except when it is necessary to create a new session or switch to another one.
  • objkey_filter: filter on the output of the operator written to file (default=all => no filter, none => no output, importnc => show operator’s output PID as text).

Examples

Import a NetCDF file excluding metadata into the session directory “session-code1”:

[OPH_TERM] >>  oph_importnc2 cwd=/session-code1;container=container1;measure=pressure;src_path=/path/of/ncfile.nc;imp_concept_level=d;import_metadata=no;

Arguments

Argument name Type Mandatory Values Default Min/Max-value
sessionid “string” “no”   “null”  
ncores “int” “no”   “1” “1” /
nthreads “int” “no”   “1” “1” /
exec_mode “string” “no” “async|sync” “async”  
container “string” “no”   “-“  
cwd “string” “yes”      
host_partition “string” “no”   “auto”  
filesystem “string” “no” “auto|local|global” “auto”  
ioserver “string” “no” “ophidiaio_memory” “ophidiaio_memory”  
import_metadata “string” “no” “yes|no” “yes”  
check_compliance “string” “no” “yes|no” “no”  
schedule “int” “no” “0” “0”  
nhost “int” “no”   “0” “0” /
ndbms “int” “no”   “1” “1” /
ndb “int” “no”   “1” “1” /
nfrag “int” “no”   “0” “0” /
measure “string” “yes”      
run “string” “no” “yes|no” “yes”  
src_path “string” “yes”      
cdd “string” “no”   “/”  
exp_dim “string” “no”   “auto”  
imp_dim “string” “no”   “auto”  
subset_dims “string” “no”   “none”  
subset_filter “string” “no”   “all”  
subset_type “string” “no” “index|coord” “index”  
time_filter “string” “no” “yes|no” “yes”  
offset “real” “no”   “0”  
exp_concept_level “string” “no”   “c”  
imp_concept_level “string” “no”   “c”  
compressed “string” “no” “yes|no” “no”  
grid “string” “no”   “-“  
hierarchy “string” “no”   “oph_base”  
vocabulary “string” “no”   “CF”  
base_time “string” “no”   “1900-01-01 00:00:00”  
units “string” “no” “s|m|h|3|6|d” “d”  
calendar “string” “no” “standard|gregorian|proleptic_gregorian|julian|360_day|no_leap|all_leap|user_defined” “standard”  
month_lengths “string” “no”   “31,28,31,30,31,30,31,31, 30,31,30,31”  
leap_year “int” “no”   “0” “0” /
leap_month “int” “no”   “2” “1” / “12”
check_grid “string” “no” “yes|no” “yes”  
description “string” “no”   “-“  
objkey_filter “string” “no” “all|none|importnc” “all”