Ophidia Terminal: advanced features

Ophidia terminal is a robust, comprehensive, effective and extremely usable command line client. Essential information of the software is described in the the terminal basic usage section. This guide describes the client-server interaction and the advanced features provided by the terminal, such as history management, auto-completion, output rendering functionality, variable substitution and much more.

Client-server interaction

Oph-Terminal contains a client function that is concerned to interact with the server through the SOAP protocol. SOAP may operate on different network protocols, among which the most commonly used is HTTP. In Ophidia we chose to use the HTTPS to secure communication with SSL or GSI/VOMS infrastructure.

SOAP is based also on XML metalanguage for defining parameters required for communication, and for containing the so-called payload, i.e. the content of the sent or received message, whose structure follows the Head-Body configuration, similar to HTML.

For each user request, the terminal sends a new message with the appropriate connection parameters and the string corresponding to the request for execution of a particular operator, which is then interpreted by the functions of the Ophidia framework.

Once the operator execution terminates, Oph-Terminal receives the corresponding response message with the output data of the operator, that are then interpreted and displayed appropriately to the user.

Oph-Terminal environment

Oph-Terminal provides different environment variables and commands, that can be used for storing recurring values and performing “local” and “remote” operations.

A description of the environment variables follows:

  • OPH_TERM_PS1 is the color of the prompt (default is red);
  • OPH_USER is the username to be used to access Ophidia;
  • OPH_PASSWD is the password to be used to access Ophidia;
  • OPH_SERVER_HOST is the address/DNS name of the host where Ophidia Server is running;
  • OPH_SERVER_PORT is the port number of Ophidia Server;
  • OPH_SESSION_ID is the identifier of current session (initialized after the first remote request);
  • OPH_EXEC_MODE is the execution mode: async for asynchronous mode (default) or sync for synchronous mode;
  • OPH_NCORES is the number of cores to be used by default (default is 1);
  • OPH_CWD is the current working directory within Ophidia virtual file system;
  • OPH_CDD is the current data directory within the data repository rooted at BASE_SRC_PATH (see framework configuration notes for further details);
  • OPH_DATACUBE is the current value of parameter cube (taken from output of operators that create a new datacube);
  • OPH_TERM_VIEWER is the output renderer and should be set to one of the following values: dump (view output as it is, in JSON format), basic (view output in tabular format), coloured (basic, with colors), extended (same as basic but with added information about data sources, consumers, producers, objkeys, etc.), extended_coloured (extended, with colors);
  • OPH_TERM_FORMAT defines the output format of the command ‘view’: classic (print all the data in JSON Response), compact (print the JSON Response without the Workflow Task List);
  • OPH_TERM_IMGS indicates whether graphs and/or trees produced as output of some operators have to be saved on local disk as images and/or auto-opened on the screen: no_op (images are not processed), open (images are created, saved on disk and automatically opened) or save (images are created and saved on disk but not opened);
  • OPH_GRAPH_LAYOUT sets the graph layout in case of a command producing an image file representing a graph. The values available are: horizontal_compact (nodes are arranged from left to right), vertical_compact (nodes are arranged from top to bottom), horizontal_ranked (nodes are arranged from left to right and according to their rank/depth), vertical_ranked (nodes are arranged from top to bottom and according to their rank/depth);
  • OPH_RESPONSE_BUFFER represents the maximum size (in kB) of a response in order to be rendered by Oph_Term;
  • OPH_WORKFLOW_AUTOVIEW if set to on Oph_Term will automatically call the command “view <wid> 0 5” after a workflow submission (<wid> is the related workflow identifier); this option enables the workflow to be automatically shown graphically, provided that OPH_TERM_IMGS is set to open. In case of synchronous execution mode (exec_mode set to sync) only the final representation is shown. Otherwise, in case of asynchronous execution mode (exec_mode set to async), the workflow is shown even during the execution; then, the image is periodically updated until the final representation. If set to on_X_Y Oph_Term will call the command “view <wid> <X> <Y>”, using “X” as the number of iterations and “Y” as the time interval [in seconds]; the default value off prevents this behaviour from happening;
  • OPH_TOKEN is the token to be used to access Ophidia;
  • OPH_HOST_PARTITION is the name of the default host partition used for import operations;
  • OPH_TIMEOUT is the maximum waiting time (in seconds) for a reply after submitting a command (default is 3600).

The program environment also includes several commands among which:

  • help [cmd | var]: with no arguments it lists all available commands and variables; adding the specific name of a command or a variable you can read a more detailed description, namely its usage;
  • history [-c]: it prints out the contents of the file .oph_term_history placed in the user home with the list of commands previously executed, with specification of time and date in a user-friendly format like dd/mm/yyyy hh:mm:ss; with the -c any content of the same file (not the file itself) is deleted;
  • env: it displays the contents of all the environment variables;
  • setenv env_var=”value”: it assigns value to the environment variable env_var, e.g. setenv OPH_USER=”user1” can be used to set the username for accessing the server to the value user1;
  • unsetenv env_var: it deletes the contents of the environment variable env_var;
  • getenv env_var: it displays the contents of the environment variable env_var, if any;
  • clear: it “cleans” the screen of Oph-Terminal;
  • quit (or exit): it terminates the program;
  • alias: list all aliases with their values;
  • setalias ALIAS=”value”: set the alias ALIAS to “value”;
  • unsetalias ALIAS: clear the content of the alias ALIAS;
  • getalias ALIAS: print the value of the alias ALIAS (if it exists);
  • version: show the current version of the terminal;
  • update [-f]: download the latest version of the XML files of all active operators for the specified server (OPH_SERVER_HOST:OPH_SERVER_PORT) in order to use the Oph_Term autocompletion features on Ophidia operators. OPH_SERVER_HOST and OPH_SERVER_PORT must be set before launching this command. Local files removed from server are NOT preserved locally. In case of -f option, force Oph_Term to completely remove all local files and download the latest version of the remote XMLs.
  • resume [SessionID] [n] [-v]: switch to another session specifying a new SessionID (or consider the current session) and view at the same time (at most) the last “n” commands launched in that session (default number is 10). The output will have the format [workflowid] [status] command. With -v the output will have the format [workflowid] [status] command <newline> [jobid];
  • view WorkflowID[#MarkerID] [-s status] [iterations [interval]] or view -j JobID [-s status] [iterations [interval]]: view the output/status of a workflow associated to the current session, identified by a workflow id, or view the output/status of a particular task associated to the current session, identified by a workflow id and a marker id. With option -j view the output/status associated to a particular JobID, even belonging to a different session. The output of a single-and-simple-task workflow will be the output of the task, the output of a single-and-massive-task workflow will be the list of tasks generated by the massive task (with marker ids), the output of a multi-task workflow will be the list of the workflow tasks (with marker ids). To view the output of a task within a generic workflow, use the JobID or a combination of workflow id and marker id. The output will be prefixed with a string with the format [workflowid#markerid] command [jobid]. According to the value of OPH_TERM_IMGS, it can produce an image representing the real-time status of a multi-task workflow. For running workflows, the option -s can be used to filter jobs based on their status (see status_filter of OPH_RESUME for further information). It is also possible to specify the number of iterations the command will perform (0 for viewing status until workflow ends) and the time interval (in seconds) between iterations (default value is 5 seconds).
  • check [-m] workflow.json [param1 [param2 [...]]]: validate “workflow.json” against the Ophidia Workflow JSON Schema. With one or more parameters, the parameter substitution ($1,$2 etc.) is performed before the validation phase together with variable substitution. After the validation phase, according to the value of OPH_TERM_IMGS, it can produce an image representing the workflow. With the option -m it returns the metadata relative to name,author and abstract and, if present, sessionid,exec_mode,ncores,cwd,cube,callback_url,on_error,command,on_exit and run. In this case validation is not performed. Finally, in order to call an Ophidia operator, the user can directly specify its name on the command line with a series of key-value pairs with all appropriate parameters.
  • watch [-n seconds] cmd: runs the command “cmd” repeatedly, displaying its output. By default, the program is run every 2 seconds; use -n to specify a different interval.

Internally the commands resume and view are translated into sequences of calls for the operator OPH_RESUME.

For instance, in case the user wishes to switch from current session to session http://hostname/ophidia/sessions/8912794871982734981234/experiment, the command

resume http://hostname/ophidia/sessions/8912794871982734981234/experiment 3

is translated into several commands for Ophidia Server. One is used to retrieve the number of commands/workflows already submitted in the new session:

oph_resume session=http://hostname/ophidia/sessions/8912794871982734981234/experiment;

Then, there are the requests to retrieve the last three commands submitted in the new session and the related exit status (provided that at least three commands were actually submitted). Assuming that the number of commands already submmited is 15, the requests for the last three commands are

oph_resume session=http://hostname/ophidia/sessions/8912794871982734981234/experiment;document_type=request;id=13;
oph_resume session=http://hostname/ophidia/sessions/8912794871982734981234/experiment;document_type=request;id=14;
oph_resume session=http://hostname/ophidia/sessions/8912794871982734981234/experiment;document_type=request;id=15;

Finally, there are the commands to change the session

oph_resume session=http://hostname/ophidia/sessions/8912794871982734981234/experiment;save=yes;

and retrieve the configuration parameters of the new session (last working directory and idenfier of the last created cube)

oph_get_config key=OPH_CWD;
oph_get_config key=OPH_DATACUBE;

The following command

view 10

corresponds to the following pair of submission strings

oph_resume id=20;document_type=request;
oph_resume id=20;

where the former is used to retrieve the submission string associated with the workflow, the latter returns the workflow task list (or directly the task output in case of single-task workflows).

Similarly, the command

view 10#20;

is translated into the following calls

oph_resume id=20;id_type=marker;document_type=request;
oph_resume id=20;id_type=marker;

History management

The Readline library and particularly the History library allow convenient management of commands history in Oph-Terminal. Immediately after starting the terminal, the file .oph_term_history is loaded in memory as a list of all the commands called earlier with precise information about the date and time of execution.

You can then navigate in the history simply using the arrow keys up and down on your keyboard to display respectively the previous command and the next one, or you can make use of the character ! for history expansion. For example, you can use, along with the exclamation mark, the number corresponding to a particular command, or you can write a command, in whole or in part, to recall the last entry in history for that command.

In addition to different shortcuts for editing the command line, such as the combinations Ctrl + a and Ctrl + e to move to the beginning or end of the line, Oph-Terminal also provides the keystrokes Esc + < and Esc + > to move to the beginning or end of the commands history. Note that all realized combinations faithfully reproduce what is already present in a system shell like bash, such as the recursive search of a command within history with Ctrl + r.

When closing the terminal, the memory containing the commands list is deallocated. Then the user can log back into Oph-Terminal with automatic recovery of the commands related to the previous session. During normal program execution, finally, the user can view the history as a whole or possibly erase it with the history command previously described.

Embedded help and auto-completion

Oph-Terminal provides a number of features to help the user and make it easier to enter commands. The help command is a good starting point if you’re interfacing with the terminal for the first time, it gives a description of all commands and variables in the program.

In addition, anything written in the command line undergoes an intelligent parsing: for example extra spaces positioned at the beginning or end of the line are identified and not considered, and care is taken for the presence of quotation marks (”), quotes (‘) or special characters. However, since each command requires distinct arguments and/or options, you should refer to the manual to avoid making mistakes.

Another interesting feature of Oph-Terminal is the auto-completion. Using, as a normal shell, the TAB key, we can make the terminal complete the word in question on the basis of the characters already written.

Different levels of auto-completion have been implemented, making use of all the XMLs describing the Ophidia operators made available by the server and downloaded by Oph-Terminal at startup:

  • auto-completion over Oph-Terminal commands;
  • auto-completion over Oph-Terminal environment variables;
  • auto-completion over Ophidia operators;
  • auto-completion over the name of the arguments of each operator;
  • auto-completion over all possible values for the arguments of each operator.

And if you find yourself with more options to complete a particular word, just re-use the tab key to get a list of all the relative terms or use the Shift key together with the tab key to perform an inline completion.

Rendering output

With the OPH_TERM_VIEWER variable the user is able to appropriately configure the visualization component, implemented with the aim of managing in the best way the rendering of the responses received from the Ophidia server after the submission of a command. Each response will be structured in a JSON file, which may include objects of different types:

  • simple text;
  • grids and tables;
  • trees and graphs.

The viewer is able to recognize this type of objects and thus represent them in the most appropriate manner on the screen. The user can choose between different modes:

  • dump: with the environment variable set in this way, the terminal will output the JSON file received as a response without any type of parsing;
  • basic: in this case the terminal interprets the JSON response and prints out the various objects contained in the response in a compact and tabular format;
  • coloured: in this mode the terminal behaves as in the “basic” case but highlighting the various objects with the color set by the user for the command prompt;
  • extended: with this value the terminal prints out, in addition to the information provided from the “basic” case, also other data such as the data source, the producer, identifiers or objkeys associated with the various objects in the response received from the server;
  • extended_coloured: in this mode the terminal behaves as in the “extended” case but with the addition of the colors, as shown in the figure.
Extended coloured mode sample

The component has been developed so as to adapt also to the actual dimensions of the terminal from which the user submits the various commands. The viewer is in fact able to recognize the height and width of the shell in use, and print tables dynamically adapting to the space available on the screen. The terminal interprets the JSON file received in response to a request to the Ophidia server and, in the case of an object in a table format, is concerned to adjust the rows height to the content displayed and adjust the columns width in proportion to the maximum size of text placed in them.

Cube Provenance

Another environment variable, OPH_TERM_IMGS, instead allows you to create images from a JSON response containing objects such as trees or graphs. In particular, one can use the operator oph_cubeio to have a report of all the transformations performed on a set of data, with the detail of the produced datacubes at each step and the description of the particular operation performed.

In most cases, these are linear sequences of operations, starting from the insertion of new data into the system (import), passing through a series of operations of subsetting and data reduction, ending with the storing on disk of new files containing the analyzed data (export).

In other cases, however, the operations can create real strongly interlinked graphs. In these situations, the ability to view the entire session with an image is undoubtedly a better solution to the reading of numerous lines of plain text.

In the event that the terminal runs in a Desktop environment, then with a GUI and an image viewer, we can set the above variable to one of these three values:

  • no_op: to not create new images;
  • save: to create new images and save them to disk;
  • open: to create new images, save them to disk and automatically open them with the default image viewer on the system, as can be seen in the following figure.

Furthermore, in case of a command producing an image file representing a graph, OPH_GRAPH_LAYOUT allows you to setup the graph layout chosing among the following:

  • horizontal_compact: to arrange nodes from left to right;
  • vertical_compact: to arrange nodes from top to bottom;
  • horizontal_ranked: to arrange nodes from left to right and according to their rank/depth;
  • vertical_ranked: to arrange nodes from top to bottom and according to their rank/depth.
Cube provenance with image

Variable substitution

The terminal has also the ability to define new environment variables, so that users can possibly store values of interest that have not been provided with the default variables and to use these values into submission strings with a variable substitution very similar to that which is implemented, for example, in bash. Using $varname or ${varname}, Oph-Terminal will try to replace the value of the variable varname if present, or the empty string within the submission string. Without the parentheses, the first character other than a letter, number or underscore is treated as the end of the variable name. Variable substitution is recursive and runs until all $s have been replaced, so for example if the value of a variable is another $varname Oph-Terminal will recursively replace the last value found.

Aliases

In addition to the default commands, the terminal offers the user the ability to define new commands using aliases. Oph-Terminal incorporates the concept of aliases already present in bash as wrappers of other commands and extends it with the ability to set parametric aliases.

Here’s an example of use in which we define a new alias and then it is used with a parameter:

[OPH_TERM] >> setalias my_alias = "command param1=val1;param2=$1;"
[OPH_TERM] >> my_alias val2

We can specify only one alias per submission string. Oph-Terminal will find the first word of the submission string and in case this is an alias it would pass to the replacement of the arguments in the alias definition string in correspondence of $1, $2, etc.

Pre-loaded aliases are listed below.

cc=oph_createcontainer container=$1;dim=lat|lon|time;dim_type=double|double|double;hierarchy=oph_base|oph_base|oph_time;
cd=oph_folder command=cd;path=$1;
cdd=oph_fs command=cd;path=$1;
del=oph_delete cube=$1;
drop=del [*]
getprogress=oph_resume level=0;id=$1;
jobs=oph_loggingbk job_level=1;mask=010;session_filter=@OPH_SESSION_ID;parent_job_filter=%;
kill=oph_cancel id=$1
ll=oph_list path=$1;level=2;
lr=oph_list path=$1;level=2;recursive=yes;
ls=oph_list path=$1;
lsd=oph_fs command=ls;path=$1;
man=oph_man function=$1;
mkdir=oph_folder command=mkdir;path=$1;
mv=oph_movecontainer container=$1|$2;
mvdir=oph_folder command=mv;path=$1|$2;
ncdump=oph_explorenc src_path=$1;
new=oph_manage_session action=new;
pdd=getenv OPH_CDD
pwd=getenv OPH_CWD
rc=oph_randcube container=$1;dim=lat|lon|time;dim_size=$2|$3|$4;exp_ndim=2;host_partition=test;measure=example_measure;measure_type=double;nfrag=$2;ntuple=$3;compressed=no;concept_level=c|c|d;filesystem=local;ndb=1;ndbms=1;nhost=1;
rm=oph_deletecontainer container=$1;hidden=no;delete_type=physical;
rmdir=oph_folder command=rm;path=$1;
show=oph_explorecube show_time=yes;