API Reference

Chiminey Stage APIs

mytardis – MyTardis APIS

The MyTardis module provides functions that allow publishing Chiminey results to a connected MyTardis System, allowing the online storing, access and sharing capabilities of data and metadata.

Datastructures

paramset

Module Functions and Constants

Example:

def _get_exp_name_for_make(settings, url, path):
                   return str(os.sep.join(path.split(os.sep)[-2:-1]))

               def _get_dataset_name_for_make(settings, url, path):
                   return str(os.sep.join(path.split(os.sep)[-1:]))

               self.experiment_id = mytardis.create_dataset(
                   settings=mytardis_settings,
                   source_url=encoded_d_url,
                   exp_id=self.experiment_id,
                   exp_name=_get_exp_name_for_make,
                   dataset_name=_get_dataset_name_for_make,
                   experiment_paramset=[],
                   dataset_paramset=[
                       mytardis.create_paramset("remotemake/output", [])]
                   )

storage – Storage APIS

This package provides a file-like api for manipulating local and remote files and functions at locations specified by platform instances.

Datastructures

Module Functions and Constants

sshconnection – Manipulation of Remote Resources

Datastructures

ssh_client

Module Functions and Constants

compute – Execution Of Remote Commands

Using an open ssh_client connector from sshconnector, these commands execute commands remotely on the target server.

Datastructures

Module Functions and Constants

messages – Logging communication for Chiminey

This package, modelled off the django logging module, posts status messages for the BDP system. There are two classes of context for the user of this API:

  1. Within stage implementation, messages will be displayed within the status field in the job list UI.
  2. During job submission, messages will be displayed on the redirected page as a header banner.

Messages are processed by a separate high-priority queue in the celery configuration. Note that message delivery may be delayed due to celery priority or db exclusion on the appropriate context, so this function should not be used for real-time or urgent messages.

Datastructures

By convention, error messages are final messages in job execution to indicate fatal error (though job might be restarted via admin tool) and success is used to describe final successful execution of the job.

Module Functions and Constants

Send a msg at the required priority level, as per the django logging module.

Uses contextid field of settings to determine which running context to assign messages.

run_settings – Contextual Namespace for Chiminey

The current state of any BDP calculation is represented as a context model instance, with an associated run_settings instance. This run_setting serves as input, output and scratch symbol table for all job execution.

The contextual namespace is used for numerous purposes in the BDP, including:

  • Input parameters for Directive submission UI dialog
  • Single source of truth for building settings dicts for BDP API modules.
  • Job execution state
  • Diagnostics and visualisation of job progress
  • Stage triggering and scheduling during directive execution

Conceptually run_settings is a set of parameter sets each of which is conformant to a predefined schemas, that are defined in the admin tool.

run_settings is a two-level dictionary, internally serialised to models as needed.

Datastructures

context

A two level dictionary made up of schema keys and values. Conceptually, this structure is equivalent to a two-level python dictionary, but should be accessed via the API below. For example,

{ http://acme.edu.au/schemas/stages : { “key”: id, “id”: 3} }

Keys are concatenation of schema namespace and name, for example: http://acme.edu.au/schemas/stages/key

Module Functions and Constants

chiminey.runsettings.getval(context, key)

Retrieves the current value of the key within the run setttings context.

Parameters:
  • context – current context
  • key – the key to search
Raises:

SettingNotFoundException if the schema part of the key is not in the context

Returns:

the value for the key

chiminey.runsettings.setval(context, key, value)

Sets the value of the key within the run settings context.

Parameters:
  • context – current context
  • key – the key to search
Throws:

runsettings.SettingNotFoundException if the schema part of key is not in the context.

Throws:

runsettings.IncompatibleTypeException if the value is not compatible with the type of the key, according to the associated schema definition.

cloudconnection – Cloud Connection

Datastructures

Module Functions and Constants

corestages – Processing Steps in a directive

This is an abstract class that forms the interface for all directives, both smart connectors and utilties to provides steps in a calcuation.

Datastructures

Module Functions and Constants

simpleui – UI view members