Resource Management

A Chiminey platform supports access to computation and storage resources. A computation resource is where the core functionality of a smart connector is executed while a storage resource is used to retrieve input files and store output files. Prior to submitting a job, end-users need to register at least one computation and one storage resources. In this section, following topics are covered:

• Registering Compute Resources
• Registering Storage Resources
• Updating Resources
• Removing Resources

Registering Compute Resources

Various computing infrastructure and tools can be registered as compute resources. These resources are broadly categorised under cloud, high performance computing (HPC), analytics, and continuous integration resources.

Cloud Compute Resource

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Compute Resource from the Settings menu.
5. Click Register Compute Resource
6. Click the Cloud tab.
7. Select the resource type from the drop down menu. You may have access to more than one type of cloud service, e.g., NeCTAR and Amazon.
8. Enter a unique resource name.
9. Enter credentials such as EC2 access key and EC2 secret key
10. Click Register. The newly registered cloud-based compute resource will be displayed under Cloud - NeCTAR/CSRack/Amazon EC2.

Figure. Registering a cloud-based compute resource

HPC Compute Resource

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Compute Resource from the Settings menu.
5. Click Register Compute Resource
6. Click the HPC tab.
7. Enter a unique resource name.
8. Enter IP address or hostname of the HPC cluster head node or the standalone server.
9. Enter credentials, i.e. username and password. Password is not stored in the Chiminey platform. It is temporarily kept in memory to establish a private/public key authentication from the Chiminey platform to the resource.
10. Click Register. The newly registered resource will be displayed under HPC - Cluster or Standalone Server list.

Figure. Registering a HPC compute resource

Analytics Compute Resource

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Compute Resource from the Settings menu.
5. Click Register Compute Resource
6. Click the Analytics tab.
7. Select Hadoop MapReduce as the resource type from the drop down menu.
8. Enter a unique resource name.
9. Enter IP address or hostname of the Hadoop MapReduce resource.
10. Enter username and password. Password is not stored in the Chiminey platform. It is temporarily kept in memory to establish a private/public key authentication from the Chiminey platform to the resource.
11. Click Register. The newly registered resource will be displayed under Analytics - Hadoop MapReduce list.

Figure. Registering an analytics compute resource (Hadoop MapReduce)

Registering Storage Resources

Remote file systems and data curation services like Mytardis mytardis_storage are used as a storage resources.

Remote File System

1. Navigate to the Chiminey portal.
2. Log in with your credentials.
3. Click Settings.
4. Click Storage Resource from the Settings menu.
5. Click Register Storage Resource
6. Click the Remote File System tab.
7. Enter a unique resource name.
8. Enter IP address or hostname of the remote file system.
9. Enter credentials, i.e. username and password. Password is not stored in the Chiminey platform. It is temporarily kept in memory to establish a private/public key authentication from the Chiminey platform to the resource.
10. Click Register. The newly registered resource will be displayed under Remote File System list.

Figure. Registering a remote file system as a storage resource

MyTardis Storage Resource

1. Navigate to the Chiminey portal.
2. Log in with credentials
3. Click Settings
4. Click Storage Resource from the Settings menu
5. Click Register Storage Resource
6. Click the MyTardis tab.
7. Enter a unique resource name.
8. Enter IP address or hostname of the MyTardis instance.
9. Enter credentials, i.e. username and password. Username and password are stored on the Chiminey platform.
10. Click Register. The newly registered storage resource will be displayed under MyTardis list.

Figure. Registering MyTardis, a data curation service.

Updating Resources

Follow the steps below to change he details of registered resources.

1. Navigate to the Chiminey portal.
2. Log in with credentials
3. Click Settings
4. From the Settings menu, depending on which resource you wish to update, click either Compute Resource or Storage Resource. All registered resources will be listed.
5. Locate the resource you wish to update, then click Update.
6. Make the changes, and when finished click Update

Removing Resources

In order to remove a registered resource, follow all the steps from Updating Resources but click Remove instead of Update. The resource will be removed from the resources’ list.