The name Awoonga comes from Lake Awoonga, formed on the Boyne River in central Queensland.
Awoonga provides a batch scheduler environment and is pre-populated with a range of computational application software. The hardware and configuration of Awoonga is optimised for running multiple independent jobs, each using a small number of processors cores. It is ideally suited to large parameter sweep or ensemble applications.
Awoonga built on the following hardware:
- 1032 Intel cores across 43 compute nodes, each with 24 cores, 256GB memory and 300GB of disk
The cluster provides the following resources:
- 2 login nodes, with a load balancer.
- The open source PBS TORQUE batch system with Maui scheduler.
- 500TB of shared storage connected via GPFS and accessible across Awoonga, FlashLite, and Tinaroo clusters.
- Home directories with tight quotas and *no* backups. Backup is the responsibility of the user.
- Two temporary storage filesystems for staging data: /30days and /90days.
- The 500 GB of local scratch storage on each compute nodes as $TMPDIR.
Awoonga is available for use by any researcher who belongs to a QCIF member institution or partner organisation.
To access to Awoonga, you need to:
1. Create a QRIScloud account.
- Go to the QRIScloud portal (http://www.qriscloud.org.au)
- Click on the “Account” link
- Log in using your Australian Access Federation (AAF) credentials
- Accepting the Terms and Conditions
- Updating your user profile information (click on the “My Profile” link)
2. Once logged into the QRIScloud portal, you can then request a new service.
- Click on “My Services”
- Click on “Order new services”
- Click on “Register to use Awoonga”
- Complete the request form and submit the request.
3. Once your request has been processed you will need to generate your QRIScloud service access credentials (QSAC).
- Click on “My Credential”
- Click on the “Create credential” button
- You will be presented with the username and password that can be used to login to Awoonga. Please make a careful note of these.
You will be contacted by email when your account has been registered.
Registered users should connect to the Awoonga cluster by using Secure Shell (SSH) to connect to awoonga.qriscloud.org.au
If you are connecting from a Linux or Mac system, you can use the ssh command from a command shell. For example:
$ ssh <qsac-account>@awoonga.qriscloud.org.au
# enter your qsac password when prompted
On Windows, you can use the third-party PuTTY tool to SSH to Awoonga.
When you log in to Awoonga, you will find yourself logged in on one of three identical login nodes (awoonga1, awoonga2, or awoonga3). We recommend that you connect to the hostname awoonga.qriscloud.org.au to ensure you get the least loaded available login node. The awoonga hostname provides a load balancer for the three login nodes.
Major file systems
Your home directory on Awoonga is /home/$USER. It is created automatically when you login for the first time. Your home directory can be accessed on the Awoonga login nodes and the Awoonga compute nodes. The purpose of your home directory is to hold software that you have brought to the system, batch scripts and configuration files, and a relatively small amount of data.
The /home has quota control of storage and numbers of files (refer to default filesystem quota settings below).
Important Note: The /home file system is NOT backed up. If files are accidentally deleted we are unable to restore them. It is your responsibility to backup files located in your home directory. We advise you to regularly transfer any valuable files from your home directory to some other system that is backed up.
Each Awoonga user is allocated a directory on the /30days file system. It is present on the Awoonga login nodes and the Awoonga compute nodes. The main purpose is to hold large data sets on a temporary basis while you are computing against them. It is designed to be a data staging area.
Users have a quota of 1 TB (1000 GB) on the /30days file system. This file system is NOT backed up. Furthermore, files left on this file system are automatically deleted 30 days after they were created.
Each Awoonga user is allocated a directory on the /90days file system. It is present on the Awoonga login nodes and the Awoonga compute nodes. The main purpose is to hold moderately large data sets on a temporary basis while you are computing against them. It is designed to be a data staging area.
Users have a quota of 400GB on the /90days file system. This file system is NOT backed up. Furthermore, files left on this file system are automatically deleted 90 days after they were created.
The /sw file system contains all of the currently available software modules that can be used on Awoonga. It is present on the Awoonga login nodes and the Awoonga compute nodes, and is read-only for normal users.
The RDS collections are available to use on Awoonga via network filesystem mounts. These mounts are permanently connected with only members of the collection project team able to access their collection data. Quotas and other limits imposed on the collection will also apply when collections are accessed from Awoonga.
Quota settings on file systems
|/home||20||204,800||Indefinite but no backup|
|/30days||1000||3,145,728||Files deleted 30 days from creation|
|/90days||400||1,048,576||Files deleted 90 days from creation|
Filesystem dos and don'ts
Use $TMPDIR when you need local disk for your batch jobs. The $TMPDIR directory is created automatically as part of your batch job and is removed for you automatically at the end of the job.
Saving user data randomly into local disk on a node (outside of $TMPDIR) can adversely impact other users. Please DON'T do that.
Compute Nodes are periodically rebuilt and the local disk space is reformatted, so do not rely on using local disk on compute nodes unless via $TMPDIR.
Don't forget that $TMPDIR is unique for each job and job-array sub-job.
Although the path may be the same, the $TMPDIR directory will probably contain different files on different nodes and for different jobs.
If you need to work with many small files, please keep them bound together in a single archive file (ZIP or tar) and copy the archive file to local disk (i.e. $TMPDIR) before unpacking them to work on them in local disk space.
Further information about storage is provided in storage user guide (via rcc.uq.edu.au)
The batch system
The Awoonga cluster uses PBSPro, with detailed information availabe in the PBSPro User Guide.
We account for all usage on the Awoonga cluster to satisfy our stakeholders that their entitlements are being met, and to assist with planning and resource management.
The Awoonga account groups correspond either to organisational groups or to projects that span multiple organisations.
• For UQ, the account groups are broken down to the level of a School or Centre. The group qriscloud-uq will not work as an account group for job submission.
• For other organisations, the account group will be the appropriate qriscloud-xxx group for the organisation.
If you are a member of multiple accounting groups, it is important that you chose the most appropriate group when submitting jobs.
Other Accounting Tips
- Changing your Linux default group (using the newgrp command) at the command line does not affect accounting. Make sure you use the -A option with the appropriate account group within your submitted batch job.
Awoonga is a ROCKS cluster.
- A lot of software lives on the local disk of each compute node and has been deployed as part of the cluster imaging mechanism.
- The command rocks list roll will summarise the deployed rolls. Some rolls (e.g. biotools) contain many individual applications.
- Some software and software development tools are located on teh shared storage point under /sw which is available on all nodes.
- Project and discipline areas are encouraged to compile and maintain their own software if currency is an issue.
Installed software and software development tools can be found in the /sw file system available on all nodes.
The current list of available software modules is displayed by running the command:
$ module avail
Not all modules show up when you run the module avail command. Modules that depend on compiler modules are hidden from view until the compiler module has been loaded.
In addition to user training and the associated training materials, there are a number of other ways to get help:
- Documentation on standard Linux commands, the PBS job submission (qsub) and management commands and many other things can be viewed using the Linux man command
- User documentation for most of the applications and tools available via the modules mechanism.
- Do NOT post any commercial software documentation that you find in /sw on a website, or forward copies to anyone else
- The system message of the day will occasionally carry information about forthcoming work and other outages.
In order to use the Awoonga cluster, you will typically need a way to connect to it from a laptop or desktop system. You may also need tools for transferring files too and from the cluster, and possibly other things. There are numerous free tools available for these tasks.
To login to Awoonga, you will need a tool that is capable of running an interactive SSH session. The tools you can use include:
- For Microsoft Windows platforms: the third-party PuTTY or WinSCP tools are available.
- For Mac OSX and Linux: the ssh command is included preinstalled or from your platform's package installer.
To transfer files to and from Awoonga, you will need to use an SSH-based file transfer method such as SCP or SFTP. The tools you can use for this include:
- Cross-platform: CyberDuck (GUI) or FileZilla.
- For Microsoft Windows platforms: WinSCP.
- For Mac OSX: rbrowser, fugu, or the scp and sftp commands.
- For Linux: distribution specific browsers, and the scp and ftp commands.
If you need to run an interactive application on Awoonga with a GUI, then you will need to run an X11 server on your laptop or desktop that the application can connect to:
- For Microsoft Windows: Xming is a good (i.e. free) option.
- For Mac OSX: X11 is available in the Utilities folder.
- For Linux: if you have a “desktop” install (e.g. Gnome, KDE, etcetera) your system will already be running an X11 server.