lico 6.0.0 administrator guidedownload.lenovo.com/servers_pdf/lico_6.0.0... · change the account...

LiCO 6.0.0 Administrator Guide

Seventh Edition (August 2020)

© Copyright Lenovo 2018, 2020.

LIMITED AND RESTRICTED RIGHTS NOTICE: If data or software is delivered pursuant to a General Services Administration (GSA) contract, use, reproduction, or disclosure is subject to restrictions set forth in Contract No. GS-35F- 05925.

Contents

Chapter 1. Overview. . . . . . . . . . . 1Introduction to LiCO . . . . . . . . . . . . . . 1Features of LiCO . . . . . . . . . . . . . . . 1Terminology . . . . . . . . . . . . . . . . . 1Prerequisite . . . . . . . . . . . . . . . . . 2Operating environment . . . . . . . . . . . . . 2

Chapter 2. Administrator home page . . . . . . . . . . . . . . . . . . . 3Log in to the administrator home page . . . . . . . 4Switch roles . . . . . . . . . . . . . . . . . 4Get current version information. . . . . . . . . . 4Change the account password . . . . . . . . . . 4Cluster status view . . . . . . . . . . . . . . 5Cluster alert messages . . . . . . . . . . . . . 5

Chapter 3. User . . . . . . . . . . . . . 7User groups (using LDAP) . . . . . . . . . . . . 7

Create a user group . . . . . . . . . . . . 7Delete a user group . . . . . . . . . . . . 7

User groups (not using LDAP) . . . . . . . . . . 8Users (using LDAP) . . . . . . . . . . . . . . 8

Create a user . . . . . . . . . . . . . . . 8Import users in batch . . . . . . . . . . . . 9Export users . . . . . . . . . . . . . . . 9Edit a user . . . . . . . . . . . . . . . . 9Change the user password . . . . . . . . . 9Delete a user . . . . . . . . . . . . . . . 9Suspend a user . . . . . . . . . . . . . 10Resume a user . . . . . . . . . . . . . 10Browse user details . . . . . . . . . . . 10

Users (not using LDAP) . . . . . . . . . . . . 10Import a user . . . . . . . . . . . . . . 10Import users in batch . . . . . . . . . . . 11Export users . . . . . . . . . . . . . . 11Edit a user . . . . . . . . . . . . . . . 11Change the user password . . . . . . . . 11Delete a user . . . . . . . . . . . . . . 12Suspend a user . . . . . . . . . . . . . 12Resume a user . . . . . . . . . . . . . 12Browse user details . . . . . . . . . . . 12

Billing groups. . . . . . . . . . . . . . . . 12Create a billing group . . . . . . . . . . . 12Edit a billing group and its queue policies or storage policies . . . . . . . . . . . . . 13Delete a billing group . . . . . . . . . . . 14Copy a billing group . . . . . . . . . . . 15

Deposit/Withdraw money for a billing group . . 15Discount . . . . . . . . . . . . . . . . . 15

Set a user discount. . . . . . . . . . . . 15Edit a user discount . . . . . . . . . . . 16Delete a user discount . . . . . . . . . . 16Set a user group discount . . . . . . . . . 16Edit a user group discount . . . . . . . . . 16Delete a user group discount . . . . . . . . 16

Solutions to some user action failures . . . . . . 16

Chapter 4. Monitor . . . . . . . . . . 19List view. . . . . . . . . . . . . . . . . . 19Physical view . . . . . . . . . . . . . . . . 20Group view . . . . . . . . . . . . . . . . 21GPU view . . . . . . . . . . . . . . . . . 21Jobs . . . . . . . . . . . . . . . . . . . 21Alerts . . . . . . . . . . . . . . . . . . . 22Operations. . . . . . . . . . . . . . . . . 23

Chapter 5. Reports . . . . . . . . . . 25Job reports . . . . . . . . . . . . . . . . 25Expense reports . . . . . . . . . . . . . . 25Cluster reports . . . . . . . . . . . . . . . 26Alert reports . . . . . . . . . . . . . . . . 28Utilization reports . . . . . . . . . . . . . . 28

Chapter 6. Admin . . . . . . . . . . . 31Container images . . . . . . . . . . . . . . 31

Create a container image . . . . . . . . . 31Edit a container image . . . . . . . . . . 32View a container image . . . . . . . . . . 32Delete a container image . . . . . . . . . 32Download a container image . . . . . . . . 32Reupload a container image . . . . . . . . 32

Bills . . . . . . . . . . . . . . . . . . . 32VNC . . . . . . . . . . . . . . . . . . . 33Operation logs . . . . . . . . . . . . . . . 34Web logs . . . . . . . . . . . . . . . . . 34

Chapter 7. Settings . . . . . . . . . . 37Scheduler . . . . . . . . . . . . . . . . . 37

Create a queue . . . . . . . . . . . . . 37Edit a queue . . . . . . . . . . . . . . 38Delete a queue . . . . . . . . . . . . . 38Set the state of a queue . . . . . . . . . . 38Set the node state . . . . . . . . . . . . 38

Alert policy. . . . . . . . . . . . . . . . . 38

© Copyright Lenovo 2018, 2020 i

Create an alert policy . . . . . . . . . . . 39Edit an alert policy . . . . . . . . . . . . 39Delete an alert policy . . . . . . . . . . . 39

Notification group . . . . . . . . . . . . . . 39Create a notification group. . . . . . . . . 40Edit a notification group . . . . . . . . . . 40Delete a notification group . . . . . . . . . 40

Notification . . . . . . . . . . . . . . . . 40Scripts . . . . . . . . . . . . . . . . . . 41

Chapter 8. License . . . . . . . . . . 43Import a license . . . . . . . . . . . . . . . 43Export a license . . . . . . . . . . . . . . . 43Delete a license . . . . . . . . . . . . . . . 43

Chapter 9. HPC cluster management . . . . . . . . . . . . . . 45View HPC cluster details . . . . . . . . . . . 45Remote management of HPC cluster software . . . 45

Interface management . . . . . . . . . . 45

Command line management . . . . . . . . 45xCAT management. . . . . . . . . . . . 45

Parallel commands . . . . . . . . . . . . . 47Job scheduling commands . . . . . . . . . . 47Queue commands . . . . . . . . . . . . . . 48Job management . . . . . . . . . . . . . . 48

Appendix A. Important information . . . . . . . . . . . . . . . 51Restart LiCO . . . . . . . . . . . . . . . . 51Resolve a failed job submission . . . . . . . . 51Batch deletion of jobs in the database . . . . . . 52Failure to view or delete a VNC . . . . . . . . . 52Data sources for GPU monitoring . . . . . . . . 52Clean up jobs submitted through accounts not managed by LiCO . . . . . . . . . . . . . . 52Scheduler troubleshooting . . . . . . . . . . 52MPI issues troubleshooting . . . . . . . . . . 53Different memory usage on LiCO and Linux OS . . 53Notices and trademarks . . . . . . . . . . . 54

ii LiCO 6.0.0 Administrator Guide

Chapter 1. Overview

Introduction to LiCOLenovo Intelligent Computing Orchestration (LiCO) is an infrastructure management software for high- performance computing (HPC) and artificial intelligence (AI). It provides features like cluster management and monitoring, job scheduling and management, cluster user management, account management, and file system management.

With LiCO, administrators can manage and monitor user groups, users, and billing groups within a cluster in a centralized manner. They can also schedule and monitor jobs, alerts, and reports, and perform various settings.

Features of LiCO• Cluster resource monitoring: LiCO provides a dashboard to monitor the usage of cluster resources,

including CPU, memory, storage, and network.

• Job template storage: LiCO provides multiple job templates, including HPC and AI job templates, which help users submit jobs from Web pages with convenience.

• Customized templates: Users can create their own job templates to support other HPC and AI applications.

• Job management and monitoring: Users can directly view and manage the status and results of jobs. Various common schedulers and a wide range of job types are supported (including AI jobs such as TensorFlow and Caffe).

• E2E training: Users can train image classification models without coding. LiCO also provides E2E support for training, such as dataset management, topology management, and pre-trained model management.

• User management and billing: LiCO manages both local and domain users through the same interface. It supports user top-ups and chargebacks, and offers the ability to set billing groups and fees.

• Customizations: A range of customizations are available, such as enterprise job template customization, report customization, and 3D server visualization.

• Container image management: LiCO provides system container images for every supported AI framework. Users can upload private container images and run AI or HPC jobs on them.

• Expert Mode: LiCO provides command line tools to submit and manage jobs. Expert users can log in to the login node via another shell and execute commands.

Terminology• Computer cluster: a general reference to a collection of server resources including management nodes,

login nodes, and computing nodes

• Job: a series of commands in sequence intended to accomplish a particular task

• Job status: the status of a job in the scheduling system, such as waiting, in queue, on hold, running, suspended, or completed

• Node status: the status of a node, such as idle, busy, or off

• Job scheduling system: the distributed program in control of receiving, distributing, executing and registering jobs, also referred to as the operation scheduler or simply scheduler

• Management node: the server in a cluster running management programs such as job scheduling, cluster management and user billing

© Copyright Lenovo 2018, 2020 1

• Login node: the server in a cluster to which users can log in via Linux and conduct operations

• Computing node: the server in a cluster for executing jobs

• User group: a set of users for which the system has defined an access control policy, so that all users in the same user group have access to the same set of cluster resources

• Billing group: a group of cluster users that are to be billed under one account, also referred to as a billing account. A billing account can be made up of a single user or multiple users.

• NGC Image: NVIDIA Container Runtime for Docker, also known as nvidia-docker. It supports GPU-based applications that are portable across multiple machines. This is achieved through the use of Docker containers.

PrerequisiteLiCO currently supports Slurm as the scheduler. The commands for Slurm in this Guide are not applicable to other schedulers.

Operating environmentCluster server:

Lenovo ThinkSystem servers

Operating system

• CentOS / Red Hat Enterprise Linux (RHEL) 8.1

• SUSE Linux Enterprise server (SLES) 15 SP1

Client requirements:

• Hardware: CPU of 2.0 GHz or above, memory of 8 GB or above

• Browser: Chrome (V 62.0 or higher) or Firefox (V 56.0 or higher) recommended

• Display resolution: 1280 x 800 or above

2 LiCO 6.0.0 Administrator Guide

Chapter 2. Administrator home page

A user can assume three kinds of roles: administrator, operator, and ordinary user. Administrators can view the entire computer cluster and the information of all users. Operators can only view resources to which they have access, as well as their own information. Ordinary users can execute jobs and run operations such as job monitoring.

With the correct administrator username and password, you can open the administrator home page.

The left-hand side navigation pane shows the following elements:

• HPC.com: Cluster name. When the mouse hovers over it, the current scheduling and file service state is shown. You can refer to the LiCO Installation Guide to edit the cluster name.

• Home: the current page that shows basic cluster information

• User: allows the administrator to perform basic operations on users, user groups, and billing groups.

• Monitor: provides functions for monitoring the HPC cluster.

• Reports: allows the administrator to export reports in Excel, PDF, or HTML format based on the job, alert, or action type

• Admin: allows the administrator to check the VNC, operation logs, and Web logs

• Settings: allows the administrator to perform actions on the queues, configure alerts for the HPC cluster, and manage notification groups and notification settings

• License: allows the administrator to import, export, or delete a license. If no license is imported, only the License tab is displayed on the navigation pane, and all the tabs above are unavailable.

The upper-right corner shows the following icons:

• : shows the number of unconfirmed alerts in the current cluster. You can click this icon to enter the alert details page or to turn alert sounds on or off.

• : You can click this icon to check current user information, edit current user password, log out, or switch between user roles.


Log in to the administrator home pageStep 1. Open a browser.

Step 2. Type the IP address for the cluster’s login node, such as https://10.220.112.21.

Step 3. Type the correct administrator username and password.

Step 4. Click Log in.

The administrator home page is displayed.

Switch rolesWith the highest permission level in the system, an administrator can switch to the role of an operator or user and be redirected to the corresponding home page.

Step 1. Click in the upper-right corner.

Step 2. Click the role you want to switch to.

Click Operator to switch to the operator role.

Click User to switch to the user role.

The operator home page or user home page is displayed.

Get current version information

Step 1. Place your cursor over in the upper-right corner of the administrator home page.

Step 2. Click .

Step 3. Click a menu item to get required information.

Click User Agreement to get the user agreement.

Click Third party licenses to obtain the third party licenses.

Change the account password

Step 1. Place your cursor over in the upper-right corner of the administrator home page.

Step 2. Click .

Step 3. Type the current password, and then type the new password twice.

Step 4. Click OK.

Your password is changed.


Cluster status view

The administrator home page shows the basic status of the entire cluster. You can click to maximize or minimize the navigation pane.

The cluster overview page consists of the following elements:

Element Description

CPU Utilization rate of the CPUs in a server cluster, indicated by the percentage of CPU cores in use among the total CPU cores in a cluster

GPU Utilization rate of the GPUs in a server cluster, indicated by the percentage of GPU cores in use among the total GPU cores in a cluster

Memory Utilization of memory in the cluster, together with the amount of memory used and the total size of the memory in the cluster

Storage Utilization of storage in the cluster, together with the storage used and the total storage in the cluster

Network Capacity of the network on a server cluster, including reading and writing speeds

Nodes Number of computers turned on or off in the computer cluster

Node Status Usage status of nodes on the computer cluster, including busy, idle, and off. The primary basis for determining node usage is that there is one or more jobs running on that node.

Jobs Names and running times of jobs that are running or waiting.

Job Status Past information about the job, including the numbers of jobs running, waiting, and finished. An administrator can choose to display the number of jobs in all queues or the number of jobs in a certain queue. In terms of time, available display options include the last hour, the last day, the last seven days, and the last thirty days. In terms of job type, available display options include unfinished and finished jobs.

Messages Show the most recent operations log for the web system. When you place the cursor over the cluster name in the left navigation pane, the current scheduling and file service status is displayed.

The health of the scheduler and parallel system is indicated by the following color-coded system:

Scheduler: Green means that the scheduler is working normally; red means that the scheduler is not working normally.

Parallel File System: Green means that the parallel file system is working normally; red means that the parallel file system is not working normally.

Cluster alert messages

If alerts are triggered, the symbol in the upper-right corner of the home page produces a red numerical prompt, showing the current number of unconfirmed alerts.

You can click this icon to view details of unconfirmed alerts. To access the alert information page, you can also select Monitor ➙ Alerts from the left navigation pane.

Chapter 2. Administrator home page 5

Chapter 3. User

There are three user constructs: user group, user, and billing group (or billing account), as explained in the table below.

User Construct Description

User group A group of users on the HPC cluster with similar queue access permissions.

User List of users in the HPC cluster.

Billing group The billing account number, which can be used by one or multiple users. When members of a billing group run applications in a cluster, the balance in the billing account will be debited according to the number of CPU cores used and the time taken in running the applications.

Discount Discount in billing for a user or user group.

User groups (using LDAP)Select User ➙ User Groups from the left navigation pane to enter the user group management page.

Create a user groupIt is recommended that the administrator create a new user group.

Step 1. On the user group management page, click Create.

Step 2. Type a unique name for the user group.

Step 3. Click OK.

A new user group is created.

Delete a user groupAdministrators can delete existing user groups.

Step 1. On the user group management page, find the user group you want to delete, and then select Action ➙ Delete.

Step 2. Click OK.

The user group is deleted from the system.


Note: Before deleting the user group, you should move the users to a new group. Otherwise, the users cannot submit jobs.

User groups (not using LDAP)Select User ➙ User Groups from the left navigation pane to enter the user group management page.

Note: An administrator cannot create or delete a user group.

Users (using LDAP)Select User ➙ Users from the left navigation pane to enter the user management page.

All actions in this section are performed on this page.

Create a userStep 1. On the user management page, click Create.

Step 2. Fill in the information in the dialog box.

• Username must only contain lowercase letters, numbers, underscores, and minus signs. It must start with a letter.

• Password should consist of at least 10 characters, including at least one uppercase letter, one lowercase letter, one special symbol, and one number.

Step 3. Click OK.


A user account is created and can be used for login.

Import users in batchStep 1. Click Batch Import on the user management page.

Step 2. Click Browse, and then select the file that contains the information about the users you want to import. You can click Sample File to help yourself prepare the user information file. If gibberish characters are displayed in the imported file, change the coding mode to UTF-8 for the file and then try again.

Step 3. Click Import.

Users in the file are imported into the system.

When the user accounts are successfully imported, the users can log in.

Export usersStep 1. Click Export on the user management page.

Your browser shows a dialog, asking your desired actions with the user information file in CSV format.

Step 2. Specify a location and save the CSV file.

Information about all users on the page is exported and saved in the CSV file. Currently, exported information includes the following: username, role, first_name, last_name, email, is_activate

Edit a userAn administrator can change user information such as the role, user group, billing group, and e-mail address.

Step 1. Find the user you want to edit on the user management page, and then select Action ➙ Edit.

Step 2. Change the information in the dialog box.

Step 3. Click OK.

Change the user passwordAn administrator can change passwords for operators or ordinary users, but not those of other administrators.

Step 1. Find the user for which you want to change the password on the user management page, and then select Action ➙ Change Password.

Step 2. Type and confirm the new password.

Step 3. Click OK.

The user password is changed.

Delete a userAn administrator can delete existing users.

Step 1. Find the user you want to delete, and then select Action ➙ Delete.

Step 2. Click OK.

Chapter 3. User 9

The user is deleted.

Suspend a userAn administrator can suspend the accounts of operators or ordinary users, but not those of other administrators.

Step 1. Find the user account you want to suspend, and then select Action ➙ Suspend.

Step 2. Type the suspension time as needed.

Step 3. Click OK.

The user account is suspended for the time you specified.

Resume a userAn administrator can resume suspended operators or ordinary users.

Step 1. Find the suspended user you want to resume, and then select Action ➙ Resume.

Step 2. Click OK.

The suspended user is resumed.

Browse user detailsFind the user whose details you want to browse, and then select Action ➙ Detail.

The user details page is displayed.

Users (not using LDAP)You can select User ➙ Users from the left navigation pane to enter the user management page.


Import a userStep 1. Select User ➙ Users from the left navigation pane.



Step 3. Fill in the information as required.

Step 4. Click OK.

The system imports the user account.

When the user account is successfully imported, the user can log in.

Import users in batchStep 1. Click Batch Import on the user management page.

Step 2. Click Browse, and then select the file that contains the information about the users you want to import. You can click Sample File to help yourself prepare the user information file. If gibberish characters are displayed in the imported file, change the coding mode to UTF-8 for the file and then try again.


Users in the file are imported into the system.

When the user accounts are successfully imported, the users can log in.

Export usersStep 1. Click Export on the user management page.

Your browser shows a dialog, asking your desired actions with the user information file in CSV format.

Step 2. Specify a location and save the CSV file.

Information about all users on the page is exported and saved in the CSV file. Currently, exported information includes the following: username, role, first_name, last_name, email, is_activate

Edit a userAn administrator can change user information such as the role, user group, billing group, and e-mail address.

Step 1. Find the user you want to edit on the user management page, and then select Action ➙ Edit.

Step 2. Change the information in the dialog box.

Step 3. Click OK.

Change the user passwordAn administrator can change passwords for operators or ordinary users, but not those of other administrators.

Step 1. Find the user for which you want to change the password on the user management page, and then select Action ➙ Change Password.

Step 2. Type and confirm the new password.

Step 3. Click OK.

The user password is changed.

Chapter 3. User 11

Delete a userAn administrator can delete existing users.

Step 1. Find the user you want to delete, and then select Action ➙ Delete.

Step 2. Click OK.

The user is deleted.

Suspend a userAn administrator can suspend the accounts of operators or ordinary users, but not those of other administrators.

Step 1. Find the user account you want to suspend, and then select Action ➙ Suspend.

Step 2. Type the suspension time as needed.

Step 3. Click OK.

The user account is suspended for the time you specified.

Resume a userAn administrator can resume suspended operators or ordinary users.

Step 1. Find the suspended user you want to resume, and then select Action ➙ Resume.

Step 2. Click OK.

The suspended user is resumed.

Browse user detailsFind the user whose details you want to browse, and then select Action ➙ Detail.

The user details page is displayed.

Billing groupsLiCO allows administrators to manage user billing groups in an easy and consolidated manner.

Note: Users imported from the background are not associated with any billing groups. To associate a user with a billing group, edit the user information on the Users tab.

Create a billing groupStep 1. Choose User ➙ Billing Groups from the left navigation pane.


Step 2. Click Create.

Step 3. Fill in the required information.

• Name: name of the billing group, which must be unique.

• CPU Billing Rate: fee for using one CPU core for unit time. The billing rate can be set by hour or minute, and the actual time for using CPU cores is measured in seconds.

• Memory Billing Rate: fee for using 1 MB memory for unit time. Billing for memory usage is based on the maximum memory used. The billing rate can be set by hour or minute, and the actual time for using memory is measured in seconds.

• Generic Resource Billing Rate: billing rate for configured generic resources such as GPU.

• Initial Amount: the amount in the account when the billing group was created.

• Description: description of the billing group.

Step 4. Click OK.

Edit a billing group and its queue policies or storage policiesYou can edit a billing group and its queue policies or storage polices on its details page, where you can perform the following operations:

• “Edit a billing group” on page 13

• “Create a queue policy” on page 13

• “Update a queue policy” on page 14

• “Delete a queue policy” on page 14

• “Create a storage policy” on page 14

• “Update a storage policy” on page 14

• “Delete a storage policy” on page 14

Access the details pages of a billing groupThe details page of a billing group enables you to create, update, or delete queue policies or storage policies for the billing group.

Step 1. Choose User ➙ Billing Groups from the left navigation pane.

Step 2. Click the name of the billing group for which you want to view its details. The billing group details page is displayed, on which you can create, update, or delete queue policies or storage policies.

Edit a billing groupStep 1. On the billing group details page, click Edit in the Default Policy section.

Step 2. Edit the parameters in this dialog as required.

Step 3. Click OK.

Create a queue policyYou can create a queue policy on the details page of a billing group. If a queue policy is created, costs incurred by users in this billing group for using the queue will be calculated based on the configured queue policy.

Step 1. On the billing group details page, click Create in the Queue Policy section.


Chapter 3. User 13

Queues: You can click Add to add multiple queues.

For the description of other descriptions, see “Create a billing group” on page 12.

Step 3. Click OK.

Update a queue policyYou can update a queue policy on the details page of a billing group.

Step 1. On the billing group details page, find the queue policy you want to update in the Queue Policy section and then select Action ➙ Edit.

Step 2. Edit the parameters as required.

Step 3. Click OK.

Delete a queue policyYou can delete a queue policy on the details page of a billing group.

Step 1. On the billing group details page, find the queue policy you want to delete in the Queue Policy section and then select Action ➙ Delete.

Step 2. Click OK.

Create a storage policyCurrently, LiCO supports billing on the distributed general parallel file system (GPFS) by using storage policies. You can create a storage policy on the details page of a billing group.

Step 1. On the billing group details page, click Create in the Storage Policy section.


• Storage: name of a file system for which you will be billed. You can click Add to add multiple file systems.

• Storage Billing Rate: fee for using 1 GB storage per day.

Step 3. Click OK.

Update a storage policyYou can update a storage policy on the details page of a billing group.

Step 1. On the billing group details page, find the storage policy you want to update in the Storage Policy section and then select Action ➙ Edit.

Step 2. Edit the parameters as required.

Step 3. Click OK.

Delete a storage policyYou can delete a storage policy on the details page of a billing group.

Step 1. On the billing group details page, find the storage policy you want to delete in the Storage Policy section and then select Action ➙ Delete.

Step 2. Click OK.

Delete a billing groupStep 1. Select User ➙ Billing Groups from the left navigation pane.


Step 2. Find the billing group you want to delete, and then select Action ➙ Delete.

Step 3. Click OK.

Copy a billing groupStep 1. Select User ➙ Billing Groups from the left navigation pane.

Step 2. Find the billing group you want to copy, and then select Action ➙ Copy.


• Name: name of the billing group, which must be unique.

• Initial Amount: the amount in the account of the newly copied billing group.

Step 4. Click OK.

Deposit/Withdraw money for a billing groupStep 1. Choose User ➙ Billing Groups from the left navigation pane.

Step 2. Find the billing group for which you want to deposit or withdraw money, and then select Action ➙ Account Actions.

Step 3. Select Deposit or Withdraw from the Action drop-down list, and fill in the amount.

Step 4. Click OK.

DiscountThe discount function of LiCO enables administrators to set discounts for users and user groups. If discounts are set both for a specific user and the user group to which he or she belongs, the discount set for the user prevails.

Set a user discountStep 1. Choose User ➙ Discount from the left navigation pane.

Step 2. Click Set Discount in the upper left corner of the left pane.


• Discount: discount factor. Final cost = Total Cost x Discount. Discount can be set to a value greater than 1, which means that a larger cost than the actual cost is required.

• User: You can click Add to add multiple users.

Step 4. Click OK.

Chapter 3. User 15

Edit a user discountLiCO supports modification of configured users or discount factor for an existing user discount.

Step 1. Choose User ➙ Discount from the left navigation pane.

Step 2. Find the user discount you want to edit, and then perform either of the following operations:

a. Select Action ➙ Edit User if you want to update the user information.

b. Select Action ➙ Edit Discount if you want to update the discount factor.

Step 3. Edit the information as required.

Step 4. Click OK.

Delete a user discountStep 1. Choose User ➙ Discount from the left navigation pane.

Step 2. Find the discount you want to delete, and then select Action ➙ Delete.

Step 3. Click OK.

Set a user group discountStep 1. Choose User ➙ Discount from the left navigation pane.

Step 2. Click Set Discount in the upper left corner of the right pane.


• Discount: discount factor, which must be a number between 0 and 1. Final cost = Total Cost x Discount.

• User Group: You can click Add to add multiple user groups.

Step 4. Click OK.

Edit a user group discountLiCO supports modification of configured user groups or discount factor for an existing user group discount.

Step 1. Choose User ➙ Discount from the left navigation pane.

Step 2. Find the user group discount you want to edit, and then perform either of the following operations:

a. Select Action ➙ Edit User Group if you want to update the user group information.

b. Select Action ➙ Edit Discount if you want to update the discount factor.

Step 3. Edit the information as required.

Step 4. Click OK.

Delete a user group discountStep 1. Choose User ➙ Discount from the left navigation pane.

Step 2. Find the discount you want to delete, and then select Action ➙ Delete.

Step 3. Click OK.

Solutions to some user action failuresIn the following circumstances, some user actions may fail:

• Network problems exist on the server nodes of a cluster.


• User groups or user accounts with identical names have been created in the operating system on the server nodes of a cluster.

• There are inconsistencies in user group or user account information in the operating system on the server nodes of a cluster.

• Slurm is not running properly.

Based on the failures above, use the following solutions:

1. Make sure the network connection is good, and root accounts can be accessed without a password across all server nodes.

2. Delete all failed user groups, billing groups, and user accounts and re-create these accounts.

3. Contact Lenovo professional services for technical support.

Chapter 3. User 17

Chapter 4. Monitor

Select Monitor from the left navigation pane. The following sub-options are displayed:

• List View: shows detailed information about all nodes in a cluster, and allows the user to perform corresponding actions on nodes in that cluster

• Physical View: shows detailed node information based on the physical locations of all machines in the cluster

• Group View: shows detailed information about group nodes based on the functions of all nodes in the cluster

• GPU View: shows the monitoring information about every GPU based on the functions of all group nodes in the cluster

• Jobs: shows the running status of jobs currently on the cluster

• Alerts: shows the details of triggered alerts and manages the status of these alerts

• Operations: shows the logs of all action changes

List viewIn the List View page, information for all nodes in a cluster is displayed in a list, as shown below:

The parameters on the List View page are described as follows:

• Host Name: indicates the host name for the node.

• Status: indicates the status of the node, which can be Idle, Busy, or Off.

• Health: indicates the health condition of the node, which can be Normal, Warning, Critical, Failed, or Unknown. The health information comes from the BMC hardware health conditions. You can click the host name for more details on the Details page.

• Power: indicates whether the power is on or off.

• Type: indicate the type of the node, which can be compute, head, login, I/O, and other user-defined nodes.

• BMC IP: indicates the IP address of the head module BMC.

• OS IP: indicates the address of the node.

• Hardware: indicates the number of CPU cores on every node / the total memory on every node / the number of GPUs on every node (if there is no GPU, then the GPU section will not appear).

• Custom Group: indicates the group to which the node currently belongs.

• Action: The actions include Remote Console, OS SSH, Boot, Boot to Setup, Boot to Network, and Boot to Media.


Physical viewThe Physical View page shows server room information, including room name and location, number of nodes, and total power consumption, and presents a graphic view of the number of racks and the locations of nodes, as shown below.

By clicking on a rack, the user can view detailed information about the rack, including:

• Rack name

• Rack location (related to the server room)

• Total number of nodes on a rack

• Total power consumption of a rack

By clicking on a node in a rack, the user can view detailed information on the selected node.


By clicking on the icons above a rack, the user can switch between displays of temperature, power consumption, CPU/load, memory utilization ratios, hard drive utilization ratios, network throughput, and jobs.

Group viewIn the Group View page, the information about all nodes in a cluster is sorted by logical grouping. Click on the “Select Group” drop-down box in the top-left corner and select a group to be displayed, as shown below.

The Group View page offers the following monitoring types:

• List: a list of all nodes in this group, which has similar functionality as “List view” on page 19

• Trend: shows the trend diagram for the group, including load, CPU, memory, hard drive, network, energy consumption, temperature, and job use

• Heatmap: shows a heat diagram including load, CPU, memory, hard drive, network, energy consumption, temperature, and job use for all nodes in the group

GPU viewThe GPU View page shows GPU information about the nodes in a group based on the logical grouping of all nodes in a cluster. Click the group option in the upper-left corner of GPU View, as shown below.

This interface presents real-time GPU data for the group in graphic form and allows the user to switch between GPU usage rates, memory, and temperature. Every frame in the image represents a node, with the name of the node written in the upper-right corner of the frame. A bar inside every frame represents a GPU, and the blue portion of the bar represents the monitored values. An orange section at the top of the bar means that the GPU is in use. Using the slide in the upper-right are, the user can adjust the colors of the columns to filter and highlight GPUs in a given numerical range. Select the Color Inversion check box to the right of the slide to switch the colors that denote values inside and outside the stated range.

JobsThe Jobs page shows job information and status, as well as jobs running in the current cluster, as shown below.

Chapter 4. Monitor 21

Jobs in the list can be filtered by the changing criteria at the top of the list, which include:

• Queue: You can filter the queues running in the system.

• Filter User: You can filter jobs by users who submitted jobs. By clicking a user name in the job list, you can view all jobs submitted by this user.

• Status: You can select Running, Waiting, or Completed. LiCO V5.3.0 and later versions allow you to cancel jobs in the Running or Waiting state. To cancel a job, select Action ➙ Cancel.

• Submit Time: filters the jobs using their submission time. You can select Last 1 month, Last 3 months, Last 6 months, or All.

In the job list, Scheduler ID indicates the ID of a job in the scheduler. You can click a scheduler ID to view detailed job information in the scheduler.

AlertsThe Alerts page shows alert information for all triggered alert rules, as shown below.

The displayed information includes:

• ID: the alert ID corresponding to the alert rule


• Name: the alert name corresponding to the alert rule

• Level: Critical, Serious, Alert, or Information

• Status: Unconfirmed, Confirmed, or Fixed

• Time: the time when the alert was triggered

• Select Date: Select the start time and end time to filter the alerts, or select 1 day, 7 days, or 30 days to view the alerts in the specified period back from the current time.

• Node: the monitored node corresponding to the alert rule. When a GPU alert takes place, the GPU serial number increases. For example, node1: gpu0.

• Comment: notes for the alert

• Action: Confirm, Fix, or Delete.

Alert events are classified into current events and all events. Current events include only unconfirmed events, whereas all events include confirmed events.

Alert event information includes:

• Serial Number: unique ID for the alert event

• Alert Name: name corresponding to the alert strategy

• Alert Grade: grade of the corresponding alert strategy

• Status: current status of the alert: Unconfirmed, Confirmed, or Fixed

• Alert Time: the time when the alert occurred

• Alert Node: name of the node on which the alert occurred

• Notes: the administrator’s description of this alert

Alert information can be filtered by selecting criteria at the top of the page, and multiple choices can be made for status and grade. Alert information can be filtered by time, such as last day, last three days, last week, and last month, or by time criteria manually set with start and end dates.

The user can act on a selected node by clicking the appropriate button in the action list, or by selecting a node and then clicking “Confirm,” “Resolve,” or “Delete.” The user can also select “Act on All” to perform the same action on all alert messages. Actions are defined as follows:

• Confirm: applicable to unconfirmed alerts. After confirmation, a reminder for the alert will not be shown in the upper right corner of the home page, and after action is taken, the status will be changed to “Confirmed.”

• Fix: applicable to unconfirmed and confirmed alerts. After the administrator has handled the alert, this action can be taken and the status will be changed to “Fixed”.

• Delete: applicable to unconfirmed, confirmed, and resolved alerts. After deletion, the alert will not be shown on the list.

OperationsThe Operations page records the actions by all users for all targets in the system, as shown below.

Chapter 4. Monitor 23

The operation monitoring page shows the following elements:

• Operator: operator account to which the action information belongs

• Module: the module of the action, such as user or job

• Action: specific commands for the action, such as creation or deletion

• Target: the target of the action, such as a user or node

• Time: the time when the target action occurred

Information will be displayed at the top of the page according to the filtering criteria. Operator information can be viewed by selecting “Operator” from the drop-down list. The “Target/Action” drop-down list allows for the user to filter action information by targets and actions. Alert information can be filtered by time, such as last day, last three days, last week, and last month, or by time criteria manually set with a start and end date.

The following target/action information is recorded in action monitoring:

• User: create, update, delete

• Job: create, re-run, cancel, delete

• Node: turn on, turn off

• Alerts: confirm, fix, delete, comment

• Policy: create, update, delete

• Billing Group: create, update, delete

• Billing Account: top up, deduct


Chapter 5. Reports

Reports consist of the following types:

• “Job Reports” on page 25: statistics and details of jobs, users, and billing groups

• “Expense Reports” on page 25: billing statistics for jobs and storage, as well as user consumption ranking

• “Cluster Reports” on page 26: statistics on the running status of cluster jobs

• “Alert Reports” on page 28: statistics and details of alerts

• “Utilization Reports” on page 28: running status, connected users, user login status, and user storage statistics for the node

Job reportsThe Job Reports page allows administrators to obtain reports on jobs.

The report filters include:

• Data Type: filters by job, user, or billing group.

• Time: supports pre-defined and self-defined time periods no longer than one year.

Note: 1 day, 7 days, or 30 days: indicates last 1 day, last 7 days, or last 30 days.

• Filter Users: filters the selected users.

The preview function includes:

• Job Report Preview: supports bar graphs and tables.

• User Report Preview: supports pie charts, bar graphs, and tables.

• Details: Pie charts and bar graphs are the default, but users can also show data in table form. Click on the right side of the pie chart to refresh current user/billing group job data.

• Billing group Report Preview: supports pie charts, bar graphs, and tables.

The report exporting function includes:

• Content: supports the export of statistics and detailed data.

• Report Format: supports Excel, PDF, and HTML.

Expense reportsThe Expense Reports page allows administrators to obtain reports on expenses.





• Filter Users: filters users based on the selected username or billing group.


• Resource: displays billing statistics for CPU, memory, storage, and generic resources. Generic resources generally include GPU, FPGA, and so on.

• Queue: displays billing statistics for jobs in all queues, excluding storage billing statistics.

• Ranking: displays top ten users in terms of billing statistics, listed in a descending order.

• Trend: displays daily billing statistics for the time range currently displayed. You can drag the scroll bar horizontally to change the time range displayed.

Cluster reportsThe Cluster Reports page analyzes the running status of cluster jobs over a period of time from various dimensions.





• Filter Users: filters users based on the selected username or billing group.

• Queue: filters the selected scheduler queue.

• Queued Job Threshold: Jobs with the queue time greater than or equal to the threshold will be considered as Queued Jobs, and the others will be considered as Not Queued Jobs.


• Summary: displays information including the number of completed jobs and queued jobs, queued job percentage, maximum and average execution time of completed jobs, as well as maximum and average queueing time of queued jobs.

• Resource: displays maximum and average usage of resources such as GPU, FPGA, and CPU cores, and the number of jobs associated with the resources.

• Trend: displays a line chart to show the trends in the number of not queued jobs and that of queued jobs per day.

Chapter 5. Reports 27

• Period: displays a bar chart to show the running status of cluster jobs in different time periods.

– By hour of day: A day is divided into 12 time segments, with each segment containing two hours. Jobs are grouped into different segments based on the time point of job submission, and the average number of queued jobs and that of not queued jobs are displayed for each time segment.

– By day of week: Jobs are grouped based on the day for job submission. The average number of queued jobs and that of not queued jobs are displayed for each day.

• Time Distribution: displays a line chart to show the running time of jobs, with the X axis being the job running time and the Y axis being the number of jobs.

• Resource Distribution: displays a line chart to show the usage of a specific resource on jobs, with the X axis being the usage of the specific resource, such as CPU core and generic resources, and the Y axis being the number of jobs using the resource.

Alert reportsThe Alert Reports page allows administrators to obtain reports on alerts. It supports pre-defined and self- defined time periods of up to one year.

You can click Preview to directly preview alert data, which can be shown as a pie chart, bar graph, or table.

You can also click Export and set the filters in the displayed Export dialog to export alert reports.

The parameters in the Export dialog are described as follows:

• Content: supports the export of statistics or detailed data.

• Alert Level: supports All, Critical, Error, Warning, or Information.

• Report Format: supports Excel, PDF, and HTML.

Note: After setting the filters and the report format, click OK to export the report.

Utilization reportsThe Utilization Reports page allows administrators to obtain reports on actions, as shown below.


• Data: supports data on CPU, memory, and networks.

• Filter Nodes: filters the selected nodes. Click Select and enter the node names in the Nodes field, with multiple node names separated by commas (,).

You can click Preview to preview filtered data in a graph.


You can also click Export and set the report format in the displayed Export dialog to export action reports.

Note: After setting the report format, click OK to export the report.

Chapter 5. Reports 29

Chapter 6. Admin

After logging in, an administrator can click Admin in the left navigation pane to access Container Images, Bills, VNC, Operation Logs, or Web Logs.

Container imagesLiCO runs all AI jobs within a container. The system supports the Singularity container platform, with different AI job templates running on different container images. Administrators can use this function to manage system container images.

Select Admin ➙ Container Images from the left navigation pane to enter the container image management page.

The parameters on the Container Images page are described as follows:

• Name: self-defined container image name

• Framework: frame to which the container image belongs

• Type: type of the container image. The value system means that the container image is created by the system administrator.

• Version: self-defined container image version

• Tag: self-defined container image tag

• Action: action on the container image, which can be Edit, Browse, Delete, Download, or Reupload.

Create a container imageStep 1. Select Admin ➙ Container Images from the left navigation pane.

Step 2. Click Create.

Step 3. Filled in the required information.

• Name: self-defined container image name


• Source File: container image file selected, which must be a singularity container image file. Otherwise, the container image will fail to be created.

• Save As: name of the container image file. Ensure that there is no container image file with the same name in the storage path.

Step 4. Click OK.

Edit a container imageStep 1. Select Admin ➙ Container Images from the left navigation pane.

Step 2. Find the container image you want to edit, and then select Action ➙ Edit.

Step 3. Edit the container image information as required.

Step 4. Click OK.

View a container imageStep 1. Select Admin ➙ Container Images from the left navigation pane.

Step 2. Find the container image for which you want to view its information, and then select Action ➙ Browse.

The information is described as follows:

• Image Path: absolute path of the container image. You can use this path in your job file.

• Description: description of the container image. If no description has been filled in, this information is not displayed.

Delete a container imageStep 1. Select Admin ➙ Container Images from the left navigation pane.

Step 2. Find the container image you want to delete, and then select Action ➙ Delete.

Step 3. Click OK.

Download a container imageStep 1. Select Admin ➙ Container Images from the left navigation pane.

Step 2. Find the container image you want to download, and then select Action ➙ Download.

Step 3. Click Browse and then select a folder to save the container image. Ensure that there is no container image file with the same name in the storage path.

Step 4. Click OK.

Reupload a container imageStep 1. Select Admin ➙ Container Images from the left navigation pane.

Step 2. Find the container image you want to reupload, and then select Action ➙ Reupload.

Step 3. Click Browse and then select a new container image file you want to upload.

Step 4. Click OK.

BillsLiCO 5.5.0 and later versions can bill administrators for jobs and storage instances. Administrators can download their daily and monthly summary bills generated automatically on the system.


Select Admin ➙ Bills from the left navigation pane.

The page for download bills is displayed.

The following types of bills are available:

• Daily Bill

Daily Bill enables administrators to check the billing information of every user on a specific day.

• Monthly Bill

Monthly Bill enables administrators to check the billing information of every user in a specific month. Bills for the current month are generated in the next month.

VNCIf you have installed and started VNC and its dependent components by referring to the LiCO installation documentation, you can start and enter the VNC management page in this section.

Find the required VNC and then select Action ➙ Open.

The VNC management page is displayed.

Chapter 6. Admin 33

Operation logsLog in as an administrator, and select Admin ➙ Operation Logs from the left navigation pane. The Operation Logs page is displayed, as shown below.

To download operation logs, set the time range and click Download.

Web logsLog in as an administrator, and select Admin ➙ Web Logs from the left navigation pane. The Web Logs page is displayed, as shown below.


To download web logs, click Download.

Chapter 6. Admin 35

Chapter 7. Settings

The Settings menu allows the user to manage alert rules, alert notification groups, notification adapters, and alert triggering scripts.

After logging in as an administrator, select Settings from the left navigation pane, and then select the desired sub-items.

SchedulerSelect Settings ➙ Scheduler from the left navigation pane.

The Scheduler page is displayed, which allows administrators to create, edit, delete queues, set queue state, and set node state.


Create a queueStep 1. Click Create on the Scheduler page.


• Queue: self-defined queue name

• Nodes: identifies the node(s) to be associated with this queue. Multiple node names may be specified using simple node range expressions such as "lx[10-20]". Note that a job can only be associated with one queue at any time.

• Default: indicates whether a queue is to be used by jobs that do not have specified queues

• Priority: Jobs submitted to a higher priority queue will be dispatched before pending jobs in lower priority queues and if possible they will preempt running jobs from lower priority queues. Note that a queue's priority takes precedence over a job's priority. The highest priority is 65533, while the lowest is 1.

• Max Time: The maximum running time for jobs. If the box is selected, the time is "UNLIMITED". Otherwise, you need to input the time, the input format is "<days>-<hours>:<minutes>". For example, "2-23:59".

• Over Subscribe: specifies whether compute resources (individual CPUs) in this queue can be shared by multiple jobs. Possible values are "YES", "NO", "EXCLUSIVE" and "FORCE". An optional job count specifies how many jobs can be allocated to use each resource.

• State: specifies whether jobs can be allocated nodes or queued in this queue. Possible values are "UP" and "DOWN". The value can also be "DRAIN" or "INACTIVE" when you edit or set the queue state.

– UP: indicates that new jobs can be added to the queue, and jobs can be allocated to nodes and run from the queue.


– DOWN: indicates that new jobs can be added to the queue, but queued jobs cannot be allocated to nodes and run from the queue. Jobs already running on the queue continue to run. They must be explicitly cancelled to force their termination.

– DRAIN: indicates that no new jobs can be added to the queue (job submission requests will be denied with an error message), but jobs already queued on the queue can be allocated to nodes and run.

– INACTIVE: indicates that no new job can be added to the queue, and jobs that are already in the queue cannot be allocated to nodes or run.

Step 3. Click Submit.

Edit a queueStep 1. Find the queue you want to edit, and then select Action ➙ Edit.

Step 2. Edit the queue information as required. For the parameter description, see “Create a queue” on page 37.


Delete a queueStep 1. Find the queue you want to delete, and then select Action ➙ Delete.


Set the state of a queueStep 1. Find the queue whose state you want to change, and then select Action ➙ Set Queue State.

Step 2. Set the state of the queue.


Set the node stateStep 1. Click Set Node State on the Scheduler page.


• Nodes: one node, multiple nodes, or a node expression

• Current State: indicates the current state of the node or nodes above

• New State: the state you want to set.

Alert policySelect Settings ➙ Alert Policy from the left navigation pane.

The Alert Policy page allows administrators to view the alert policy for the current cluster, and add, update, or delete alert rules, as shown below.


Create an alert policyStep 1. Click Create in the upper-left corner of the page.

Step 2. Fill in the alert rules. An alert policy must include the following information:

• Alert name: self-defined alert name

• Monitor: LiCO provides alert monitoring for the CPU usage rate, temperature, GPU usage rate, GPU temperature, network status, storage usage rate, energy consumption, and hardware problems.

• Condition: Set an alert trigger that is larger than, smaller than, or equal to a threshold value.

• Duration: For some monitored items, set the duration of the triggering condition. The default is 60 seconds.

• Risk Level: self-defined risk level, including critical, serious, alert, and information

• Notification Group: Notify one or more groups of users after an alert policy is triggered.

• Monitoring Node: Fill in the name of the node or nodes to be monitored. If left blank, the default is to monitor all nodes.

• Configure Script: Choose a script to run automatically after an alert is triggered.

• Notice: Turn on WeChat notifications and sound notifications.

• Status: Immediately start this alert strategy.

Step 3. Click OK.

Edit an alert policyStep 1. Find the alert rule you want to edit, and then select Action ➙ Edit.

Step 2. Edit the alert rules in the dialog.

Step 3. Click OK.

Delete an alert policyStep 1. Find the alert policy you want to delete, and then select Action ➙ Delete.

Step 2. Click OK.

Notification groupNotification groups are user groups that are notified when an alert is triggered.

Select Settings ➙ Notification Group from the left navigation pane.

Chapter 7. Settings 39

The Notification Group page is displayed, which allows administrators to create, edit, and delete notification groups.

Create a notification groupStep 1. In the upper-left corner of the Notification Group page, click Create.

Step 2. Type the group name, e-mail addresses, and mobile numbers for the notification group.

Step 3. Click OK.

The newly-created notification group is displayed in the list.

Edit a notification groupStep 1. Find the notification group you want to edit, and then select Action ➙ Edit.

Step 2. Edit the notification group information.

Step 3. Click OK.

Delete a notification groupStep 1. Find the notification group you want to delete, and then select Action ➙ Delete.

Step 2. Click OK.

The notification group is deleted.

NotificationSelect Settings ➙ Notification from the left navigation pane.

The Notification page is displayed, which allows administrators to manage the e-mail, SMS, and WeChat settings for the external alert API, as shown below.


Notes:

• Turn the alert API on or off by clicking the ON or OFF buttons in the upper-right corner.

• All changes to these settings will only be saved after you click Confirm at the bottom.

• Test alert notification connections by clicking Test at the bottom of each area.

ScriptsSelect Settings ➙ Scripts from the left navigation pane.

The Scripts page is displayed, which allows administrators to manage scripts for creating alerts. The scripts displayed on the script management page have self-defined alert rules. The information displayed includes the script name, file size, and upload time, as shown below.

Notes:

• The scripts are located in the /var/lib/lico/core/alert/scripts directory. You can change the directory in the LiCO configuration file.

• For security reasons, this page does not support uploading, updating, or deleting scripts. These actions should be performed using the background platform.

Chapter 7. Settings 41

Chapter 8. License

Select License from the left navigation pane.

The License page is displayed.

The License page provides the following information:

• Number: the maximum allowable number of CPUs and GPUs in a cluster

• Status:

– Valid: indicates the license is valid.

– Expired: indicates the license has expired.

– Limited: indicates the total number of existing CPUs and GPUs in a cluster exceeds the upper limit permitted by the license.

• Customer Id: customer ID

• Import Time: time when the license is imported

• Expire Date: time when the license becomes invalid

• License Quantity: the maximum number of resources permitted by the license

Import a licenseStep 1. On the License page, click Import.

Step 2. Click Browse to select the obtained license file.


Export a licenseFind the license you want to export, and then select Action ➙ Export.

The license file is exported.

Delete a licenseStep 1. Find the license you want to delete, and then select Action ➙ Delete.

Step 2. Click Confirm.

The license file is deleted.


Chapter 9. HPC cluster management

Most HPC functions may be completed using the interface. However, because HPC cluster management is complicated, some more complex actions require command lines or other tools.

View HPC cluster detailsStep 1. Select Monitor ➙ List View from the left navigation tree.

Step 2. View the HPC cluster information.The displayed information includes:

• Hostname: the cluster hostname

• Status: Idle, Busy, Off

• Health: Normal, Warning, Critical, Failed, Unknown

• Power: On, Off

• Type: compute, management, login, I/O, and other user-defined nodes

• BMC IP: IP address of the management module XCC

• OS IP: IP address for the node

• Hardware: the number of CPU cores on every node/the total memory on every node /the total storage on every node/the number of GPUs on every node

• Custom Group: the group to which the node currently belongs

Remote management of HPC cluster software

Interface managementOn the List View page, click the “BMC IP” link in the node list to open a Lenovo XCC management module interface and perform remote hardware management, including a remote on/off switch, a remote console, and hardware configuration.

Type the username /password (Factory Default: USERID/PASSW0RD) to log in to the XCC management interface.

See the XCC User Manual for details: http://sysmgt.lenovofiles.com/help/index.jsp

Command line management

Select the required node, and click or Console to open the control panel for the selected node.

xCAT managementLog in to the management node and perform remote management using xCAT commands.

Remote on/off command rpower:

rpower <noderange> [on|onstandby|off|suspend|reset|stat|state|boot]

# rpower c[01-03],d08 reset, Restart nodes c01,c02,c03,d08


http://sysmgt.lenovofiles.com/help/index.jsp

# rpower c[01-03],d08 state, View the on/off status of nodes c01,c02,c03,d08

Set Boot Order Command rsetboot:

rsetboot <noderange> [net|hd|cd|floppy|def|stat]

# rsetboot c[01-03],d08 net, Set nodes c01,c02,c03,d08 to boot from the network

Remote view of the node hardware device asset information command rinv:

Remote view of the node hardware device log information command reventlog:


Find more command techniques using the link below: http://sourceforge.net/p/xcat/wiki/XCAT_Commands/

Parallel commandsWhen using parallel commands on the SSH login head node, the following actions can be performed on nodes in a cluster in batches:

Parallel Command PSH: PSH Use: PSH node name shell command

Node names can be represented regularly or separated by commas. For example, to run an LS command on nodes c01,c02,c03,d08:

# psh c[01-03], d08 ls

Parallel File Copying PSCP: PSCP Use: PSCP source file node name:/target directory

Node names can be represented regularly or separated by commas. For example, to copy the data.txt file from the computer to the c01,c02,c03,d08 directories:

# pscp data.txt c[01-03], d08:/opt

Job scheduling commandsLiCO supports lifecycle actions such as uploading files, or submitting, cancelling, re-running, and deleting jobs. See the LiCO User Manual.

Chapter 9. HPC cluster management 47

http://sourceforge.net/p/xcat/wiki/XCAT_Commands/

The administrator may use command lines to perform more complicated scheduling management jobs.

Queue commandsQueue management includes viewing, creating, and modifying queues. In queue management, the current user needs to log in to the head node and utilize Slurm scheduler command lines.

SSH Login for the Head Node:

• View the queue:

[root@mgt /]# sinfo

• Create a queue:

1. Modify the Slurm configuration file /etc/slurm/slurm.conf, and add the following content:

PartitionName=test Nodes=headnode, computenode1 Default=YES MaxTime=INFINITE State=UP

2. Restart Slurm-related services:

– On the management node:

[root@mgt /]# systemctl restart slurmctld

– On the compute node:

[root@mgt /]# systemctl restart slurmd

After completing the steps above, the newly-created queue can be viewed on the interface.

The queue may be modified by changing the configuration file /etc/slurm/slurm.conf. The steps are the same as those for queue creation. View the queue parameters via scontrol show partition.

For more queue management commands, refer to http://slurm.schedmd.com/.

Job managementJob management can be performed on the LiCO interface. An administrator can view and act on a job by giving commands to the scheduler.

SSH Login to Head Node:

• View job status:

[root@mgt /]# squeue -a

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

428 compute zhangtes testuser R 5:19 1 testcomputenode01

429 compute zhangtes testuser R 4:49 1 testcomputenode01

430 compute mnist-pa ls-test R 4:37 1 testcomputenode01

• View detailed job status:

[root@mgt /]# scontrol show jobs

• Use jobid to view the detailed status of a certain job:

[root@mgt /]# scontrol show jobs 428

• Use jobid to cancel a job running or in the queue:

[root@mgt /]# scancel 428


http://slurm.schedmd.com/

For more job management commands, refer to http://slurm.schedmd.com/.

Note: If a job is submitted through Slurm command lines, it will not start billing on the LiCO system.

Chapter 9. HPC cluster management 49

http://slurm.schedmd.com/

Appendix A. Important information

Restart LiCOIf LiCO do not work normally, restart it.

• Stop Service:

[root@mgt lico_6.x]# systemctl stop lico

• Start Service:

[root@mgt lico_6.x]# systemctl start lico

• View LiCO Status:

[root@mgt lico_6.x]# systemctl status lico

When LiCO starts normally, the screen shows the following information:

[root@mgt lico_6.x]# systemctl status lico

lico.service - lenovo hpc project

Loaded: loaded (/usr/lib/systemd/system/lico.service; disabled; vendor preset: disabled)

Active: active (running) since Thu 2017-11-02 16:48:05 CST; 3h 59min ago

Main PID: 381046 (lico)

Resolve a failed job submissionThe failure to submit a job on the LiCO interface may be caused by a poorly-configured Slurm scheduler. To check the cause of the failure, try the following suggestions:

• Use SSH to log in to the management node and re-submit the job using command lines: "cd" to the current user directory, find the job file, submit the job through sbatch jobfile.slurm, and then check to see which error message is returned. Resource limits may have been exceeded. For example, the job needs 100 cores, but there are only 80 in the cluster.

• Run the Slurm command sinfo on the head node and view the compute node status and resource status for the cluster.

If sinfo returns no results, no nodes have been added to the scheduling node. Open /etc/slurm/slurm.conf and add a compute node using the following format: NodeName=nodename CPUs=cores State=node status

Following the addition, restarting the Slurm service at the head node may be required for the addition to take effect.

[root@mgt lico_6.x]# sysemctl restart slurmctld

If sinfo shows that some nodes are down, check whether Slurm services have been started on the down nodes.

[root@mgt lico_6.x]# systemctl status slurmd

Run Slurm command ssccoonnttrrooll sshhooww ppaarrttiittiioonn on the management node to view the queue settings.


Batch deletion of jobs in the databaseAfter LiCO has been running for a long period, jobs accumulate. Jobs can be deleted through the Manage interface, but if you want to delete jobs in batches, they can be deleted from the database directly. LiCO uses mariadb as a database. The database is on the management node and the database name is mariadb. The corresponding job table is webconsole_job. You can use a visualization tool, and use command lines to delete unnecessary jobs from the database.

Failure to view or delete a VNCYou can restart LiCO if you fail to view a VNC session on the LiCO page.

If you fail to delete a VNC session on the LiCO page, log in to the VNC session node, view the process number of the VNC session to be deleted with the command ps -ef|grep Xvnc, and delete the VNC session using the deletion process. Use kkiillll rather than kkiillll--99 when deleting a session. Information on the LiCO page will update after about 30 seconds.

Data sources for GPU monitoringLiCO can only monitor GPUs produced by NVIDIA. Monitoring data (including GPU usage rates, memory, temperature, and usage status) is obtained through the official NVIDIA API.

To check GPU monitoring data on the node’s operating system, run nnvviiddiiaa--ssmmii on the command line to check.

Clean up jobs submitted through accounts not managed by LiCOIn some special cases, some customers use accounts that are not managed by LiCO to submit jobs. Under such circumstances, there are many empty job records in the system, which are in queuing state and cannot be deleted through the web interface. If this occurs, log in to your system service node as the root user and restart LiCO. After LiCO restarts, the state of these empty jobs is automatically set to completed, and there is no impact on system running.

Scheduler troubleshootingWhen you create, edit, or delete a queue, or set the status of a queue or node, the following error might be displayed:

This error occurs when the slurm master node and LiCO management node are deployed on different physical nodes.


Solution

• Manual synchronization

Overwrite the file /etc/slurm/slurm.conf on the slurm master node with that on the LiCO management node.

• Automatic synchronization

1. Provide the auto sync script by the administrator, using which the file /etc/slurm/slurm.conf on the slurm master node can be overwritten by that of the LiCO management node.

2. Create a directory named /var/lib/lico/core/scripts/ and upload the script to the directory on the LiCO management node.

3. Configure the script path to the LiCO config file (/etc/lico/lico.ini.d/scheduler.ini).

# The sync script must be an executable file

SCHEDULER_CONFIG_SYNC_SCRIPT_PATH = /var/lib/lico/core/scripts/sync_script

4. Run the following command to restart LiCO, so that the configuration can take effect:

systemctl restart lico

MPI issues troubleshootingWhen running an Open MPI program, the following error might be displayed:

WARNING: Open MPI accepted a TCP connection from what appears to be a another Open MPI process

but cannot find a corresponding process entry for that peer.

If the TCP connection is ignored, the Open MPI program might not be executed properly.

When the Open MPI program uses the unroutable USB NICs, whose name might be “enp0s20f0u1u6” or similar under RedHat/CentOS 7, this warning might be displayed. Select one of the following workarounds to resolve this issue:

• Disable the USB NICs on all nodes by running the following command:

psh all ifconfig enp0s20f0u1u6 down

Note: This step might interrupt the running Lenovo management tools, such as OneCLI. When you want to use OneCLI, enable the NICs for a while.

• Instruct Open MPI to ignore the NICs by running the following command:

mpirun --mca btl_tcp_if_exclude enp0s20f0u1u6

Note: It is recommended to create the custom system-wide MPI templates.

• Permanently disable USB NICs by running the following command:

rmmod cdc_ether

Note: This step might permanently disable OneCLI and other Lenovo management tools.

Different memory usage on LiCO and Linux OSThe memory used on LiCO and that on Linux OS are different, caused by the difference between Ganglia and OS.

• On Linux OS, the used memory can be obtained by running the free command.

Calculation formula: Used memory = total – free – buffers – cache – slab

Appendix A. Important information 53

• On LiCO, all the memory metrics are obtained from Ganglia, which provides all the metrics mentioned above except for slab.

Calculation formula: Used memory = total – free – buffers – cache

Notices and trademarks

Notices

Lenovo may not offer the products, services, or features discussed in this document in all countries. Consult your local Lenovo representative for information on the products and services currently available in your area. Any reference to a Lenovo product, program, or service is not intended to state or imply that only that Lenovo product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any Lenovo intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any other product, program, or service.

Lenovo may have patents or pending patent programs covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to:

Lenovo (United States), Inc. 8001 Development DriveMorrisville, NC 27560U.S.A.Attention: Lenovo Director of Licensing

LENOVO PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

Changes are made periodically to the information herein; these changes will be incorporated in new editions of the publication. To provide better service, Lenovo reserves the right to improve and/or modify the products and software programs described in the manuals included with your computer, and the content of the manual, at any time without additional notice.

The software interface and function and hardware configuration described in the manuals included with your computer might not match exactly the actual configuration of the computer that you purchase. For the configuration of the product, refer to the related contract (if any) or product packing list, or consult the distributor for the product sales. Lenovo may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

The products described in this document are not intended for use in implantation or other life support applications where malfunction may result in injury or death to persons. The information contained in this document does not affect or change Lenovo product specifications or warranties. Nothing in this document shall operate as an express or implied license or indemnity under the intellectual property rights of Lenovo or third parties. All information contained in this document was obtained in specific environments and is presented as an illustration. The result obtained in other operating environments may vary.

Lenovo may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.


Any references in this publication to non-Lenovo Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this Lenovo product, and use of those Web sites is at your own risk.

Any performance data contained herein was determined in a controlled environment. Therefore, the result obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.

This document is copyrighted by Lenovo and is not covered by any open source license, including any Linux agreement(s) which may accompany software included with this product. Lenovo may update this document at any time without notice.

For the latest information or any questions or comments, contact or visit the Lenovo Web site:

https://support.lenovo.com

Trademarks

LENOVO, LENOVO logo, THINKPAD, THINKPAD logo, TRACKPOINT, ULTRACONNECT, and Yoga are trademarks of Lenovo. Microsoft, Windows, Direct3D, BitLocker, and Cortana are trademarks of the Microsoft group of companies. Ubuntu is a registered trademark of Canonical Ltd. The terms HDMI and HDMI High-Definition Multimedia Interface are trademarks or registered trademarks of HDMI Licensing LLC in the United States and other countries. Wi-Fi, Wi-Fi Alliance, and Miracast are registered trademarks of Wi- Fi Alliance. USB-C is a trademark of USB Implementers Forum. All other trademarks are the property of their respective owners. © 2020 Lenovo.

Appendix A. Important information 55

https://support.lenovo.com

lico 6.0.0 administrator guidedownload.lenovo.com/servers_pdf/lico_6.0.0... · change the account...

Documents