ti confidential – nda restrictions 10/8/20151 “resource management on keystone devices" by...

TI Confidential – NDA Restrictions04/21/23 1

“Resource Management on Keystone Devices" by Justin Sobota

1/16/13

TI Confidential – NDA Restrictions

Agenda

• Why is Resource Management Needed on Keystone Devices?

• Keystone I Resource Manager (RM) LLD• Keystone I RM Lessons Learned• Keystone II RM• Q & A• Demo

04/21/23 2


Why is Resource Management Needed on Keystone Devices?• Resource sharing problems introduced with release of multicore devices based on

Keystone architecture– Resource contention was managed by LLDs on DSP-only multicore solutions

• NySh

– Resource contention not managed on (ARM + DSP) solutions• ARM (Linux kernel) doesn’t use LLDs• Resource contention between ARM & DSPs – running Linux, BIOS, and bare-metal

– No easy way to sync resource usage between ARM and DSP cores

• Combination of SYS/BIOS and Keystone IP initialization uncovered sharing problems at application/task level

– RTSC allows for plugging of pre-main startup routines

– Arbitration of IP hardware startup when integrating an existing post-main application with app that utilizes SYS/BIOS pre-main capabilities

• OpenMP & QMSS

• Keystone II architecture introduces devices with multiple ARMs, as well as DSPs

– Resource sharing problems will only get worse if not solved in a robust manner

04/21/23 3


Keystone I Resource Manager (RM) LLD

• Prevents DSP applications from stepping on resources taken by ARM Linux

• Initialization and usage permissions for select LLD resources defined in user-defined resource table– QMSS, CPPI, and PA LLDs

• Operates under hood of LLDs– Resource init and usage requests made between LLDs and RM directly

• LLD operates normally if resource request “allowed”• LLD returns error if resource request “denied”

– Application must take steps to handle

• Permission tables located in shared memory– Restricts RM to DSP cores only

• LLD resource storage still in hands of LLD– Full backwards compatibility with applications not utilizing RM

Quick, “dirty” resource management solution for Keystone I devices

04/21/23 4


Keystone I RM Lessons Learned

• Very tedious to add new resources since RM has static definitions in both code and data of resources it manages

• Privileges for DSP cores only skims surface• Resource table with privileges defined at compile time

– Privileges cannot be manipulated at runtime

• System integrator must align Linux DTB resources with DSP RM resource table– Tedious and error-prone

• Shared memory architecture provides no communication path to ARM– Not easily portable to new devices

04/21/23 5


Keystone II RM – Major Requirements• Remove static definitions of managed resources from RM

– Easy addition of new resources

• Enable management of resources at all levels within system software architecture– Core, task, application component (LLD)– Pre/post-main execution

• Runtime modification of resource permissions• Automate reservation of resources taken by Linux kernel• Generic, processor-independent transport interface that allows

RM instances to communicate regardless of device hardware architecture– Transport glue logic provided by application– Easy to port RM to new devices

04/21/23 6


Keystone II RM - Overview– Instance-based Client/Server Architecture:

• Three instance hierarchy:– RM Server – Global management of resources and permission policies– RM Client Delegate (CD)

» Offloads management of resource subsets from Server. » Provides singular data path to Server

– RM Client – Provide resource services to system software elements• Resource services provided via instance service API • No restrictions on RM instance topology within system

– RM Instances Communication Over Generic Transport Interface• Application must setup data paths between RM instances• Allows RM to run on any device architecture without modification to RM source

– Resources Tracked by RM Defined in Global Resource List (GRL)• GRL captures all resources that will be tracked for a given device• Based on device spec and CSL• Facilitates automatic extraction of resources used by ARM Linux from Linux DTB

– Policies Specify RM Instance Resource Privileges• Resource initialization, usage, and exclusive right privileges assigned to RM instances

– Resource assignment to RM instances allows resource management at all software system levels• Runtime modification of policy privileges

– APIs and Linux CLI (Planned)

– Resources Stored within Balanced Search Tree Allocators• Reduce memory usage and resource lookup times• Allocators facilitated by NameServer

04/21/23 7


Keystone II RM - Overview

04/21/23 8

ARM/DSP n+2

RM Client Instance

ARM/DSP n

Transport-Specific Data Path

ARM/DSP n+1

ARM DSP Transport

Transport API

RM CD Instance

Resources Allocated

from Server

CD Service Transaction

Handler

Client Service Transaction Handler

RM Server Instance

Resource Allocators

PA

QMSS

Allocation policies

CPPI

QMSS

Etc

User Mode (ARM)

Available resources are inverse of Linux DTB

Resource Policies

Transport API

PA

CPPI

QMSS

Etc

Memory Allocator

CPPI

PA

Mem Alloc

Etc

ARM/DSP n+3

RM Client Instance

QMSS

CPPI

PA

Mem Alloc

Etc

Service

Port

Service

Port

Transport API

Client Service Transaction Handler

DSP DSP Transport DSP DSP Transport

Service

Port

ARM DSP Transport

Transport API

CD Service Transaction Handler

Service

Port

Global Resource

List (GRL)

Linux DTB

DSP DSP Transport


Keystone II RM - Instances

04/21/23 9

Server Client Delegate (CD) Client• Assigned resource

permissions in policy based on instance name given at instance instantiation

• Can satisfy standard service requests via service API

• Can satisfy pre-main service requests

• Manages all system resources

• Multiple Servers within system must manage mutually exclusive sets of resources

• Manages all system policies• Maintains NameServer• Manages resource recovery

in case of fault in CD or Client instance (Planned)

• Can register with any number of CDs and Clients

• Assigned resource permissions in policy based on instance name given at instance instantiation


• Can satisfy pre-main service requests

• Manages subset of system resources provided by Server (Planned)

• Provided a sub-policy that is sync’d with Server level policy (Planned)

• Can register with any number of Clients

• Can register with at most one Server

• Assigned resource permissions in policy based on instance name given at instance instantiation


• Can register with at most one CD or Server


Keystone II RM – Instance Topology Example 1

04/21/23 10

Linux User-Space

RM Server RM Client Delegate RM Client

BIOS

IPC

LLD

DSP Multicore Application

Etc

ARM DSP 1 DSP 2

LLD Etc LLD Etc


Keystone II RM – Instance Topology Example 2

04/21/23 11

Linux User-Space

RM ServerRM Client Delegate

RM Client

BIOS

IPC

DSP Multicore Application

ARM DSP

LLD Etc

LLDLLD

RM Client


Keystone II RM - Services• Services available from any RM instance

– Instance name used as the basis for permission checks

• Current services– Allocate (initialization, usage)

• Allocate static resource• Allocate resource of unspecified base and alignment, length must be specified• Allocate resource based on existing NameServer name

– Free• Free static resource• Free resource based on existing NameServer name

– Map resource(s) to NameServer name

– Get resource(s) tied to existing NameServer name

– Unmap resource(s) from existing NameServer name

• Non-blocking service requests directly return result• Blocking service requests return ID to system

– Result of service returned at a later time via callback function provided with original service request

– Service result ID will match ID returned to system at time of request

04/21/23 12


Keystone II RM – Transport Interface

• RM instances communicate via generic transport API– Application provides transport path between RM instances

– Transport paths registered with RM instances• Destination RM instance type specified • Transport path send/receive function pointers provided

• Management of transports between RM instances handled by application

• Facilitates ability of RM to run on any device topology• RM source code porting not required in order to run on

new devices and architectures

04/21/23 13


Keystone II RM - Transport Example

04/21/23 14

DSP 1RM Client Delegate

ARM

RM ServertransHandle dstType

0x0001 Client Delegate

transHandle Transport

0x0001 ARMDSP

RM Transport APIs

transHandle dstType

0x0001 Server

0x0002 Client 1

0x0003 Client 2 RM Client 2

transHandle dstType


RM Transport APIs

RM Transport APIs

rmHandle transHandle Transport

0x0001 0x0001 DSPARM

0x0001 0x0002 DSPDSP

0x0001 0x0003 Internal

0x0002 0x0001 Internal

DSP 2

RM Client 1transHandle dstType


transHandle Transport

0x0001 DSPDSP

RM Transport APIs

ARMDSP Transport

DSPDSP Transport

App Transport Routing Map

App Transport

Routing Map

App Transport Routing Map


Keystone II RM - Global Resource List (GRL)• Specified in Device Tree Source (DTS) format

– Open source, dual GPL/BSD-licensed LIBFDT used for parsing GRL

• Input to Server on initialization• Server instantiates allocator for each resource specified in GRL• A GRL specification for a resource includes:

– Resource name

– Resource range (base + length)

– Linux DTB alias path (if applicable)

– Resource NameServer assignments (if applicable)

• Permissions not specified in GRL

04/21/23 15


Keystone II RM - GRL Example/dts-v1/;

/ {

device-name = "TCI6614";

/* Device Resource Definitions based on HW spec and CSL */

timers {

resource-range = <0 16>;

ns-assignment = “Core_0_Watchdog”, <0 1>;

};

qmss {

pdsps {


};

memory-regions {


linux-dtb-alias = "hwqueue@2a00000 regions region-12 id", <1 0>;

};

accumulator-ch {


};

gp-queue {


};

};

};

04/21/23 16


Keystone II RM - Policies

• Specified in Device Tree Source (DTS) format– Open source, dual GPL/BSD-licensed LIBFDT used for parsing GRL

• Assign resource permissions to RM instances• Available resource permissions:

– Initialization (i) – Allocate resource for initialization

– Usage (u) – Allocate resource for use

– Exclusive (x) – Resource not shared, can be allocated to no more than one RM instance

– Linux Shared (s) – Resource reserved by Linux kernel but can be shared with specified RM instances

• Policies can be modified, created, and deleted runtime (Planned)– Policy APIs available to application

– Linux CLI

04/21/23 17


Keystone II RM - Policy Example

04/21/23 18

/dts-v1/;

/* RM Server global policy */

/ {

/* All RM instances expected to be resource assignees within the policy. RM will fail at init if

* it finds a resource assignee that is not in the user-instances list */

valid-instances = "RM_Server",

"RM_Client_Delegate",

"RM_Client";

/* RM will deny any resource requests for resources not defined in the policy. */

/* Format for assigning resources to specific RM instances */

qmss {

gp-queue {

assignments = <2000 1000>, "iu=(RM_Server RM_Client_Delegate RM_Client)",

<3000 1>, "iux=(RM_Server) & iu=(RM_Client_Delegate RM_Client)";

<4000 1>, “s=(RM_Client_Delegate RM_Client)”;

};

};

cppi {

pass-rx-flow-id {

assignments = <0 20>, "iux=(*)";

};

};

};


Keystone II RM - Name Synchronization

04/21/23 19

Linux DTB

RM Service Port from RM Instance “myRmInst”

- myFirstResource- mySecondResource

Serv

ices A

PI

(*rmService) (&serviceInfo);

char resName[] = “myFirstResource”; serviceInfo.resourceName = &resName[0]; serviceInfo.resourceIndex= 4;

hw { 2nd-res { /* Kernel takes mySecondResource * in range 20 - 30 */ values = <20 10>; }; 3rd-res { … };};

Global Resource List (GRL)

RM Instance PolicyResourceManager

Resource Names must be synchronized!

/ { … myFirstResource { resource-range = <0 20> /* 20 myFirstResource’s in system */ }; mySecondResource { resource-range = <0 50>; /* Alias path must be path to value in Linux DTB */ linux-dtb-alias = “hw 2nd-res values”, <2 0 1>; };};

/ { valid-instances = “myRmInst”, “myOtherRmInst”; … myFirstResource { /* All myFirstResource can be * used by anyone in system */ assignments = <0 20>, “iu=(*); }; mySecondResource { /* All mySecondResource is split * between the two valid RM * instances in system */ assignments = <0 25>, “iu=(myRmInst)”, <25 25>, “iu=(myOtherRmInst)”; };};


BACKUP

04/21/23 20


Keystone I Resource Manager (RM) LLD (cont’d)

21

Core 0

App

QMSS LLD

App

CPPI LLD PA LLD QMSS LLD CPPI LLD PA LLD

RM RM

rmResourceTable{ {RES_ID, START, END}, { …, …, … }, …}

Shared Permissions

Core 1

…

resource_table.h

Permission validation

Permission validation

Rm_init maps rmResourceTable to internal Permission Tables

Queue/cppiChOpen/ … Queue/cppiChOpen/ …

Init/usePermissionCheck Init/usePermissionCheck

Rm_init Rm_start


Keystone II RM - Architecture

04/21/23 22

Resource Manager (RM)

Oth

er

OS

Application Code

Linux K

ern

el/X

DC

R

unti

me/B

IOS

OS

AL

DSP<->DSP Application Transport

ARM<->DSP Application Transport

Allo

catio

n P

olicy

CLI

(AR

M Lin

ux O

nly

)

ARM<->ARM Application Transport

OS Abstraction Layer OSAL APIs

LLDsBIOS

IPC

BIOS and LLD Internal RM Callout APIs

Service API

Transport API

Task


Keystone II RM - Architecture

• Key RM Interfaces– Service API

• Service ports opened from the RM service API

• Service ports provide resource services to system entities

• Services will be labeled as originated from the RM instance from which the service port originated

– Transport API• Generic transport interface over which RM communicates between

instances

• Application registers instances with one another over transport API

– Application provide transport glue function API to RM instances at registration

• RM calls registered transport glue functions when sending/receiving RM packets between RM instances

04/21/23 23


RM Server

• One RM server for a given exclusive set of system resources• Maintains global view of allocator, policy, and NameServer states• Can satisfy pre-main resource service requests

– OpenMP, BIOS, IPC, etc.

• Maintains the global policy that defines the resource permissions for all RM instances– Policy synchronization with policies stored on CDs not yet implemented

• Maintains all resource allocators– Who has which resources

• Maintains RM NameServer

04/21/23 24


RM Client Delegate (CD)

• Acts as a proxy for RM Server to RM Clients

• Quick response mechanism to Clients for certain services (Planned)

• Can be used as single path to Server when transport path to server is longer than transport path to Clients

– Instead of all Clients connecting to Server over a high cycle transport path they can connect to CD

– CD can allocate resources that have been provided it by the Server (Planned)• Will request new block of resources from Server when it runs out of resources to allocate

to Client service ports and service ports hanging off itself (Planned)

• Can satisfy pre-main allocation requests– Pre-main policy will be relayed to Server after initialization and transport

registration (Planned)

– Currently, the pre-main policy must be reflected in the global resource policy provided to the Server at initialization

• Due to policy synchronization feature not being implemented

04/21/23 25


RM Client

• “Dumb” RM instance• Sole purpose is to provide gateway to RM

services for system components• Minimum permissible policy unit• All service requests offloaded to the registered

RM CD or RM Server• No restriction on where Clients can exist

04/21/23 26


Keystone II RM – Instance Topologies

• Minimal requirements for RM instances within a system– If there are multiple RM Servers exist within a system

the resources they track must be mutually exclusive– RM CDs must be connected to at most one RM Server– RM Clients must be connected to at most one RM

Server or RM CD

• No restrictions on where RM instances can be created

• Service ports can be opened from any RM instance type

04/21/23 27


GRL and Policy Compilation

04/21/23 28

Human Read-able

Device Tree

Source (DTS) File

Converted Binary GRL

Device Tree Compiler

(DTC)

DTC Executed on Host (Linux)

Full Application

Binary

Compiled GRL

Device Tree Blob

(DTB)

Device-specific Global Resource

List (GRL)

and any Application Policies

Application Code

RM Library (compiled and linked with libfdt)

Compile Full Application

Rm_init(… *dtbPointer …)

libfdt APIs

Global Resource

DTB

RM Server Init

Data from GRL DTB used to initialize resource allocators

Converted Binary Application Policy

Compiled Policy Device

Tree Blob (DTB)

Policy DTB

Policy data is

validated at init.

Extracted later to check

permissions


Porting RM to New Devices

• Required to port RM to new device– Update GRL to exclude/include device-specific

resources that are to be tracked by RM– Update Policy resources permissions to match new

GRL resources– Update GRL with any new resource path alias to

reserved resources in Linux DTB

• No need to modify RM source• Application source will need to change

04/21/23 29


To Do

• Fault management• Policy editing Linux CLI• Remove RM NameServer

– Add NameServer interface so that an existing NameServer can be registered with RM

• Resource and policy front-loading on RM CD

04/21/23 30


Linux DTB Resource Extraction

• Cannot rely on Linux DTB to match GRL resource specification exactly and vice versa

• Alias path is clean way of extracted Linux reserved resources from Linux DTB

• Alias path specifies node path down to node containing value that should be reserved by RM for Linux kernel

• Resources specified as reserved by the Linux kernel are not able to be used by any RM instance unless otherwise specified in the Policy

04/21/23 31


Keystone II RM - Name Synchronization• Key to abstraction of static resource definitions away from RM• Global Resource List (GRL) defines each resource that RM will track

and how many of that resource exist• Policies define which RM instances are allowed to use the resources

defined in the GRL and how the instances can use them• Utilize Linux Device Tree framework specification, ePAPR, to define

easy to read, modify, and parse GRL and policy specifications– Facilitates extraction of resources used by Linux kernel from Linux DTB

file

– LIBFDT library included by RM to read, parse, and modify the GRL and Policy DTBs

• Dual GPL/BSD licensed

04/21/23 32


Allocators

• RM Allocators implemented using a open-source binary balance search tree algorithm– BSD-licensed

• Resource allocator nodes specify a resource base and length as well as a list of RM instances the resource is allocated to

• Wrapper written around the tree algorithm that expands and contracts the allocator tree nodes as resources are allocated and freed

• Saves time when searching for resources to allocate• Saves space when RM can be potentially managing

thousands to millions of resources04/21/23 33


Allocators - Example

04/21/2334

1

qmss-gp-queues1024

Assumptions:

-System has 1024 GP Queues

-Policy specifies blocks of 256

0-255Free

256-511Free

512-767Free

768-1023Free

2

qmss-gp-queues1024

-Allocate queues 335 – 345

-Allocate queue 767

256-334Free

335-345Reserved

346-511Free

512-766Free

0-255Free

256-511Parent

512-767Parent

768-1023Free

767Reserved

3

qmss-gp-queues1024

-Free queues 335 – 345

-Allocate queues 384-400

256-345Free

346-511Parent

512-766Free

0-255Free

256-511Parent

512-767Parent

768-1023Free

767Reserved

346-383Free

384-400Reserved

401-511Free

• Not exact to tree algorthm


RM NameServer

• Internal, simple NameServer• Not intended for long term use• Allows association of resources with a user-specified name• System entities can use service ports to map and unmap

resources to names, and allocate/free resources by name– Instance permissions will not be checked against the name given

the resource but the resource itself• Planning for future revisions of RM to have external NameServer where

RM would not have knowledge of names

• Aware of growing of NameServers in SDK offering– Plan to remove RM NameServer and replace with API allowing for

hookup to an external, existing NameServer

04/21/23 35


Pre-Main Resource Allocation

• Available on Servers and CDs• Startup policy checked on pre-main allocations• After initialization startup policy will be synced with

Server and along with any allocations– Startup policy must be in sync with global policy given

to server at init time

04/21/23 36

ti confidential – nda restrictions 10/8/20151 “resource management on keystone devices" by...

Documents