OFI SW - Progress

Sean Hefty - Intel Corporation

Overview

• Ability of the underlying implementation to complete processing of an asynchronous request

• Need to consider ALL asynchronous requests– Connections, address resolution, data transfers, event

processing, completions, etc.

• HW/SW mix

www.openfabrics.org 2

All(?) current solutions require significant software components

Decisions

• Single or multiple progress models– Separate at provider level– Separate based on operation

• Performance implications– Visibility to the application– Performance versus ease of use

www.openfabrics.org 3

Current Proposal Concepts

• Define multiple progress models– Visible by application– Provider indicates optimal mode for their HW– See following slides

• Application can modify progress model– Trade-off in favor of ease of use

• Progress is at provider level– May be too coarse

www.openfabrics.org 4

Auto-Progress

• Underlying implementation will make progress on a request with no explicit intervention from the app– Necessary work may be done within directly related

function call

• May require provider thread allocation– Includes kernel support

www.openfabrics.org 5

Simplest solution for app

Implicit Progress Model

• Underlying implementation uses application thread to make forward progress

• Progress depends on app calling into provider– Any function is usable– Progress may be limited to a specific ‘progress

domain’ (e.g. specific endpoint)

• Well defined behavior needed to prevent deadlock– E.g. provider may not block application thread for an

arbitrary timewww.openfabrics.org 6

Explicit Progress Model

• Provider exposes ‘progress’ call that application must invoke to advance progress

• Advances progress across all provider resources associated with the app

www.openfabrics.org 7

Notification Methods

• If software advances progress, callbacks may be the highest performing method of reporting completions– Immediate notification from progressing thread– Restricts what an application can do from the callback– Application queuing of an event may be more efficient

than provider queuing event

• Event queues are ideal if hardware advances progress

www.openfabrics.org 8

Discussion

enum fi_progress {FI_PROGRESS_AUTO,FI_PROGRES_INDIRECT,FI_PROGRESS_EXPLICIT

};

int fi_progress(domain, enum fi_progress progress);int fi_control(domain, FI_[START|STOP]_PROGRESS, flags);

www.openfabrics.org 9

How to expose this?