Briefing Presentation to RAC ReviewersNovember 6, 2017

1

Admin

2

● Please mute unless you’re speaking.● Questions either vocally or through Vidyo chat.● We are recording.

3

Committee Chair Committee meeting

Astronomy & Subatomic Physics Victoria Kaspi Astronomy: Wednesday Jan 10, 11:00am - 1:00 pm ESTSubatomic Physics: Monday January 15, 1:00 pm - 3:00 pm EST

Bioinformatics, Neuroscience and Medical Imaging

Tomas Paus Thursday January 18, 12:00 pm - 3:00 pm EST

Chemistry, Biochemistry, Biophysics Natalie Cann Friday January 12, 11:00 am - 2:00 pm EST

Engineering, Mathematical and Computer Sciences

Cécile Devaud TBC

Environmental and Earth Sciences Paul Kushner Friday January 19, 11:00 am - 2:00 pm EST

Humanities and Social Sciences Anabel Quan-Haase Monday January 8, 12:00 pm - 2:00 pm EST

Nano, Materials and Condensed Matter Jeremy Schofield Monday January 15, 1:00 pm - 4:00 pm EST

Research Platforms and Portals Katja Fennel Monday January 8, 10 am - 1:00 pm

Tomas Paus, Acting NRAC Chair

Committee Chairs and meeting dates

Key Dates

4

Start Finish Activity

Oct 3 Nov 16 RPP and RRG submission

Nov 17 Nov 21 Chairs accept or reject proposals to be reviewed in their committees

Nov 21 Nov 24 Scientific reviewers declare conflicts of interest

Nov 27 Dec 1 Chairs to assign proposals to reviewers

Dec 4 Dec 5 Eduardo to send assigned proposals to scientific reviewers

Dec 4 Jan 5 Review submission: all scientific reviews must be submitted on CCDB by January 5, 2018

Jan 8 Jan 19 Virtual committee meetings (calendar invites have been sent)

Feb 14 Feb 4 RAC Chairs F2F meeting in Toronto

RAC Structure

5

Reviewed by RAC Admin CommitteeReuse scientific score from previous RPP app

Regular Stream

Large Stream

Portal

Platform

RRG

RAC

Repeating apps (eligible PIs only)

Repeating apps (multi-year awards)

Fast Track

New application

RPP

New application

Annual Progress Report

Scientific and Technical Review

Technical ReviewReuse scientific score from previous RRG app

}

Scientific and Technical Review}

Full applicationRequires CCV

Lightweight applicationNo CCV required

Full applicationRequires CCV

Lightweight reportNo CCV required

6

Historical Applications Submitted

Resources Under Pressure

7

8

Some contextual info (FYI)

● Compute Canada does not have a Chief Science Officer at the moment

● ~21 legacy systems will be defunded by April 2018 and are no longer allocatable.

○ Many people will have to be migrated

● 297 PIs were invited to Fast Track: this is the largest number ever

○ 225 applications received (submission deadline was November 2)

● We expect to receive ~35% less new applications than last year, which should make the review process a bit more lightweight and give committees more time to discuss discrepancies or large applications

● Total number of allocatable CPU cores will be a bit higher than last year but less GPUs: it will still be a tough competition.

● Storage is Ok (except for cloud storage, which is a bit tight).

Conflicts of Interest

9

10

Permissions and Conflict of interest

• Reviewers first have to accept NDA and COI policy online in CCDB.

• Reviewers in one committee can view and review all proposals assigned to that committee, but they cannot see reviews submitted by any other reviewer.

• This year, some RRG reviewers will be asked to review large RPP applications. For this reason, all committees will have access to all RPP applications and will be able to review any of them as required by the committee chair.

• Reviewers will be asked to declare direct and indirect COIs with any applications in their disciplinary committee AND the RPP competition before the Chairs assign proposals.

Direct Conflict of Interest

Reviewers are in direct conflict if they:

● are from the same university department as the applicant(s)

● have been a research supervisor or graduate student of the applicant(s) within the past two (2) years

● are providing direct or indirect support for the application

● have collaborated with the applicant(s) within the past two (2) years or have plans to collaborate in the immediate future

● are an employee of a non-academic organization with which the applicant has had collaboration within the past six (6) years

● are in any other potential conflict of interest (e.g., personal, financial)

11

Indirect Conflict of Interest

12

Reviewers are in indirect conflict if they:

● are from the same immediate institution or company as the applicant, and who interact with the applicant in the course of their duties at the institution or company;

● are a close personal friend or relative of the applicant;

● have had long-standing scientific or personal differences with the applicant;

● are in a position to gain or lose financially from the outcome of the application (e.g., hold stock in the company of an industry partner or a competitor), or for some other reason feel that they cannot provide an objective review of the application

Conflicts of Interest

• Essential to follow COI policy in all meetings• The agenda of the meeting will include the list of

proposals in the order in which they will be discussed and the name of reviewers in conflict with any application

• Reviewers in direct conflicts must leave the discussion (can rejoin the meeting afterwards). • Reviewer whose own application is being discussed• Reviewer who directly collaborate with PI• Reviewer from same university + same department

• Reviewers in indirect conflicts can stay in the meeting but cannot participate in the discussion of the proposal in conflict.

13

We welcome your feedback

We will be looking for your input at the end of the 2018 competition. Please take notes on what should be improved and pass to your RAC chair or to CC.

14

Review Process Overview

15

• Ensure application assessment is completed using a fair and non-conflicted process.

• Evaluate the merit of each proposal to the RRG or RPP Competition.

• Provide feedback to all applicants submitting to RRG or the RPP Competition.

• Award resource allocations based on review.

Review Objectives

16

17

Where to view applications?

RRG Streaming

Regular stream:○ Batch Compute between 50-1999 Core Years○ Storage between 10-999 TBs○ GPUs between 10-199 GPU years○ Cloud Compute between 80-499 VCPUs○ Persistent Cloud between 10-99 VCPUs○ Workload: Shorter Technical Justification document.

Large stream: Exceeds at least one of the resource limits for Regular as defined above

○ Workload: Larger and more detailed Technical Justification document (similar to last year’s template)

18

19

Stream Evaluation

Regular ● At least 2 reviews and 1 technical review.● Discussed at meeting to determine final science score.

Large ● At least 3 reviews and 1 technical review. ● Discussed at meeting to determine final science score.● Discussed at the Chairs meeting to determine final allocation.

RRG Streaming - Evaluation

RRG Evaluation Criteria

• Reviewers can see all proposals in their committee. They only need to score what they are assigned, but should be encouraged to read others to participate in the discussions.

• Reviewers submit scientific review forms - score on 5-point scale:5 = Exceptional4 = Outstanding3 = Very Strong2 = Strong1 = Moderate0 = Insufficient

Note - this score will be used for final scaling. Reviewers are encouraged to use the available dynamic range.

• Note: although scoring system is similar to NSERC's, the goal of the RAC review process is not the same.

20

RRG - Quality of the ScienceWhen ranking the merit of the proposed research project, consider such factors as:

• originality and innovation,• significance and expected contributions to research,• clarity and scope of objectives,• clarity and appropriateness of methodology,• feasibility, and discussion of relevant issues.

Your assessment should also include consideration of Highly Qualified Personnel (HQP) training and support, in particular for the potential for unique training opportunities for HQP.

21

RRG - Quality of the Science• Exceptional (5 points) - Proposed research program is ambitious and feasible, is

clearly presented, is extremely original and innovative, and is likely to have impact by leading to groundbreaking advances in the area and/or leading to a technology or policy that addresses socio-economic or environmental needs. Long-term vision, short-term objectives, timescales, and expected impact are clearly defined and justified. The methodology is novel, clearly described, and appropriate.

• Outstanding (4 points) - Proposed research program is clearly presented, is highly original and innovative, and is likely to have impact by contributing to substantial advances in the area and/or leading to a technology or policy that addresses socio-economic or environmental needs. Long-term goals, short-term objectives, timescales, and expected impact are well described and justified. The methodology is clearly defined and appropriate.

22

RRG - Quality of the Science

• Very Strong (3 points) - Proposed research program is clearly presented, is original and innovative, and is likely to have impact by leading to advancements and/or addressing socio-economic or environmental needs. Long-term goals are defined, short-term objectives are planned, and timescales and expected impact are well defined. The methodology is clearly described and appropriate.

• Strong (2 points) - Proposed research program is clearly presented, is original and innovative, and is likely to have impact and/or address socio-economic or environmental needs. Long-term goals and short-term objectives are described. The methodology is described and appropriate.

• Moderate (1 point) - Proposed research program is reasonably well presented, has original and innovative aspects, and may have impact on and/or address socio-economic or environmental needs. Long-term and short-term objectives are described. The methodology is partially described and/or appropriate.

• Insufficient (0 points) - Proposed research program as presented lacks clarity, is of limited originality and innovation, and/or does not clearly address socio-economic or environmental challenges. Objectives are not properly described and/or likely not attainable. Methodology is not clearly defined and/or appropriate.

23

RRG - Quality of the Team

Exceptional (5 points) - Acknowledged as a leader who has continued, over the last five years, to make influential accomplishments at the highest level of quality, impact, and/or importance to a broad community.Outstanding (4 points) - The accomplishments presented in the application were deemed to be far superior in quality, impact, and/or importance to a broad community.Very Strong (3 points) - The accomplishments presented in the application were deemed to be of superior quality, impact, and/or importance.Strong (2 points) - The accomplishments presented in the application were deemed to be solid in their quality, impact, and/or importance.Moderate (1 point) - The accomplishments presented in the application were deemed to be of reasonable quality, impact, and/or importance.Insufficient (0 points) - The accomplishments presented in the application were deemed to be below an acceptable level of quality, impact, and/or importance.

24

RRG Scientific Review form

25

Please encourage people to provide feedback to the applicant here

RPP Streaming

26

1. Service Portal: a. Do not require large computing or storage resources, but may require

support effort by the Compute Canada technical team. b. Often use the Compute Canada Cloud, generally require a static IP

address, and may (or may not) have more stringent up-time requirements than most research projects.

c. Reduced documentation/justification expected.2. Compute and Storage Platform

a. Any RPP proposal requiring at least 50 core years (or 50 VCPUs in the cloud) OR at least 50 TB of storage will be considered a Compute and Storage Platform.

Note: Some corner cases may appear in the RAC 2018 round, e.g, a large portal that is requesting >50 CY but that is not really a platform. Because the request falls in the platform limit, it should be scored by two reviewers but not punished for asking what they need.

27

RPP Streaming - Evaluation

Stream Evaluation

Service Portal ● At least 1 review and 1 technical review.● May be awarded without further ERC discussion if reviews are

positive.

Compute and Storage Platform

● At least 3* reviews and 1 technical review.● Mandatory scoring discussion at the RPP ERC meeting.● Potentially discussed at the Chairs meeting to determine final

allocation.

*At least 2 reviews from RPP committee and 1 review from disciplinary RRG committeeRRG reviewers will be asked to submit the review but do not have to attend the RPP committee meeting

● Reviewers submit scientific review forms - score on 30-point scale:Evaluation of the Strategic Plan (up to 25 points)

■ Goal and Alignment (up to 5 points)■ Use and Development of a Platform/Portal (up to 5 points)■ Expected Impacts (up to 5 points)■ Management and Operation of the Platform/Portal (up to 5 points)■ Highly Qualified Personnel (up to 5 points)

Appropriateness of Resources Requested (up to 5 points)

● Each section of the application must be scored individually based on the ability to address each sub-criteria above.

● Finals RPP scores are converted to a 5-point scale to match RRG scores.

28

RPP Evaluation Criteria

29

RPP Scientific Review form

Please encourage people to provide feedback to the applicant here

Important Principles

• PIs should not be punished for asking for what they need. We should not punish large asks.

• “Reasonableness” applies to whether amount requested is suitable for proposed task.

• Focus of the committee is Science+Reasonableness.

30

Expert Review Committee Meetings

• Reviewers will complete pre-assessments on CCDB by January 5

• Discuss each proposal:• First reviewer leads discussion, provides brief

overview of proposal.• Second reviewer presents any important

disagreements with first reviewer. • Chair will focus time on contentious cases. If two

reviewers agree on score, may skip presentation. • Primary purpose of the meeting is to agree on

scoring for each proposal.

31

• RAC-Admin committee meets F2F and attempts to make the allocations actually fit on the machines that we have available.

• may involve moving some allocations• may involve splitting some allocations• will certainly involve applying a final scaling function based on

science scores. For example:• proposals above score=X.Y get 100% of target• proposals above score=Y.Z but less than X.Y get 85% of

target• ...• proposals below score=Q.Q get 0% of target

• Not applied blindly! Multidisciplinary committee could agree that giving a certain project below a certain target is fatal to that project and that it should not be scaled below that number.

32

What Happens to your Recommendations?

Questions?

33