802.11 comment resolution – a tutorial

doc.: IEEE 802.11-13/0230r0

Report

March 2013

Adrian Stephens, Intel CorporationSlide 1

802.11 Comment Resolution – a Tutorial

Date: 2013-03-xx

Name Company Address Phone email Adrian Stephens Intel

Corporation [email protected]

Authors:

doc.: IEEE 802.11-13/0230r0

Report

Introduction

• This tutorial is intended to help 802.11 members to approve adequate comment resolutions to working group and sponsor letter ballots

• The audience is anybody in 802.11 involved in writing or voting on comment resolutions

• See also 11-11/1625 “Comment Resolution Guide” for more detail

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Why do we need a tutorial?

• We haven’t always don’t an adequate job in the past, which has cost us time/hot-air/embarrassment/quality

• We have specific rules that we need to follow

• New people are coming in all the time, which dilutes “corporate memory” of what to do and what to avoid

• We should, as much as we can, avoid “learning by making mistakes” because the cost of making mistakes in this process can be significant

March 2013


doc.: IEEE 802.11-13/0230r0

Report

The purpose of comment resolution

• To increase the level of support by ballotters for a draft

• To improve the quality of a draft standard

• To engage with voters and gain from their viewpoint/experience

• To complete a ballot with a draft of high quality that meets the approval of most (>75%, typically 95%) of its voters in a timely fashion.

• Note the tension – frequently cannot satisfy all the commenters and complete in a timely fashion

March 2013


doc.: IEEE 802.11-13/0230r0

Report

What can happen if we do this badly?

• We do not gain sufficiently from the possible benefits of engaging with voters

• We produce a poor quality draft document

• We get inadequate involvement from WG members who might give up trying to the process

• We risk delay by not dealing with real issues in a responsive or timely fashion

• We risk delay in the EC through appeals to process

• We risk delay in the IEEE-SA standards board through appeals to the process

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Valid Comments• A comment needs to identify where the issue is in the draft

• It needs to identify what the issue is

• It needs to identify a proposed change in sufficient detail that the CRC can readily identify changes that they would reasonably expect to satisfy the commenter.

• The wording from the IEEE-SA Standards Board Operations Manual (2010 p24) is: “This vote must be accompanied by one or more specific objections with proposed resolution in sufficient detail so that the specific wording of the changes that will cause the Do Not Approve voter to change his or her vote to Approve can readily be determined .”

• Needs to be in scope (next slide)

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Valid comment scopeA comment is in scope if:

•It is in the first ballot

•It is in a recirculation, and relates to material in the balloted draft changed since the previous draft, or relates to material affected by that change

•Is in a recirculation, and relates to text that is the target of an unsatisfied “must be satisfied” comment from a “no” voter.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Examples of invalid comments - 1

March 2013


There’s a bug somewhere in the document

fix it This is an invalid comment. It fails to locate and identify the issue. Fails to identify changes in sufficient detail so that the specific wording of the changes … can be determined. This is a wholly inadequate comment.

doc.: IEEE 802.11-13/0230r0

Report


March 2013


On p123 line 45 in equation 8.12 why is the lower limit of the summation a zero?

Clarify This is an invalid comment. The comment is asking a question. It is not proposing a change that can in any sense be interpreted as “specific wording” The CRC can attempt to answer the question in its comment resolution response, but is under no obligation so to do.

doc.: IEEE 802.11-13/0230r0

Report


March 2013


In clause 10, coexistence with mode X devices has not been considered.

Add a coexistence with mode X device solution.

This is an invalid comment. The comment identifies a potentially “big issue”, but doesn’t provide specific changes – it is essentially giving the CRC permission to do more work. In some cases comments of this type make the unstated assumption that the “big issue” is a requirement for the draft, even if no such requirement appears in the PAR or has otherwise been agreed: effectively the comment says “I think that coexistence with mode X devices should be a mandatory requirement.”

doc.: IEEE 802.11-13/0230r0

Report

Examples of invalid comments – 4

March 2013


The 802.11 style manual indicates that all Clause 10 parameter names should be in MixedCase.

Review all parameter lists in Clause 10 and adjust to conform to the MixedCase style.

This is an invalid comment. The commenter is giving the CRC (or probably the TG’s editor) permission to do more work. A volunteer (probably the editor) may agree to perform this task given an “accept” resolution. But if no volunteer to do the work is found, a rejection on the basis of lack of “sufficient detail” is also appropriate. This resolution may or may not cause the commenter to come back with an improved set of proposed changes.

doc.: IEEE 802.11-13/0230r0

Report

Recommendations for comment resolution

• Address all comments – be responsive to the technical issue wherever this is possible– Answer the comment made, not the comment you wish they’d made.

Answer all the points in the comment.

– The proposed change should relate to the comment directly, not be an excuse to fix something else.

• Minimize procedural resolutions, except near the “end”– Address the technical issues in preference to using a “scope” or “invalid

comment” resolution.

• Use neutral language – address the issue, not the commenter

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Technical aspects of comment resolution - 1• Use the correct resolution

– Accepted means “we agree with the comment and the proposed change”. It cannot be used if the proposed change is not sufficiently detailed that the editor and TG members know what the intended change is, e.g. if the commenter offers alternative solutions.

– Revised means “we agree with the comment, but we approved a change different from the commenter’s proposed change”

– Rejected means “we disagree with the comment, and made no changes”. Don’t use “Rejected” if the comment is addressed in another comment. Instead, repeat part of the resolution of that comment as appropriate. Don’t use “Rejected - Fixed in Draft x.y” as a resolution because it doesn’t say what you changed to resolve the comment.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Technical aspects of comment resolution - 2

• Avoid over-use of references to submissions– This forces the commenter to do work that you could have done.

– Copy and paste textual resolutions from submissions.

– Reserve references for material that cannot be represented in plain text, e.g. graphics and heavy use of insert/delete.

• Use URLS– Any references to external documents should be URLs, not 802.11

document numbers. This is particularly important in sponsor ballot, because

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Technical aspects of comment resolution - 3

• Avoid references to other comment resolutions– When we present unsatisfied comments to the EC (or RevCom

read them), they see only a subset of comments. When RevCom look at a comment resolution, they do not necessarily see the same comment ID as the TG works with.

– These both mean that references to comment IDs in resolutions are useless.

– Instead of referencing a comment resolution – copy it.

• For example:– Revised. At page 1234 line 56, change ‘nurgle’ to ‘flange’.

– Note to editor (this is the same as comment resolution for CID 321).

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Technical aspects of comment resolution – 4 – promises of future action

• Invalid Resolution: “Rejected. We will consider this comment in a later ballot/the next revision, but we ran out of time to discuss it now.”

• Avoid promises of future action. Any such promise is an immediate red flag to the EC / RevCom.

• A member of the TG who is in the sponsor ballot pool might offer to submit certain comments from the end of WG ballot into sponsor ballot; but if he does so, it is an individual act, not an official one.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Technical / General / Editorial comments

• Generally comments are resolved without regard to how the commenter classifies them, except:– Usually the editor gets to propose resolutions to most of the

editorial comments

– Editorial comments that are not part of a no vote might be resolved “This comment has been passed to the TG technical editor for consideration during preparation of a subsequent draft”

– Near the end of the sponsor ballot, editorial comments might be resolved with “This comment will be passed to the IEEE-SA editor for consideration during publication editing.”

• The last two are permissible exceptions to the “don’t promise future action” rule.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Examples of Unresponsive Comment Resolutions

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Responsive / unresponsive - 1

March 2013


Comment Proposed resolutionIn clause 10, coexistence with mode X devices has not been considered.

Add mandatory coexistence mechanism Y as described in document Z.

• Resolution: Reject. The TG considered the comment and did not agree with the proposed resolution.

• Analysis: Unresponsive. The response is generic and does not include the subject matter of the comment.

doc.: IEEE 802.11-13/0230r0

Report


• Resolution: The TG wasn’t sure whether to accept the comment or not and couldn’t agree on a change

• Analysis: Unresponsive. The response is wishy-washy and leaves the commenter with inadequate information on the TG’s position.

March 2013




doc.: IEEE 802.11-13/0230r0

Report


March 2013




• Resolution: Reject. Coexistence with mode X devices is not a requirement of the PAR. The TG considered the comment and did not agree with the proposed resolution on the basis that it added significant complexity with insufficient benefit.

• Analysis: Responsive. The response includes the subject matter of the comment, and provides concise reasons why the comment was rejected. It is not necessary for the resolution to provide extra detail on precisely how much complexity would be tolerable and how the TG members traded off complexity versus benefit.

doc.: IEEE 802.11-13/0230r0

Report

Types of Procedural Rejection

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Procedural Rejections – 1 (insufficient detail)

March 2013


Context:

The comment fails to locate and identify the issue. Fails to identify changes in sufficient detail so that “the specific wording of the changes … can be determined”.

Example resolution:

Disagree. The comment fails to identify a specific issue to be addressed. It fails to identify changes in sufficient detail so that the specific wording of the changes that will satisfy the commenter can be determined.

doc.: IEEE 802.11-13/0230r0

Report

Procedural Rejections – 2 (asking a question)

March 2013


The comment is asking a question Disagree. The comment fails to identify a specific issue to be addressed. It fails to identify changes in sufficient detail so that the specific wording of the changes that will satisfy the commenter can be determined. In reply to the commenter, the lower summation limit is zero because this is the starting index for variable I representing iteration over subcarriers.

doc.: IEEE 802.11-13/0230r0

Report

Procedural Rejections – 3 (big issue)

• Disagree. The comment fails to identify changes in sufficient detail so that the specific wording of the changes that will satisfy the commenter can be determined.

• Note, the CRC can (but is not required to) admit or accept the validity of any reported “big issue”. If it wants to leave the resolution as a straightforward reject, this comment can be used to later introduce changes to resolve the big issue. Note also that the CRC doesn’t need a valid comment to make any changes it wishes, and if it agrees that there is a big issue and some ballot cycles later determines how to resolve that issue, it is at liberty to make those changes.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Procedural Rejections – 4 (out of scope)

• Disagree. The comment is out of scope: i.e., it is not on changed text, text affected by changed text or text that is the target of an existing valid unsatisfied comment.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Procedural Rejections – 5 (can’t agree on a change)• Context: The TG cannot reach a consensus on any other

resolution. – The CRC have extensively debated the comment and there are different

views on how to respond to it. One or more other resolutions have been considered but no such resolution has received 75% approval.

– It should be recorded in the minutes what other solutions have been considered, together with the results of any straw polls or motions on those alternative solutions so that the commenter can understand that this solution was used as a last resort.

• Disagree. The CRC could not reach consensus on any change to the current text that would satisfy this comment.

• Use with caution. The TG chair may well have to stand up in front of the EC or RevCom and justify use of this resolution.

• You have been warned!

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Typical Ballot Lifecycle• Approval cycle:

– WG Ballot

– EC approval to proceed to sponsor ballot

– Sponsor Ballot

– EC approval to proceed to RevCom

– IEEE-SA RevCom and Standards Board approvals

• Permissive vs restrictive

• Solicit input from “no” voters

• Recirculate unchanged

• EC approval

• RevCom / IEEE-SASB approval

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Ballot Lifecycle - Permissive vs restrictive

• In the first ballot, all comments are in scope (although not all comments are necessary valid).

• Early in the process, the TG will attempt to satisfy comments regardless of whether they are valid

• As the draft nears the end of a sequence of recirculation ballots, the TG wants to proceed to the next stage in the process.

• At this point they are likely to become restrictive, rejecting comments on the basis of validity.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Ballot Lifecycle - Solicit input from “no” voters

• Comment resolution provides an opportunity for 2-way communication between the commenter and the TG.

• There is no reason to limit this to communicating via the official tools. The TG are encouraged to engage directly with commenters to understand better what they are saying in a comment, and to seek feedback on its possible resolution.

• The TG chair should seek feedback from unsatisfied comments from previous rounds so that the actual unsatisfied comments are identified for presentation to the EC.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Ballot Lifecycle - Recirculate unchanged• At the end of some recirculation ballot, unless the ballot

received no comments, the group will determine that there are reasons to validly reject all the comments received.

• In this case an unchanged draft is sent to the next (final) recirculation along with the reasons for these rejections

• Comments might be received from this final recirculation, but they are either a reiteration of an earlier comment (i.e., not new) or they are out of scope.

• Thus the process is guaranteed to terminate, should the TG so wish– i.e., no commenter can force the process to continue indefinitely

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Ballot Lifecycle - EC approval

• The approval of the EC is required to progress between WG ballot and Sponsor ballot, and to go to RevCom

• Approval is granted (either unconditional or conditional) at plenary sessions of 802 (March, July, Nov)

• The TG produces and approves a report on the status of the ballot and unsatisfied voters/comments to solicit this approval.

• In the case of conditional approval, an email on meeting the terms of the conditional approval is sent by the WG chair to the EC when the terms are met.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Ballot Lifecycle - RevCom / IEEE-SASB approval

• RevCom reviews material it retrieves from the MyBallot system in order to determine whether the process for sponsor ballot has been properly followed.

• They might review unsatisfied comments, and have visibility of all comment resolutions.

• RevCom makes a recommendation to the standards board (IEEE-SASB), which is responsible for the actual approval.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Difference between WG and Sponsor Ballot

• During WG comment resolution, the TG resolves comments– Motions can be made only during 802.11 sessions, except

– Once conditional approval has been obtained from the EC, a TG telecon can approve resolutions by motion

• During Sponsor ballot comment resolution, the WG chair delegates responsibility to a comment resolution committee (CRC) to resolve comments– Any 802.11 voter can join this CRC, which is practically

indistinguishable from a TG, except that it can vote during telecons.

March 2013


doc.: IEEE 802.11-13/0230r0

Report

Requirements on TG officers• Post timely spread sheets. All 802.11 members should

have essentially equal access to the comments and their proposed resolutions.

• Ensure all comments have a valid resolution

• Ensure resolutions are actioned in the draft. This is mainly the responsibility of the editor. – Nothing discombobulates commenters more than the group

approving a change than then not making that change.

• Chase unresponsive “no” voters

• Prepare reports to the EC

• (Tools, support and training are available from WG officers)

March 2013


doc.: IEEE 802.11-13/0230r0

Report

The end?

• If this tutorial informed you, but has no effect on your future behaviour, your attendance was a waste of time.

• What are you going to do to improve the quality of your comment resolutions?

• Call to action: Be critical of your own and others comment resolutions.– Spending time getting this right now should save you time in the

future.

March 2013


802.11 comment resolution – a tutorial

Documents