quality control of ieee conferences: ensuring excellence ...cn.ieee.org/files/mini poco bin...
TRANSCRIPT
Quality Control of IEEE Conferences: Ensuring Excellence and Integrity
IEEE China Mini Panel of Conference Organizers (POCO) 19 September 2015
Bin Zhao IEEE Technical Program Integrity Committee (TPIC)
Agenda
9/19/2015 2
IEEE Conferences
Issues in Conference Quality
Quality Control and Assurance
Recommended Practice
Q & A
• R10/China is very important to IEEE from many perspectives • IEEE wants to be the favorite place for R10/China’s
Electronic and Electrical Engineering Community: • to hold conferences or conventions • to publish technical papers
IEEE Conferences
9/19/2015 3
Bright minds share the latest research at IEEE sponsored and co-sponsored conferences around the world.
1,600 Annual Conferences
Research Collaboration Publications
IEEE Global Conference Overview In 2014 IEEE….
9/19/2015 4
With Universities and other not-for-profit organizations
Partnered with over 1000 non-IEEE entities
Sponsored 1635 conferences
805 financially sponsored
technically co-sponsored 830
in 90+ countries
Over 480,000 attendees
266K Financially sponsored conferences
Technically co-sponsored conferences 215K
Acquired 1510 conference proceedings
Conference Quality Issues
9/19/2015 5
“Pay-to-Publish” behavior – most authors are no-shows
Paper Stuffing – 1000 papers from conference
w/100 attendees
Proceedings Hijacking – unauthorized people
substituting content
Citation Stacking – numerous self-references in a
single paper
Sponsorship Shopping – mainly in TCS
Machine-generated papers
Content outside of conference scope or Organization technical scope
Plagiarized papers
Poor English / grammar
Potential Consequences
9/19/2015 6
Loss of IEEE’s Reputation Loss of IEEE Conference Publications’ Ability to be Indexed
9/19/2015 7
The IEEE Brand Promise Advancing Technology for Humanity
Maintain IEEE leadership as the trusted source of advancements and applications of technology
Create standards, innovative research venues, and a global technology forum
Provide opportunities to conquer technical challenges with imagination
Enable today’s innovators to improve tomorrow’s quality of life, as our tagline promises
9/19/2015 8
The IEEE – Who We Serve Global Reach
Academia, government and corporate organizations around the world rely on IEEE publications via our online database: the IEEE Xplore Digital Library. – Search >3,650,000 items
– IEEE is the most cited publisher in new technology patents
Root Causes and Driving Forces
9/19/2015 9
Need for Authors to Publish – Major force in parts of Asia
Monetary Profit – $200k revenue per pay-to-publish conference
Different Cultural Viewpoints
Inexperienced Conference Organizers or IEEE OUs
Decentralized, Trust-Based Model in IEEE
Problems mostly associated with “hotspots” – Confluence of needy authors, predatory organizers, and
naïve IEEE organizational units (OUs) – Mostly occurs in areas of the world experiencing rapid
growth and in TCS conferences
Corrective Actions
9/19/2015 10 Result of Cross-IEEE Partnership
New conference approval policies
Joint Volunteer / Staff Conference Quality Team – TPIC: Technical Program Integrity Committee
Dedicated support staff – Tracking, Analyzing and Following Up on Complaints
– Screening 10% of Incoming Xplore® Content
Communications to change behavior
Education program developed and launched
Operation Screening
9/19/2015 11
Issues and shortcomings in conference organization – Insufficient engagement by co-sponsoring IEEE OUs
– Inadequate resources dedicated to peer review
– Implication of a pay-to-publish/no-show business model
– False advertising and misrepresentation
– Partnering with co-sponsors whose non-profit status is questionable
– Very short time span between conference initialization request and the actual event sometimes four to six months or less normal practice is one to two years
Content Screening
9/19/2015 12
Conference content scope and quality issues – Papers well outside of the stated subject matter scope of the
conference
– Papers well outside of IEEE’s engineering/technology related subject areas
– Papers that are poorly or unintelligibly written
– Fraudulent content, e.g., machine-generated papers
– Papers that lack some or all of the basic elements commonly found in scientific research papers: Introductory question or hypothesis -> brief summary of related
previous scholarly work by others -> research methods -> results -> conclusions
Decision Process
9/19/2015 13
Third-Party Company Evaluates Every Paper from Conference, Searching for Suspect Papers – Out of scope – Poor writing quality
Volunteers Screen Subset of Papers, Then Decide – Check third-party evaluation – Examine other factors (review process, program, etc.)
Accept / Reject Entire Conference – Evaluating the Conference, not Individual Papers
Scope Definition
9/19/2015 14
Would This Paper Be Presented at an Seminar in an EECS Department at a Major University?
Aerospace
Bioengineering
Communication, Networking & Broadcasting
Components, Circuits, Devices & Systems
Computing & Processing (hardware/software)
Engineered Materials, Dielectrics & Plasma
Engineering Profession
Fields, Waves & Electromagnetics
General Topics for Engineers (Math, Science & Engineering)
Geoscience
Nuclear Engineering
Photonics & Electo-Optics
Power, Energy Industry Applications
Robotics & Control Systems
Signal Processing & Analysis
Transportation
IEEE Covers Many Areas of Technology
15 9/19/2015
RENEWABLE ENERGY SMART GRID
AEROSPACE COMMUNICATIONS
OPTICS
INFORMATION TECHNOLOGY IMAGING SEMICONDUCTORS
CIRCUITS BIOMEDICAL ENGINEERING
LTE WIRELESS BROADBAND ELECTRONICS
NANOTECHNOLOGY
CLOUD COMPUTING
More areas than just Electrical Engineering & Computer Science
Writing and Content Quality
9/19/2015 16
Main Criteria – Does the writing quality interfere with the ability to understand the technical content? – We do not expect or require perfect English!
Fraudulent content – Plagiarism – Machine-generated papers
Also Look for Basic Scientific Paper Content – Hypothesis – Previous work in the field – Methods / Apparatus – Results / Data – Discussion – Does the Data support the Hypothesis? – Conclusions
Current TPIC/Quality Focus
9/19/2015 17
Plagiarism Checking – Being required for all conference papers – Software will be free if IEEE holds copyright – Conference organizers must evaluate output
Continue Refine Scope Definition – Engineering Management – Life Sciences / Bio-Engineering – Materials Science
Online Training Modules – Make training easy – Modules in Chinese
Closely Engage with Sponsoring OUs – Well-defined approval & oversight processes
Key Factors Influencing Conference Quality
Sponsorship
Technical Program
Paper Quality
18 9/19/2015
TPIC/Conference Quality team are no longer solely in reactive/enforcement mode; we are in proactive/outreach mode
9/19/2015 19
Sponsorship & Engagement
Goal: Run High Quality Conferences!
Vet Conferences Carefully Before Granting TCS
Don’t Grant TCS and Then Ignore
Require and Maintain “Direct and Substantial Involvement”
Pay Attention to Inquiries from MCE Staff
Communicate with Your Chapters
Don’t Promise that Content will be in Xplore®
Report Suspicious Activity to [email protected]
9/19/2015 20
High Quality Technical Program
The portion of the conference that contains the presentation and discussion of research, generally in the form of: – Oral Presentations, Poster Sessions, Workshops and Tutorials Technical papers are collected and published as the proceedings of the event, and are often accessed in a digital repository, such as IEEE Xplore – Peer review required for publication of conference proceedings in IEEE Xplore Effective Technical Program Development – see a reference: https://www.ieee.org/conferences_events/conferences/organizers/planning_technical_program.html
9/19/2015 21
Characteristics in Technical Program of High-Quality
Technical program flow and content – Technical scope determined tracks and sessions – Allocating appropriate time for presentation and interaction – Assigning appropriate space
Supported by high-quality peer review – Identifying and organizing reviewers – Establishing review guidelines – Indicating review process in Call for papers – Integrated approach in managing Paper submissions Reviews Final program development
9/19/2015 22
Conference Organization Timeline
Establish TP Committee
10-12 Months
Recruit Reviewers Plan Key Dates
9-10
Months
Call for Papers
6-8 Months
Abstract or full paper
submission deadline
4-6 Months
Peer Review Process
3-4 months
Notice of acceptances
sent out (final paper) to authors
10-12 weeks
Graphic by John Tracy
9/19/2015 23
Peer Review
– Number of reviews per paper, and papers per reviewer We suggest 3 reviews per paper (2 is an absolute minimum) A maximum of 8 to 10 papers per reviewer
– Abstract or extended abstract vs. full papers (or two-
step approach) We recommend a deep-dive review of an extended abstract,
followed by a later, shorter review of the full paper
– Open, single-blind, and double-blind review We recommend single-blind review (the reviewers are
unknown to the authors)
9/19/2015 24
Review Criteria
Scope – Determine whether the paper is in the conference’s
stated field(s) of interest (and IEEE’s)
Quality – Is the paper well-written? – If it is an original research paper, did the author follow
the traditional IMRAD model (introduction, methods, results, and discussion)?
Originality – Is the author proposing a new idea or a novel approach?
9/19/2015 25
Acceptance and Rejection
You can require that authors make certain changes to papers before they are accepted
Communicate to authors the results of the review process and any next steps – Provide link to PDF eXpress – Requirements for presentation at the conference – Final paper submission dates
Consideration to use pre-screening – Improved possibility to pass quality check although it is
not guaranteed – Understand the issues and root causes more specifically
for future events and taking corrective actions
9/19/2015 26
Screening for Plagiarism: An Introduction to CrossCheck
CrossCheck compares submitted manuscripts to a database of published articles (including content behind publishers’ paywalls)
Crosscheck identifies papers that surpass a predetermined threshold of similarity to previously-published papers
IEEE recommends a 30% similarity threshold (i.e., any submitted paper with 30% or more similarity to previously published content will be flagged for a visual, side-by-side review with similar, previously-published papers)
9/19/2015 27
Using CrossCheck
Conference Technical Program Chairs can register for CrossCheck through the CrossCheck Portal ohttps://crosscheck.ieee.org/crosscheck/
Several peer review tools used widely within the IEEE conferences community are integrated with CrossCheck
9/19/2015 28
Challenges in Peer Review
Almost every conference faces these challenges: – Gathering enough high-quality papers by the submission
deadline – Finding enough experts to complete their reviews by the
deadline
Good conferences attract good papers and good reviewers!
9/19/2015 29
Non-Presented Papers “No-shows”
IEEE empowers conference organizers to limit the distribution of any paper that was not presented in person at the conference – Authors unable to attend the conference should arrange for a
qualified substitute – Organizers must notify authors in the conference call for papers
if they choose to implement IEEE’s non-presented papers policy
If a paper is included in the proceedings distributed on site and is later identified as a non-presented paper, the paper should still be included in the proceedings delivered to IEEE – Flag the paper when generating the Packing List so the paper
can be archived in our internal content management system
9/19/2015 30
Conference Quality Opportunities
– Main quality risks:
Papers outside the stated subject matter scope of the conference and IEEE’s fields of interest
Papers that are poorly or unintelligibly written
Content that indicates author(s) wrongdoing, e.g., machine-generated papers, plagiarism, excessive self-citations
– Serious engagement needed by co-sponsoring IEEE OUs
– Appropriate resources need to be dedicated to peer review
– Beware pay-to-publish business models
The IEEE Technical Program Integrity Committee (TPIC) and the IEEE Conference Quality team co-manage an ongoing quality assurance process
9/19/2015 31
Working Together with A Common Goal – Working closely with Conference Organizer Engagement and
Education team to develop online and in-person training materials (e.g., peer review best practices, stringent criteria for IEEE co-sponsorship)
– IEEE volunteers and staff continue to help conference organizers and leaders of IEEE organizational units whose conferences have been compromised by poor quality or inappropriate content
– Communication is the key to our success: we are engaging with our community to identify, develop and refine best practices
TPIC and Conference Quality team want to work with conference organizers / IEEE OUs to improve quality, maintain the health of Xplore, boost technical activities
9/19/2015 32
Q & A