Electronic Management of the Peer Review Process

G. Jason Mathews and Barry Jacobs
NASA/GSFC, Code 633, Greenbelt, MD 20771, USA

mathews@nssdc.gsfc.nasa.gov
bjacobs@nssdca.gsfc.nasa.gov

May 6-10, 1996 - Paris, France

What Makes up the Peer Review Process?

1. Call for papers announcement
2. Collect paper submissions from authors
3. Select pool of reviewers
4. Assign papers to reviewers
5. Collect reviews from reviewers
6. Make final selections
7. Notify authors of decision
8. Collect revised papers
9. Publish papers in conference proceedings

What is an EMS?

• A framework and methodology

• An organization of processes

• A Toolkit
  • Database client/server applications
  • CGI scripts
  • Oracle
  • IDL graphics software
  • Other utilities

EMS Overview

Data collection

• Entering data via a form (reviews, assignments, selections, etc):
  • Authentication
  • Validate fields
  • Process or reject operation
  • Build SQL statement
  • Call database application

• Submitting papers via ftp and follow up via Web form:
  • Anonymous upload
  • File format (zip, tar, gzip, other)
  • File integrity
    • Corrupted archives (e.g., file transferred in ASCII mode)
    • Corrupted files (bad GIF images)
  • File contents
    • HTML compliancy (HTML errors)
    • Bad links (case sensitive file systems and file names)
    • Single HTML document vs. multi-page document

EMS User Interface

Database Operations

• Multiple database operations
  • And search
  • Or Search
  • Key Search
  • Insert
  • Update
  • Delete
  • Clear

• Same form to query records used to fill in the data for display and data entry

• HTML form used as a template

• Database table and column information embedded within HTML form

Data Query

Searching for a record to browse a paper or enter a review

• Enter query key fields into blank form (e.g. search for papers dealing with VRML or JAVA)
• Submit query to search database
• Generate a list of records (e.g., 14 records found below)

Data Query/Entry

Searching for a record to browse a paper or enter a review

• Search via a blank form
• Select record from list
• Blank form filled in with values from database
• Perform additional operations on selected record (update, delete)

Report Tools

• Standard Report Tools
  • Canned reports (e.g., List all papers that need to be assigned to reviewers)
  • Generated via CGI script innovation
  • Embedded arguments (script, table, columns, layout, etc.) in URL

• Ad hoc Report Tools
  • Custom reports
    • Row reports (sample as data query form but no update operation)
    • Tabular reports (ASCII or HTML 3.0 output)
    • SQL reports (enter SQL statements to query database directly)
    • Graphics Reports (bar plots, pie charts, 3d surface plots, etc.)
  • Embedded arguments (script, table, columns, layout, etc.) in HTML form

Graphical Report Tools

• 2-D Plot options
  • Bar plot
  • Line plot
  • Pie chart

• 3-D Plot options
  • Surface Plot
  • Scatter Plot

Sample 2-D plot form for submissions table

Find trends in the conferences

WWW4 and WWW5 novelty scores for all reviews, respectively

Graphs to answer questions

Who is submitting papers to the conferences?

More graphs

How many reviewers are reviewing a paper?

Future Work

Conference Peer Review EMS

• Support next Web Conference WWW6

EMS in general

• Enhance security mechanism to role-based control at the database row-level

• Offer EMS toolkit as a commercial package (distributed by REI Systems)

Summary

• EMS represents an organization view of a complex process

• First Web-based approach for managing the complete peer review process

• All the subprocesses and processes are in terms of a hierarchy presented in HTML pages

• Development of the entire system required little or no programming
  • Generic database application tools
  • Database specific information embedded in the HTML forms
  • Common set of report tools generated directly from the database schema