Previous Up Next

3  Phases and events

The organization of the conference takes a limited amount of time, which is fixed at the beginning. We distinguish the following main phases:
  1. assembly
  2. paper submission
  3. reviewing, consensus finding about the selection of the submission
  4. registration
The phases themselves are ordered linearly, the exact duration or distance of the single events should be adaptable. In the following, we discuss the purpose of the phases the outcome of each. Most (but not all) phases are separated by events.

3.1  Event: Configuration

Configuration serves a rather simple goal, namely the preparation and setup for the rest of the phases. After installation, its includes to fix basic data of the conference, which means it's name, it acronym, the date (begin and end) and the location of the conference. Furthermore, the chairs of the program committee are (probably) known.

3.2  Phase: Assembly

The assembly is done by the program chairs. The result of the assembly phase is the program committee, i.e., the collection of experts or individuals that will, in later phases, decide about the program.

The program committee is not self-assembled. It is the task of the program committee to invite members via email.

Invitees are free to decide, whether they wish to participate, which means they should actively acknowledge participation or they should reject. Before they acknowledge, they are called program committee candidates, or candidates for short. By default, candidates are not members unless they positively acknowledge participation. They can register with the committee by using a web-interface. By acknowledging, they also fill in further relevant personal data, such as affiliation (=professional address), full name, preferred email address, home page.

3.3  Event/document: Call for papers

Once the program committee is fixed the conference is ``advertised'', i.e., authors are invited to contribute to the event. The document, basically an url i.e., the ``web-page'' which contains this advertisement is called the call for papers. It must announce all information about the conference relevant for authors, which
  1. basic conference data
  2. important dates for authors
  3. reference to the submission page
  4. short description as free form text
  5. names, affiliations and perhaps other information about the organizing people.

3.4  Phase: submission

The effect of the announcement is that authors decide to contribute to the conference. The contributions are called papers. The paper is uploaded by an author onto the appropriate web page, the act of doing so is the submission.

A paper has at least one author, but may have more than one. One author may have more than one paper. One author of a paper is considered as the main author, known as the corresponding author. This is the author the organization of the conference deals with.

An author can decide to retract a paper. In this case, no older versions (if any) of the paper are restored, but the paper is removed completely.

3.5  Event: submission deadline

The submission phase is terminated by the submission deadline, a pre-announced time after which no submission is possible. Until that time an author can submit many versions of the same paper. Only the last one before the deadline counts, i.e., each later version overwrites the earlier one.1

It is possible for an author to remove a paper even after the deadline (see Section 3.4).

3.6  Phase: Reviewing

Reviewing is the process of selecting from the submission a set of papers. The selection is done by the program committee in a joint effort using the web server. The (informal and conflicting) goals are:
quality of paper selection:
Since in general there are more papers than time on the conference, the best papers are to be selected.
load balance for reviewers:
In order to form an opinion about a paper, a reviewer must read and understand it. This workload must be distributed in a fair manner for the members.
load balance for papers:
Each paper gets (in general) more than one reviewer, to make the decision less random. Each paper should get an equal share of the reviewing task.
quality of reviewer selection:
Each reviewer gets the papers that he wants to and/or he is the best expert for.
The above specification obviously is informal and unprecise as it allows a number of interpretations. We do not fix an exact specification here, because there are probably many plausible solutions. Instead we discuss aspects of the mentioned goals. In the specification task, we would like more explicit proposals to solve this problem.

3.6.1  Assignment of papers

One task is the assignment of papers to reviewers. It is to be expected that there are more papers than reviewers, and furthermore one should cater for the case that each paper gets more than one reviewer. Preferably, and unlike the selection of the papers, the assignment is done automatically, i.e., without general discussion.

Furthermore, the assignment should be ``fair'' wrt. the reviewers and wrt. the papers, in that the load is equally shared. An easy and not very useful solution would be to make a random assignment, under the side condition of approximate load balance. The disadvantage is that, in general, the members of the committee have slightly different fields of expertise, and preferably a member evaluates papers in a field he is a strong expert of.
Assignment by topic
Papers are classified according to a finite list of topics. The topics are predefined for the conference, and the author must pick those he feels his paper fits in. He might choose more than just one topic. Also each reviewer, beforehand, chooses a number of topics which he prefers to read papers from. Once the papers are in, the software tries to take the preferences of the reviewers into account, but of course still maintaining load balance concerning the numbers of papers per reviews and the numbers of reviewers per paper
Assignment by paper
This approach does not rely on predefined topics.2 Each referee shortly looks at the list of papers and declares preferences (or dislikes) according to some schema. It this might be very simple like ``I want 2, 17, and 42''. Also, it should be possible to state: ``I cannot review this paper.''.3 Again in this scheme, the selection mechanism should take the choices into account, but adhering to the side condition of balance. In other words: if someone only picks one paper, it does not mean he will get only one. If 15 people find paper 76 very interesting, it does not mean that paper 76 gets 15 reviewers.
One can imagine to combine those approaches, or to make it a chooseable alternative.

3.6.2  Selection of papers

In general there are more papers than there's time, so the intention is, of course, to pick the best of them. To talk about finding the ``best papers'' is misleading, though, because this uses the idealistic assumption that there are best papers and one just does not know yet which ones they are. On the other hand: even if it is more than questionable whether there is a globally and universal quality scale to be applicable to the papers, it does not mean that some papers are better than others, in the sense that most everyone would agree on that. The task is to come fast and efficiently to an agreement about this issue.

Let's assume two fundamentalistic approaches, which sheds light about the range of possibilities. Both sketched approaches are in practice not very useful and should be avoided. In order to talk about the best papers, one obviously assumes an (imaginary) linear order which needs to be determined by consensus and now the question is, how to reach this order.
Discuss everything
One standpoint is: all participants discuss all papers in a free-form manner until all agree on some order, and this fixes the best papers. This solution is impractical: A rational agreement, i.e., an agreement based on common understanding, would require that all reviewers read all papers (which one wants to avoid ...). And even if all papers are read and discussed by all committee members, to reach at a common order lead to endless dispute.
Discuss nothing
The opposite standpoint is: There is no discussion at all. Each reviewer gives the paper(s) he reviews a numerical value, say a mark. At the end the marks for each paper are averaged,4 the results are ordered linearly, and then the best are chosen.5 That is the most efficient solution but it might easily lead to bad decisions. In general, there is more than one reviewer per paper but there are in most cases not more than 4, and this makes the mean value of ratings rather random.

As perhaps from the discussion, there will not be a clean, mathematically optimal solution for the problem; basically the decision finding requires human intelligence and social interaction. The trick will be to assist in this social process, to make it more efficient than free discussion but more rational than random selection.

Possible states
Next we discuss which general states a paper can have during the reviewing phase. Ultimately, the judgment for each paper will be: ``accepted'' or ``rejected'', which is when all the papers are decided. A proposal for a schema of states could be:
status ::= decided | undecided
decided ::= accepted | rejected
undecided ::= unreviewed | unclear
unclear ::= conflict | inconclusive

There, a paper is yet undecided if it's not yet reviewed or the situation is unclear. Two causes are that there is a serious disagreement about the quality. For instance if one reviewer thinks the is very good in one category, and another one says it's very bad in the same category, this is an indication that one better looks at this point again. One could distinguish from that a situation where a paper is in the ``so-la-la''-range. In general, most submissions are in the middle-field. In this case there might not be enough statistical evidence to distinguish between two contributions with slightly different ratings.

Decision herding
The core of a solution is to focus the process. Certain discussion is unavoidable/wanted, but the participants should focus (or rather helped to be able to focus) on the right, i.e., discussion-worthy things. Since the goal of the discussion is to find an agreement, discussion-worthy things, in first approximation, are those which are not yet decided.

Basically, the ones taking part in the discussion, must be assisted to get a good overview of the status of the debate and what things profitably to do next. This includes some form of visualization (which might be as simple as a table) of relevant information. Relevant information could include:
per reviewer:
A reviewer will (if not ``forced'' otherwise) concentrate on ``his'' papers perhaps his reviews. So he should be presented ``his'' part of the task first. If not restricted, he might of course look also at other parts/aspects of the information.
``executive info'':
short, high-level overview over the status and progress (how many papers are decided, how many still to be discussed.)
Indication about missed deadlines (someone has not yet send his reports or similar)
chosen focus:
Some papers are chosen (for instance by the chair) to be discussed next.
Of course, the access to the information must obey the restrictions concerning the various user groups mentioned in Section 4.1.

As said before, the reviewers study the paper to come to an opinion about the quality of the contribution. In general one does not wish (only) a uniform single numerical value, but a (reasoned) rating in various categories. Those categories could include
overal rating:
A single numerical value which expresses the overall quality of the paper, taking all aspects into account
how new is the result/content?
Is the technical content sound or are there serious errors in the argumentation/proofs/results ...
how good does the paper fit into the theme of the conference
How well is the paper written? How sloppy is it? Is the English (or German ...) ok?
How confident is the reviewer about his own opinions? This depends on whether he understood most of it, whether he considers himself an expert in the topic etc.
The list should be adaptable per conference, but the above could be taken as default.

3.7  Event: Notification

Notification is the event which informs the authors about the final decision of the reviewing process. There are only two possible outcomes, namely yes or no for each paper. Besides the binary decision, the author is informed about the ``opinion'' concerning his submission. Concerning what information the authors are allowed to see, cf. also Section 4.

3.8  Event/document: Call for participation

As the call for papers (cf. Section 3.3), the call for participation is basically an advertisement. This time the addressees are not the potential authors, but potential participants of the conference itself. The call for participation contains similar information about the conference, but as additional information of course the program, i.e., the list of accepted papers with authors etc.

3.9  Phase: registration

This phase is characterized by the interaction of participants of the conference with the tool. Users can register with the conference, i.e., announce their participation. Again this will be done via some interface. The registration should be acknowledged.

The participant provides the usual personal information (name, title, affiliation). Furthermore, he is offered a number of options he must choose from: Furthermore, if a user registers before a predefined deadline (``early registration''), the fee is reduced.

last generated October 26, 2004 (ŠPublic License)
Previous Up Next