1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
<meta http-equiv="content-type" content="text/html;charset=iso-8859-1">
<title>BOCA Manual for the judges</title>
</head>
<body bgcolor="white" link="blue" vlink="blue">
<font size="5"><b>BOCA Online Contest Administrator</b></font>
<p><b>BOCA Reference Manual for judges<font size="5"> - </font>
version October/2011 (BOCA 1.4.1+)</b>
</p>
<p><font size="1">Copyright (c) 2006-2011 Cassio P. de Campos (cassio@ime.usp.br).<br>
Permission is granted to copy, distribute and/or modify this document under
the terms of the GNU Free Documentation License, Version 1.2 or any later
version published by the Free Software Foundation; with no Invariant Sections,
no Front-Cover Texts, and no Back-Cover Texts. A copy of the license may be
found in <a href="http://www.gnu.org/licenses/">http://www.gnu.org/licenses/</a></font></p>
<p>BOCA is a software created to control a contest with the <i>ACM International Collegiate
Programming Contest</i> rules. It has been developed in PHP and the interaction between judges
and the system is done through a <i>web browser</i>. In the following we describe
the set of available features for a judge in the system.</p>
<p>It is assumed here that the judge has already logged in the system with their
<i>username</i> and <i>password</i> in a windows of their browser. Tthe <i>URL</i>
to access BOCA will depend on the setup of your server. Usually this URL ends with
<i>/boca/</i>, where a simple authentication form will show up.
<p>
After a successful login, the web page contains the judge identification in the upper-left
corner. In the upper-right corner, the clock of the contest is shown, indicating if it has
started or not, if it has already ended, stopped, or in progress, in which case the number of
minutes to go are displayed. Below that, there is a set of options in a vertical menu, namely
<i>Runs, Score, Clarifications, History, As Team, Options e Logout.</i>
There is still an extra option named <i>Chief</i> which only appear to the chief judge, who
has to be appointed in the admin's interface (with in the <i>Site</i> options).
</p>
<p><font size="4"><b>Runs</b></font></p>
<p> In this area, the judge might visualize the runs that are still to be judged and shall be
taken care of. In order to process a run, the judge has to click in its number (eventually a
message might pop up here indicating the run was already taken; this happens if another judge
got the run before you). After clicking on the run to judge, a new set of data appear about it:
<ol>
<li><b>Site</b>: site number of the run.</li>
<li><b>Number</b>: run number.</li>
<li><b>Time</b>: minutes from the start of the competition until this submission.</li>
<li><b>Problem</b> <i>X</i>: <i>X</i> is the name of problem. There are links
to download the input and output files (click on the names), or simply to visualize them in the browser (click on view).
These links are mostly used when the <i>autojudge</i> is turned off or is not running properly for a given problem/language.
Instead, if the <i>autojudge</i> feature is working correctly, then the judge can directly to go the links in the bottom part of
the page in order to see the expected output together with the output generated by the team.
</li>
<li><b>Language</b> <i>Y</i>: <i>Y</i> is the language chosen by the team. The scripts that are used to
compile and execute the submission with this language are available through links.
When using the <i>autojudge</i>, note that is not necessary to use this links, because the results are
already shown in the bottom of the page.
</li>
<li><b>Source code</b>: here it is possible to <i>view</i> or download the submitted file. An important
task is to check if the name and extension of this file are correct with respect to the chosen problem, language,
and the specification in the booklet of problem descriptions.
</li>
<li><b>Answer</b>: the judge has to choose, among the options available in the system (which were configured by the admin),
the correct answer to be sent to the team.
</li>
<li><b>Autojudging answer</b>: here it is presented the suggestion from the <i>autojudge</i> about this submission.
Usually the <i>autojudge</i> properly identify the answer that has to be sent to the team. However, it may
fail to do so. For example, the <i>autojudge</i> might indicate a <b>wrong answer</b>, because the
<i>diff</i> procedure identified the output and expected output to be different, while the only mistake
happened in the punctuation/accent of a letter (this is usually a case of <b>presentation error</b>, but
may vary according to the contest). Other issues might happen, and the role of the judge is to check everything
for eventual mistakes.
</li>
<li><b>Autojudged by</b>: indicates the computer that has acts as <i>autojudge</i>. Usually this is not
much relevant for judging a submission.
</li>
<li><b>Standard output</b>: links for downloading and visualizing the output generated by the team's code are
available. This is the output generated by the team, which is to be compared with the expected output of
the given problem. It might also be possible to see some error, in case it happened.
</li>
<li><b>Standard error</b>: links for downloading and visualizing the standard error output generated by the team's code.
When a error happens (including compilation or runtime errors, but not restricted to them), this is the most probable file to identify it.
The judge must always check the content of this file. In the end of it, the judge also finds the output of the <i>diff</i> command
that has compared the team's output and the expected one (to facilitate the visual inspection).
</li>
</ol>
<p>By pressing the button named <b>Judge</b>, the judge submits their veredict about the <i>run</i>, and then
has no access to the this <i>run</i> anymore. It is also possible to give up judging the run by clicking
on <b>Cancel</b>, which will send the run back to the pool (and it will be eventually judged by another judge).
</p>
<p>In the tab <i>Runs</i>, the judge can see all the submissions that are yet to be judged. The colors
represent their status, meaning that they are already been judged by others, or that they are waiting
you to judge them (red color). You must act with respect to those in red, as only you can judge them (or
send them back to the pool).
</p>
<p></p>
<p><font size="4"><b>Chief</b></font></p>
<p>
Besides acting as a normal judge, there is a designated judge with access to the tab <i>Chief</i>. This tab
is used for the chief judge to resolve disputes regarding submissions that received different answers by different
judges (they are shown in red). The prodecure is equivalent to that of a normal judge, but the decision made in the tab of the chief is
final. The idea is to have the chief judge acting as a tie-breaker between the distinct answers that were assigned to
a <i>run</i>. Hence, if the chief judge is acting as a normal judge (and they may do so), it is recommended to use
the tab <i>Runs</i> instead, leaving the tab <i>Chief</i> only for resolving issues. (In the very special case where there
is a single judge in the competition, then this judge has to be designated as chief and shall use the tab <i>Chief</i>
to judge the runs, otherwise they will never receive the judgement of another judge and thus will not be sent to teams.)
In this same tab, the chief judge is able to ask the <i>autojudge</i> to be re-executed for some submissions (selected by
the boxes besides each of them) or to completely re-open the submission for judging again, which implies in a new round
of autojudging and judging by actual judges. The <b>team does not become aware</b> of this rejudging unless the new final
result for the run has to be changed (with respect to the first round of judgement).
</p>
<p></p>
<p></p>
<p><font size="4"><b>Score</b></font></p>
<p>In this tab the judge can see the scoreboard of the competition. It has to be noted that the scoreboard available
for the judges is complete and not subject to the freezing of the final part of the contest. Hence, the judge is
expected to keep the scoreboard in secret. In case the scoreboard is consolidated among different sites, the final part
of the other sites is not shown (*this fact is for technical reasons, to be discussed later*).
</p>
<p><font size="4"><b>Clarifications</b></font></p>
<p>This tab allows the judge to answer <i>clarifications</i> submitted by team regarding
a specific problem or any general aspect of the contest.
In order to reply to a <i>clarification</i>, the judge must first <b>click on the clarification number</b>.
The box available in the bottom part is meant for creating a new clarification, as judges are allowed to do
so. The new clarification will only be meaningful if the judge that gets it to answer selected the reply to all
option (explained in the next paragraph). This is useful for the judges to send a general information to the teams.
</p>
<p>After clicking on the clarification number, the judge sees information about it, such as
site number, clarification number, time in minutes from the start of the competition, and the
problem to which the clarification regards. There are two text boxes: one with the question (on top)
and one to be filled with the reply. Finally, there is a selection box to indicate if the reply
should be sent to all the teams or just the the team that posted the question.
</p>
<p>The system also allows the judge to select the <u><i>No response</i></u> button. In general
this is used for questions that are already stated in the booklet of problem descriptions, or
that has already been replied, or even that should not be answered at all (for example, about
the timelimit of a problem, or about some sensitive information of the input/output).
</p>
<p><font size="4"><b>History</b></font></p>
<p>This tab shows the history of <i>Clarifications</i> and
<i>Runs</i> that were processed by the judge so far.
</p>
<p><font size="4"><b>Options</b></font></p>
<p>This tab shows the information of the judge, such as <i>Username,
User</i> <i>full name </i>,<i> User description</i>. It is possible
for the judge to update their password, although this is not necessary neither
recommended. Instead it is recommended that the admin who created the users already
specifies secret and safe password for all users.
</p>
<br>
</p>
<p><font size="4"><i><b>Logout</b></i></font></p>
<p>Button to log out from the judge interface.</p>
<p><font size="4"><b>Important hints for judges</b></font></p>
<p> While judging a submission, it is necessary to be very careful. Even if it is possible
to alter the judgement of a run afterwards, this has to be avoided as much as possible, because
the team might suffer an undesired situation.
</p>
<p>Every time a judge is about to answer a problem for its first time (that is, the problem has not
been submitted before, or has not received yet any YESes), when possible it is interesting to have other judges
also checking the submission, which should be analyzed with even greater care. Sometimes this is the
moment when an unfortunate issue with the <i>autojudge</i> or with inputs or outputs is discovered
(obviously nobody wants it to happen, but it might).
</p>
<p>In order to reply to <i>clarifications</i>, it is important that the judge has read and understood
the question and the problem to which it concerns. When possible, it is interesting to have the opinion
of the person or group that created the problem description, its input and output. After that,
it is necessary to think if such reply has to be sent only
to this user or to all teams in the contest. If it is a relevant issue and the contest is also offered in
other sites, then the judge shall find a way to contact other sites about the clarification.
Some examples of clarifications that are not disclosured (that is, are replied with a <i>no response</i>)
are
<i>What is the timelimit for this problem?</i>,
<i>Given this input, which is the correct output?</i>,
questions that are already explained in the problem description,
etc.
In case the judge is not certain about it, they must contact the chief judge of the site or the
chief judge of the contest for help.
</p>
<p><font size="4"><b>About BOCA and this document</b></font></p>
<p>BOCA System and this document have been created by Cassio Polpo de Campos and can be found at
<a href="http://www.ime.usp.br/~cassio/boca/">http://www.ime.usp.br/~cassio/boca/</a>.</p>
<hr>
<p>
<a href="http://validator.w3.org/check?uri=referer">
<img src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!" border="0" height="31" width="88"></a>
</p></body></html>
|
