Skip to main content

Every award, verified and celebrated

Sign in

How Judges Score Entries: A View From the Panel

Judging · 10 min read ·

The judge's side of the table: criteria and weights, rubrics, calibration, conflicts of interest, discussion and what a fair panel does with close results.

Illustration: A judging table in a dark green room lit by brass lamps: stacks of folders, scoring sheets with grids, a brass bell and a ledger, in engraved-plaque typography

When the doors close on a judging session, entrants imagine a grand and mysterious process. In practice it is usually a group of busy people with a stack of papers, a scoring sheet and a deadline, trying to be fair. How they go about it determines whether the result deserves the trophy.

This guide takes you to the judge's side of the table. It is useful if you enter awards, because you will write better entries. It is useful if you run an awards programme, because you will judge more fairly. And it is useful if you read award announcements, because you will know what to ask.

What judges are asked to do

A judge's job is to assess entries against criteria and to give a score or a recommendation, without fear or favour. That sounds simple. It is demanding.

  • Read many entries carefully, often in a short window.
  • Apply the same standard to each.
  • Set aside personal preferences, relationships and first impressions.
  • Explain or defend their judgement in discussion.
  • Keep entries confidential.

Many judges are volunteers who give their time because they care about the field.

Criteria first

Fair judging starts before any entry is read, with clear criteria. Criteria are the standards against which entries are measured. They should be published in advance, so entrants know what is being asked, and judges can be held to them.

Typical features:

  • A small number, usually three to six.
  • A clear definition of each. What does "innovation" mean here? New to the world? New to the sector? New to the company?
  • Weights, if some matter more. "Impact counts for forty per cent."
  • A statement of evidence expected.

Without criteria, judging drifts into impressions, and impressions favour the best writers and best known names.

Rubrics

A rubric is a scoring guide that lists the criteria for a piece of work and describes the levels of performance for each. It is familiar from education, where it helps markers apply standards consistently.

In awards, a rubric might have a scale of one to five or one to ten for each criterion, with short descriptions of what each level looks like.

For "customer impact", for example:

  • Top level: substantial, evidenced improvements for many customers, with independent corroboration.
  • Middle level: clear benefits for some customers, with some evidence.
  • Low level: claimed benefits without evidence.

Descriptors turn a vague feeling into a comparison. They also let entrants understand what is valued.

Independent scoring

In many programmes, each judge scores entries privately before any discussion.

Why? Because group discussion can pull everyone towards the opinion of the loudest or most senior person. Private scoring first captures each judge's view. Then discussion tests it.

Typical steps:

  1. Judges read their allocated entries. Often each entry is read by several judges.
  2. They score each criterion, noting reasons.
  3. Scores are collected and combined, perhaps by averaging or adding with weights.
  4. Outliers are noted. A score far from the others may be an error or a different insight.
  5. The panel discusses the top entries and any with large differences.
  6. Scores may be adjusted after discussion, with reasons recorded.
  7. A final ranking is agreed.

Calibration

Different judges apply scales differently. One is generous, another stingy, a third rarely uses the extremes. Calibration reduces this.

In a calibration session, judges score a small set of sample entries independently, then compare. They discuss why they differed and agree how to apply the scale. After that, their scores are more comparable.

A programme that calibrates is taking consistency seriously.

Conflicts of interest

A conflict of interest arises when a person's private interests could affect their judgement in their role. In judging, conflicts include:

  • A judge's employer, client or competitor is an entrant.
  • A judge has a financial interest in an entrant.
  • A judge is a friend, relative or close colleague of an entrant.
  • A judge has previously advised or reviewed the entry.

A well-run programme:

  • Asks judges to declare relevant connections up front.
  • Excludes judges from scoring or discussing the affected entries.
  • Records declarations.
  • Considers perceived conflicts as well as real ones.
  • Has a procedure if a conflict emerges late.
  • Separates the commercial side (sales, sponsorship) from the judging side, so that nobody judging knows who has paid for what.

If an organiser cannot tell you how conflicts are handled, be cautious.

Bias and how it creeps in

Judges are human. Several biases are known to affect assessment.

Confirmation bias. The tendency to favour information that confirms what you already believe. A judge who forms an early view looks for evidence that supports it.

Halo effect. A strong impression in one area colours judgement in others. A beautifully written entry can seem more impressive than its evidence deserves.

Familiarity. Well-known names feel safer.

Order effects. Entries read first or last, or after a very strong or weak one, may be scored differently.

Fatigue. Judging the fortieth entry is harder than the fourth.

Similarity. People favour entries like their own experience.

Anchoring. An early score influences later ones.

Good programmes reduce these through clear criteria, multiple judges, randomised order, anonymised entries where possible, calibration and breaks.

Anonymity

Some programmes remove names from entries before judging, a practice known in other fields as blind assessment, so that judges assess the work and not the reputation. It is not always possible: in many awards the identity is integral. Where it is possible, it can help. Where not, other safeguards matter more.

Peer review in scholarly publishing offers a parallel: independent experts assess work, sometimes without knowing who wrote it, to improve fairness and quality. It is imperfect, and it is a model of careful assessment.

Reading evidence

Judges quickly learn to separate claims from evidence.

  • Claims are statements: "We transformed the sector."
  • Evidence is something that supports them: figures, dated results, independent sources, named customers, demonstrations.

Judges also check plausibility. Does the evidence match the claim? Is the period appropriate? Is the figure explained? Is there something missing?

Many programmes verify key claims, especially for winners. Advertising rules in many places say that objective claims must be backed by documentary evidence, and a programme that publishes unchecked claims in its own materials risks repeating errors.

The discussion

After individual scoring, many panels meet. Good discussion:

  • Starts with the criteria, not with preferences.
  • Lets each judge speak, quieter voices first if possible.
  • Examines big differences in scores.
  • Asks for evidence.
  • Records the reasons for decisions.
  • Stays courteous.
  • Keeps notes for feedback.

Poor discussion is dominated by a strong personality, wanders from the criteria or is cut short by time.

A chair who manages the process, keeps time and ensures each entry is treated fairly is invaluable.

Ties and close calls

When two entries score the same or nearly the same:

  • A stated tiebreak is used, such as the score on the most heavily weighted criterion.
  • The panel discusses and decides, with the reasons recorded.
  • A second round might be held, such as interviews or demonstrations.
  • Joint awards may be given, where the rules allow.

Whatever the method, it should be written down before the results are known.

Feedback

Judging produces valuable information for entrants. Some programmes provide scores or comments to unsuccessful entrants, either automatically or on request. If you can get it, do. It is one of the best ways to improve. If you run a programme, offering feedback is a mark of respect.

Confidentiality and integrity

Judges are bound by rules.

  • Keep entries confidential.
  • Do not discuss deliberations outside the panel.
  • Do not accept gifts or hospitality from entrants.
  • Do not contact entrants about their entries.
  • Report approaches by anyone trying to influence them.

Entrants must respect this too. Do not contact judges about your entry. Lobbying can disqualify you.

What entrants can learn

Knowing how judging works helps you enter.

  • Answer the criteria. Judges score against them.
  • Make evidence easy to find.
  • Write plainly. Tired judges reward clarity.
  • Be precise. Vague entries are scored low.
  • Be honest. Inflated claims make judges suspicious.
  • Make a short, memorable case, because discussion rewards entries that can be summarised in a sentence.
  • Do not rely on fame. Many programmes try to judge the work.

What programmes can learn

  • Publish criteria, weights and method.
  • Train and calibrate judges.
  • Declare and manage conflicts.
  • Score independently before discussing.
  • Record reasons.
  • Provide feedback.
  • Audit the process, and publish a summary.
  • Be willing to correct errors.

A short worked example

A programme has fifty entries in a category, four judges and three criteria with weights. Judges attend a calibration session where they score two sample entries and compare. Each entry is read by two judges, who score privately and write a sentence of reasoning per criterion. One judge declares that her company competes with an entrant; she is excluded from that entry. Scores are combined, and the top six go to a panel discussion, where the chair asks judges to speak to the evidence. Two entries tie on total score; the stated tiebreak is the score on the most heavily weighted criterion, which separates them.

The winner is announced with the criteria and a short citation. The programme sends all entrants a summary of how they scored by criterion. A process like this takes more work than a quick vote, and it produces a result that can be defended.

On this site

On this site, the trophy leaderboards rank cabinets by points, which is a different thing from a judging panel, and paid visibility, community votes and verified facts are kept separate and labelled. There is a community vote for Award of the Year, one vote per verified person, and votes can never be bought. The founders page introduces the people behind this place.

The short version

Fair judging starts with published criteria, weights and a rubric that describes each level. Judges score independently before discussing, calibrate on sample entries, declare and manage conflicts, guard against biases such as confirmation bias and the halo effect, separate claims from evidence, record reasons and use a stated tiebreak. Entrants should answer the criteria with clear, honest evidence, and programmes should provide feedback and publish their method. A trophy is worth what the process behind it is worth.

Questions and answers

How do judges score entries?
Typically against published criteria, using a scale and descriptors for each level, then combining scores, often followed by discussion.
Are judges biased?
Everyone has biases. Good programmes reduce them with clear criteria, independent scoring, calibration, declared conflicts and diverse panels.
What is calibration?
A step in which judges score a few sample entries together and compare, so that they apply the scale in the same way.
What happens when scores are tied?
A fair programme has a stated tiebreak, such as discussion, extra criteria or a second round.
Can an entrant contact a judge?
Not about their entry. Contact during judging is usually forbidden and can disqualify an entry.

Sources

Need help?