Skip to main content

Criteria Elicitation

This page is under construction

Summary​

Criteria Elicitation is a process where a set of people (the participants in the process) learn what concept(s) a specific person (the protagonist1 of the process) has in mind when (s)he uses a particular term (the subject of the process). The process is complete when the participants can use the term in the same way as the protagonist, thereby showing that they now all have the same understanding of these concept(s).

The process assumes that it is run in a particular context, within which all participants are motivated to actually understand the term in the meaning of the protagonist. Such contexts might include a project in which they all participate, or any other collaboration of the participants in which they work to realize certain objectives, for which the actual understanding of that term is a relevant part.

Triggers​

There are many signals that may serve as a trigger to start this process.

One such signal is the determination that people (that work together) start quarreling about some term. Another signal could be the context in which a terminology is actually designed. [a reference to 'terminology design' is needed here] We expect that other signals may exist.

Whatever the signal, the process should only start after having ensured that all participants have a sufficient interest, and are sufficiently motivated to spend the time and effort to realize the intended result.

Activities​

Running the process consists of executing the following activities:

  1. Ask the protagonist to formulate the criterion that it uses to distinguish between what is, and what is not, an instance of the term. A criterion is anything that all other participants can evaluate, i.e. use to make a judgement about whether or not something satisfies that criterion.

  2. Ask a participant (which might be the protagonist) to come up with a use-case (situation) in which the criterion would be applicable, and then ask all participants to evaluate the criteria as they understand it, and share their judgements.

  3. While the judgements differ, allow (and help) the protagonist to improve the criteria, and repeat the evaluation of the improved criteria by all participants, until all judgements are the same.

  4. Repeat steps 2 and 3, until every participant states that (s)he is convinced that (s)he can make the same distinctions as the protagonist does in arbitrary use-cases that are relevant for the context in which the process is run.

Issues/Exceptions​

There are a few situations that require the process execution to be deferred for some time.

  1. The protagonist may formulate a criterion that is not readily evaluable by other participants. In some cases, this issue can be resolved by asking the protagonist to reformulate the criterion. However, the criterion may also (heavily) rely on terminology that the protagonist uses, but that the other participants do not yet understand. In such cases, the participants may need to learn other terms from that terminology, perhaps also how these terms relate to one another, in order to understand the term that is the subject of the process. It depends on the context in which the process is run, how the participants interact and work together, how to proceed (or terminate the process).

  2. anything else?

Tips​

Here are some tips to consider when running the process.

Be friendly and respectful towards the other participants. Everyone has its own truths, backgrounds, ideas, etc. The process intends that you learn what the others think (and they will learn what you think). So don't make judgements (other than the ones based on the evaluation of the protagonists criteria).

It may be helpful to choose a meaningless word or phrase that you can use during the process to refer to the concept(s) that the protagonist has in mind. This helps you and the other participants to stay focused on eliciting the meaning of the term as the protagonist intends, and prevents you from being distracted by the meaning that you typically associate the term with. Here are some possibilities: "Dwork", "Rilfel", "Quoth", "Mauwer", "Prakken". But you can of course invent your own.

Help the protagonist formulate the criterion that fits his/her meaning of the term. Often, the protagonist has a very good, yet unconscious idea of what the term means. Making this conscious is not a trivial task. A protagonist may be hesitant to come up with a criterion, because (s)he already feels that it isn't quite right. But it is an honest attempt. Coming up with use-cases and evaluating the criterion will provide new inspiration to improve the criterion. Start with 'easy' use-cases, and make them increasingly 'difficult' only after the easy ones

Example​

An account of how this process was run several times in a project is described in this real-life example

  1. We consider the protagonist of the process to also be one of its participants.↩