Taxonomy of Thorendal

June 22, 2005

Binary relevance meassuring in today’s information management systems

by Thorendal

Within the LIS sector (Library and Information Science) there excits two concepts that everyone knows: Precistion and recall. They are interesting to have a closer look at because they are very traditional meassurement within the domain and at least earlier on very used and respected perameters.

The Cranfield II Studies
These parameters were developed through some expiriments called the Cranfield II studies back in the early 50′ies in the UK. In these studies there was a distinction between two kinds of relevance: ‘Stated relevance’ and ‘User relevance’. It is obvious that the difference is that the former is predefined and the latter is based on the users assesement - and therefor subjective - of relevance (Ellis 1990, p. 15-16).

In the Cranfield II Studies outer paramenters such as context, user level e.g., have not been taken into account. This has of course been strongly criticized by all theory within this domain (Harter, 1996, p. 45).
The relevance in the Cranfield II studies looks at relevance as either a document is relevant and it is retrieved or it is not relevant and therefor it is not retrieved (Bruhns, 1998, p.17). Pretty basic.

The question is if they are adequate parameters for the meassurement of relevance in intellegent information retrieval systems nowadays? And are these perameters called precision and recall?

There is an ongoing discussion on how many aspects relevance as concept has. As the Cranfield II Studies operates with a binary concept of relevance it can be stated that it is way too narrow. On the other hand if the judgement relevant is rather broad it can cover all the grey areas of what is really relevant to the user or the search.
The most normalt thing nowadays is though to have more than two levels for the concept of relevance.

Precesion is defined as:
The ratio between the number of relevant and retrieved documents and the number og retrieved documents.

Recall is defines as:
The ration between the number of relevant and retrieved documents and the total number of relevant documents in the system - retrived or not. This meassure provides that the total number of relevant documents in the system for the specific search is known.

The somewhat conclusion must be then, that if a system can be trained sufficiently and is intelligent enough the precision and recall meassures should be accepetable for meassurering binary relevance - but then a new question emerges: Is anyone interested in a relative narrow or very broard definition of ‘relevant’?

References:
Bruhns, 1998,
Ellis, P. 1990,
Harter, 1996,

Filed under Taxonomy at 10:46 pm

2 Responses to “Binary relevance meassuring in today’s information management systems”

  1. Bear_One wrote:

    Come on - there gotta b more interestinng things in the world than this (yawn) :-)

  2. Thorendal wrote:

    Well well I happen to think it is interesting, but I do understand your lack of excitement though!

Leave a Comment

authimage

To customize the avatar that appears by your comment, visit Gravatar.com. The trackback URL for this post is here.