The Intermediate Notator Agreement is a measure of how two (or more) annotators can take the same rating for a given category. There are actually two ways to calculate the agreement between the annotators. The first approach is nothing more than a percentage of overlapping choices between the annotators. This approach is a bit biased, because it is perhaps a pure chance that there is a high horse. In fact, this could be the case if there are only a very limited number of category levels (only yes versus no, or so), so the chance of having the same remark is already 1 in 2. It is also possible that the majority of observations belong to one of the levels of the category, so that the horses at first sight are already potentially high. For my master`s degree, I worked with 44 undergraduate students, divided into 11 groups. Each individual has 100 unique tweets and 50 tweets that overlap with comments that the other three members of the group have also referred to with comments. This led to 50 tweets annotated from four different annotators and 400 tweets from a single annotator. Another approach to concordance (useful when there are only two advisors and the scale is continuous) is to calculate the differences between the observations of the two advisors. The average of these differences is called Bias and the reference interval (average ± 1.96 × standard deviation) is called the compliance limit.
The limitations of the agreement provide an overview of how random variations can influence evaluations. This story discusses best practices in annotations and examines the ILO`s two metrics for qualitative notes: Cohen and Fleiss` Kappa. Cohen kappa is calculated between a pair of annotators and Fleiss` Kappa on a group of several annotators. Now you can run the code below to calculate the inter-annotator chord. Note how we first create a data frame with two columns, one for each annotator. Pearson`s “R-Displaystyle,” Kendall format or Spearman`s “Displaystyle” can measure the pair correlation between advisors using an orderly scale. Pearson believes that the scale of evaluation is continuous; Kendall and Spearman`s statistics only assume it`s ordinal. If more than two clicks are observed, an average match level for the group can be calculated as the average value of the R-Displaystyle r values, or “Displaystyle” of any pair of debtors.