Latent Class Models for Analyzing Agreement

Introduction

Latent class models for agreement data have become increasingly popular. The premise of this approach is that cases in a population belong to two or more latent classes--"latent class" simply means that the class membership of a given case is not directly observed. A latent class may be thought of as a case subtype or "genotype."

A case's probability of being assigned a given rating level is assumed to depend on the case's latent class. For example, a medical population may consist of two latent classes: disease-negative (normal) cases and disease-positive cases. The probability that case is diagnosed "positive" by a rater or procedure depends on which latent class the case belongs to.

Analysis aims to estimate (1) the proportion of cases in each latent class, and (2) for each latent class, the probability that a member of that latent class will elicit each rating level from a given rater. When the latent class model fits, these parameters can be used to assess rater agreement, provide information about rating accuracy, or or be put to other practical uses.

Advantages of the latent class approach are as follows:

1. The model is relatively simple, reasonably plausible, and statistically testable.

2. It allows one to analyze agreement among more than two raters or procedures simultaneously.

3. It may permit inferences about the accuracy of ratings or diagnoses (e.g., their Sensitivity and Specificity) when a "gold standard" is unavailable (Walter & Irwig, 1988; Uebersax, 1988; Hui & Zhou, 1998).

4. Latent class models can be estimated with easily available software (Clogg, 1977; van de Pol, Langeheine & de Jong, 1989; Vermunt, 1998; Vermunt & Magdison, 2000).

5. One can use results to construct Bayesian decision rules for the optimal diagnosis or classification of cases using ratings by two or more raters or procedures (Uebersax & Grove, 1990).

Limitations of the approach are as follows:

1. With dichotomous data, the method generally requires that there be at least three raters or procedures.

2. The usual latent class model assumes that, within each latent class, the rating level assigned by one rater is statistically independent of the rating levels assigned by other raters. This assumption--"conditional independence" or "local independence" (independence of ratings, conditional on latent class)--can be unrealistic. For example, in medical applications, some disease-positive cases may have a stronger or more salient disease level; such cases are more likely to elicit a positive diagnosis by all raters, violating the conditional independence assumption. Simple extensions of the basic latent class model, however, relax this limiting assumption (see A Practical Guide to Local Dependence in Latent Class Models. Other methods for avoiding this limitation are discussed by Qu, Tan and Kutern (1996) and Uebersax and Grove (1993).

3. Special provisions are required for ordered-category ratings (Uebersax, 1993; Uebersax & Grove, 1993; Vermunt, 1998; Uebersax, 2000).

For a great deal more information on Latent Class Modeling, including FAQs, Software information, Links, and detailed References, visit the Latent Structure Analysis pages.

(Top of page)
(Back to Agreement Statistics)

Bibliography: Latent Class Models for Agreement

Where to Start
Latent Class Models in General
Latent Class Models for Agreement Data
Software

(Top of page)
(Back to Agreement Statistics)

Where to Start

Clogg CC. Latent class models. In: Arminger G, Clogg CC, Sobel ME (Eds.), Handbook of statistical modeling for social and behavioral sciences (Ch. 6; pp. 311-359). New York: Plenum, 1995.

Uebersax JS, Grove WM. Latent class analysis of diagnostic agreement. Statistics in Medicine, 1990, 9, 559-572.

(Top of Bibliography)

Latent Class Analysis in General

Clogg CC. Latent class models. In: Arminger G, Clogg CC, Sobel ME (Eds.), Handbook of statistical modeling for social and behavioral sciences (Ch. 6; pp. 311-359). New York: Plenum, 1995.

Goodman LA. Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika, 1974, 61, 215-231.

Heinen T. Latent class and discrete latent trait models: Similarities and differences. Thousand Oaks, California: Sage, 1996.

Lazarsfeld PF, Henry NW. Latent structure analysis. Boston: Houghton Mifflin, 1968.

McCutcheon AC. Latent class analysis. Beverly Hills: Sage Publications, 1987.

(Top of Bibliography)

Latent Class Models for Agreement Data

Dawid AP, Skene AM. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 1979, 28, 20-28.

Dillon WR, Mulani N. A probabilistic latent class model for assessing inter-judge reliability. Multivariate Behavioral Research, 1984, 19, 438-458.

Espeland MA, Handelman SL. Using latent class models to characterize and assess relative error in discrete measurements. Biometrics, 1989, 45, 587-599.

Hui SL, Zhou XH. Evaluation of diagnostic tests without gold standards. Statistical Methods in Medical Research, 1998, 7, 354-370.

Qu Y, Tan M, Kutner MH. Random effects models in latent class analysis for evaluating accuracy of diagnostic tests. Biometrics, 1996, 52, 797-810.

Uebersax JS. Validity inferences from interobserver agreement. Psychological Bulletin, 1988, 104, 405-416.

Uebersax JS. Statistical modeling of expert ratings on medical treatment appropriateness. Journal of the American Statistical Association, 1993, 88, 421-427.

Uebersax JS. Probit latent class analysis with dichotomous or ordered category measures: conditional independence/dependence models. Applied Psychological Measurement, 1999, 23, 283-297.

Uebersax JS, Grove WM. Latent class analysis of diagnostic agreement. Statistics in Medicine, 1990, 9, 559-572.

Uebersax JS, Grove WM. A latent trait finite mixture model for the analysis of rating agreement. Biometrics, 1993, 49, 823-835.

Walter SD, Irwig LM. Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. Journal of Clinical Epidemiology, 1988, 41, 923-937.

(Top of Bibliography)

Software for Latent Class Analysis

Clogg CC. Unrestricted and restricted maximum likelihood latent structure analysis: a manual for users. Working paper 1977-09, Pennsylvania State University, Population Issues Research Center.

van de Pol F, Langeheine R, de Jong W. PANMARK user manual. Netherlands Central Bureau of Statistics, Voorburg, The Netherlands, 1989.

Vermunt J. LEM users manual. Department of Methodology, Tilburg University, Tilburg, The Netherlands, 1998.

Vermunt JK, Magidson J. Latent GOLD User's Guide. Belmont, Mass.: Statistical Innovations, Inc., 2000.