Measuring inter-rater reliability of the sequenced performance inventory and reflective assessment of learning (SPIRAL).

Full Abstract

PURPOSE:
To measure the inter-rater reliability of the Sequenced Performance Inventory and Reflective Assessment of Learning (SPIRAL), a twenty-three item scoring rubric designed to assess first and second-year students' competencies such as "acquisition of knowledge," "peer teaching and communication skills," and "professional behavior" in a problem-based learning (PBL) curriculum at the University of North Dakota School of Medicine and Health Sciences.

METHOD:
In 2001, the authors constructed a 69-item multiple-choice questionnaire consisting of descriptions of "prototypical" (representing real students) PBL student performances. For each of the 23 SPIRAL items, the authors chose competency descriptions of an "emerging," a "developing," and an "advanced" student (69 prototypical students). These descriptions were obtained from narrative comments by PBL facilitators of real students in their classes in 2000-2001. Seventeen experienced facilitators (PBL tutors) were subsequently asked to rank the 69 prototypical students based on SPIRAL anchor points in the three competency levels. Kendall's coefficient of concordance was used to test inter-rater reliability. Modes were also determined to illustrate the extent to which the ratings of the 17 facilitators aligned with the ratings anticipated by the authors.

RESULTS:
Overall, inter-rater scoring for all items combined was considered to be reliable (Kendall's coefficient of concordance W =.75). Inter-rater scoring was also determined for each of the 23 SPIRAL items, with Ws ranging from.97 to.53. Facilitators rated students according to 55 of the 69 SPIRAL descriptors as the researchers anticipated.

CONCLUSIONS:
The scoring rubric is reliable. Further criterion-related validation study is warranted.

Learn Faster Today Improve your study skills

Author information

Author/s: Olson, Linda (L); Schieve, A Dean (AD); Ruit, Kenneth G (KG); Vari, Richard C (RC);

Affiliation: Office of Medical Education, University of North Dakota School of Medicine and Health Sciences, Grand Forks, 58202-9037, USA. liolson(-atsign-)medicine.nodak.edu

Journal and publication information

Publication Type: Evaluation Studies; Journal Article

Journal: Academic medicine : journal of the Association of American Medical Colleges (Acad Med), published in United States. (Language: eng)

Reference: 2003-Aug; vol 78 (issue 8) : pp 844-50

Dates: Created 2003/08/13; Completed 2003/09/17; Revised 2004/11/17;

PMID: 12915381, status: MEDLINE (last retrieval date: 11/6/2008)

Sourced from the National Library of Medicine. Abstract text and other information may be subject to copyright.

External Links for this article (including full text providers, if available):

Click Electronic Full-text Provider Links to see options for finding the electronic full text links to this article. Note there may be a subscription or fee required for access to the full text. See our FAQ for information on finding FREE full text articles.

This article may also be located in paper journal collections available in many libraries. Use the Journal and Publication Information above to find the full article.

MeSH headings (categories)

This article was linked to the MESH Headings shown below.

This article has not been indexed for related articles as yet, however you can still use the live related article search links below.

See 100+ related articles.