This article originally appeared in The Bar Examiner print edition, September 2017 (Vol. 86, No. 3), pp 20–25.

In its commitment to assist jurisdictions in grading the written portion of the bar exam, NCBE prepares thorough grading materials for the Multistate Essay Examination and the Multistate Performance Test, which effectively guide graders through the grading process. In addition, NCBE holds an MEE/MPT Grading Workshop at its Madison, Wisconsin, headquarters the weekend following the bar exam—a one-day event that begins with a presentation by NCBE staff on grading fundamentals, followed by dedicated hands-on grading sessions for each MEE and MPT item. The grading sessions are led by drafting committee members who are familiar with the questions, grading materials, and content areas and who are experienced Grading Workshop facilitators.

During the Grading Workshop, participants read, grade, and discuss actual examinee answers from jurisdictions across the country. The Grading Workshop prepares graders for grading their own jurisdictions’ papers by familiarizing them with the questions and grading materials, identifying answer trends, refining grading weights, and discussing with other graders how to resolve grading issues. Participants also get a head start on calibration—the development of coherent and identifiable grading judgments so that rank-ordering is consistent throughout the grading process as well as across multiple graders. Discussion at the Workshop focuses additionally on assessing the quality of the writing in the examinee papers.

Participation in NCBE’s Grading Workshop has expanded over the years, with hundreds of graders representing most MEE and MPT jurisdictions participating in the Grading Workshop in person, by conference call, or via on-demand streaming following the workshop. The recent July 2017 Grading Workshop set a record for all three categories of participants, and we took the opportunity to prepare this photo essay to offer a look behind the scenes at staff preparation for the Workshop and at what occurs during the Workshop’s hands-on sessions.

The titles of NCBE staff featured in this photo essay are as follows:

  • Judith Gundersen, NCBE President and CEO (holding the title of Director of Test Operations at the time of the Workshop)
  • Joanne Kane, Ph.D., Associate Director of Testing
  • Sonja Olson, MEE/MPT Associate Program Director
  • Andrew Mroch, Ph.D., Research Psychometrician
  • Ellen Embertson-Merrill, Educational Programming Coordinator
  • Mike Petterson, Executive Assistant to the COO
  • Ingrid Jorgenson, Test Production Coordinator
  • Gina Bailey, Senior Test Associate
  • Lu Riese, Senior Test Associate
  • Kasey Brinkman, Test Associate
  • Katie Chu, Test Associate

Preparation for the Grading Workshop

Registration for the Grading Workshop is handled by NCBE’s Educational Programming Coordinator, who sends confirmations and instructions to registrants specific to their type of participation—in person, by conference call, or via on-demand video—and arranges hotel logistics and prepares registration materials for in-person attendees.

Photo of Ellen Embertson filing papers

Ellen Embertson-Merrill

  • NCBE Test Operations Department staff members contact jurisdictions to request that they send in a selection of examinee answers after test administration to be used in the Workshop; those answers are then sorted, redacted for identifying information, and approved by NCBE’s Director of Test Operations and MEE/MPT Associate Program Director for use in Examinee Answer Books for Workshop participants.
  • Examinee Answer Books—which include the item, scoring guide, examinee answers, and analysis for MEE items—are assembled, bound, and boxed for transportation to the in-person ­participants by NCBE Test Operations Department staff. Those participating by conference call or later via on-demand video are provided with instructions for secure online access to the Examinee Answer Books.

Photo of Testing Booklets Photo of Testing Booklets Photo of Making the Testing BookletsPhoto of employees packing up the Testing BookletsPhoto of employees making the Testing Booklet Photos of Testing Booklet Boxed up

  • Two pilot programs were launched at the July 2017 Workshop to enhance the interactive experience of graders. Poll Everywhere (a secure real-time audience response system that gathers live responses via texting, web, and a mobile app) was used in all sessions for graders to submit their grades—replacing the prior method of texting their grades to a staff member or relaying their grades verbally—allowing quicker tabulation of grades and providing anonymity for graders when submitting their grades. Zoom (a secure cloud-based video conferencing product allowing graders to join the session in a variety of ways) was used in two sessions and allowed graders participating by conference call to connect with the facilitator in a live, virtual environment much like attending the live session in person. Also new this year was an app for the Grading Workshop itself. These were all tested extensively by NCBE staff members prior to launch, and mock Zoom sessions were offered to graders so that they could familiarize themselves with the format and various ways to connect.

Photo of computer

Mock Zoom Session

Photo of Mike Petterson

Mike Petterson testing the pilot programs and app

Friday, July 28: The Day Before the July 2017 Workshop

NCBE staff and facilitators gather at the conference hotel to finalize preparations for the Workshop.

Facilitators Jon Carlson and Shelly Kurtz (both Iowa), Mike Petterson

Workshop attendees, facilitators, and NCBE staff members attend a reception hosted by NCBE on Friday evening.

Photo taken at conference of Frankie Jones

Frankie Jones (AZ)

Photo taken at conference of Rebeka Fortess and Sonja Olson

Rebeka Fortess (NH), Sonja Olson

Photo taken at conference of Ted Tollefson and Matt Gunn

Ted Tollefson and Matt Gunn (both ID)

Photo taken at conference of Martin Sjolie

Martin Sjolie (WA)

Photo taken at conference

 

Saturday, July 29: The Day of the July 2017 Workshop

NCBE staff members arrive at the conference hotel early on Saturday morning.

Photo taken at conference

Photo taken at conference of NCBE Staff

Ellen Embertson-Merrill, Ingrid Jorgenson, Katie Chu, Lu Riese, Kasey Brinkman, Mike Petterson, Gina Bailey

Photo taken at conference of NCBE Staff

Judy Gundersen, Joanne Kane

Photo taken at conference NCBE Staff and facilitator

Sonja Olson, facilitator Shelly Kurtz (IA), Gina Bailey, Mike Petterson, Lu Riese, Katie Chu

Staff preparation includes setup for videotaping the sessions for later on-demand viewing.

Photo taken at conference

Photo taken at the conference

Saturday morning begins with a presentation on grading fundamentals by three NCBE staff members.

Photo taken at conference of attendees

Photo taken at conference of Judy Gundersen

Judy Gundersen

Photo taken at conference of Joanne Kane and Andy Mroch

Joanne Kane, Andy Mroch

Seven facilitators help lead the 10 hands-on sessions that take place throughout Saturday morning and afternoon. (Not pictured: Alexander Scherr [GA], who facilitated remotely.)

Facilitators Adisa Harrington and Catherine Klein

Facilitators Adisa Harrington (TX) and Catherine Klein (DC)

Photo taken at conference of Jon Carlson and Shelly Kurtz

Facilitators Jon Carlson and Shelly Kurtz (both IA)

Photo taken at conference of Alan Palmiter

Facilitator Alan Palmiter (NC)

Photo taken at conference of Judy Gundersen

Facilitator Judy Gundersen

Sessions begin with an overview of the item and grading materials, and any questions about the area of law (MEE) or assigned task (MPT) are addressed.

Photo taken at conference of attendees

Participants then silently read the real examinee answers collected from jurisdictions across the country.

Photo taken at conference of attendees

Photo taken at conference of attendee

Participants grade the answers.

Photo taken at conference of attendeesPhoto taken at conference of attendees

Staff members in each session assist the facilitators with compiling the attendees’ grades.

Photo taken at conference of Alan Palmiter and Lu Riese

Facilitator Alan Palmiter (NC), Lu Riese

Photo taken at conference of Gina Bailey and Shelly Kurtz

Gina Bailey, facilitator Shelly Kurtz (IA)

Participants and facilitators discuss the grades assigned, with graders being called on to explain the grades they gave—particularly if a grade might be an outlier from grades assigned by other graders in the session.

Photo taken at conference of attendees

Based on the review and grading of the sample of examinee answers and the ensuing discussion between graders and facilitators, grading materials may be refined or grading weights adjusted. The final versions of these grading materials are then made available to graders in user jurisdictions after the Workshop.

Photo taken at conference of a camera videotaping a session

Photo taken at conference of NCBE's Staff

All sessions are videotaped for on-demand viewing.

The July 2017 MEE/MPT Grading Workshop had a record number of participants:

  Graders Participating  Jurisdictions Represented 
In Person    68 22
By Conference Call  140  18
Via On-Demand Video  225 30

Photo taken at conference of attendees

Attendees from Kansas: Patrick Hughes, Donald Peterson, Fran Angell, David Cooper, Peter Johnston (Kansas attendee Molly Wood not pictured)

Photo taken at conference of attendees

Attendees from Hawaii: Front row: Annette Chock, Hon. Adrianne Heely; Back row: Shellie Park-Hoapili, Leighton Hara, Lester Oshiro

Photo taken at conference of attendees

Attendees from New Mexico: Front row: Twila Hoon, Sophie Martin, Bonnie Stepleton, Henry Alaniz; Back row: Olga Serafimova, Hon. Edward Chavez, Michael Thomas, Kevin Hammar, Karen Grohman, Michael Hoses (New Mexico attendee Michelle Haubert not pictured)

Contact us to request a pdf file of the original article as it appeared in the print edition.

  • Bar
    Bar Exam Fundamentals

    Addressing questions from conversations NCBE has had with legal educators about the bar exam.

  • 2021
    2021 Bar Admission Guide

    Comprehensive information on bar admission requirements in all US jurisdictions.

  • NextGen
    NextGen Bar Exam of the Future

    Visit the NextGen Bar Exam website for the latest news about the bar exam of the future.

  • BarNow
    BarNow Study Aids

    NCBE offers high-quality, affordable study aids in a mobile-friendly eLearning platform.

  • 2020
    2020 Year in Review

    NCBE’s annual publication highlights the work of volunteers and staff in fulfilling its mission.

  • 2020
    2020 Statistics

    Bar examination and admission statistics by jurisdiction, and national data for the MBE and MPRE.