In this issue, we bring you the second part of a section on how questions are drafted for NCBE’s exams—a process that involves multiple stages of drafting, review, and revision by the volunteer members of NCBE’s 10 drafting committees, with the support of NCBE’s attorney editors.
Part One in the Fall 2019 issue shared the process for developing questions for the Multistate Bar Examination (MBE) and the Multistate Professional Responsibility Examination (MPRE) from members of those drafting committees, including the important step of pretesting that all questions for NCBE’s four exams undergo.
Continue reading to learn from two members of the Multistate Essay Examination (MEE) and Multistate Performance Test (MPT) Drafting Committees about how questions are written for those two exams.
Check out this article to learn more about the MEE and MPT test development process: MEE and MPT Test Development: A Walk-Through from First Draft to Administration (June 2015)
The Gestation of an MEE Question: A Rigorous Process
By Sheldon F. Kurtz
As the longtime chair of NCBE’s Multistate Essay Examination (MEE) Drafting Committee, I welcome the opportunity to reflect on the MEE from a drafter’s perspective and to explain the rigorous process by which an MEE question is created.
MEE questions are drafted collaboratively by the MEE Drafting Committee, currently consisting of six people: one appellate judge and five law school faculty members. Over the course of the exam’s existence, the subject matter tested by the MEE has changed. Currently, there are 12 areas of law that may be covered by the questions on any MEE. Generally, only six of these subjects are tested in a particular administration, although some questions may require analysis of more than one subject area. Each of these subject areas is an area of expertise of at least one member of the committee.
Development of an MEE question from initial drafting to test administration typically takes about three years because of the care and attention to detail paid to the development of each question.
For each question on an administered exam, there is at minimum (1) the initial draft; (2) drafting committee review; (3) outside expert review, followed by drafting committee review; (4) pretesting, followed by drafting committee review; (5) assignment to a test form or to another category (test form referring to a particular set of test questions administered at a given time), followed in either case by additional reviews at later meetings; and (6) policy committee review, followed by drafting committee review. Throughout, NCBE staff members, including its attorney test editors, provide another valuable layer of editorial review, which only serves to strengthen the quality of the question.
The First Draft and Drafting Committee Review
After the need for a question in a specific subject area and the specific topic(s) within that area is determined (e.g., from the MEE subject-matter outline, this might be “business associations—corporations and limited liability companies”), a first draft of a question is prepared by either a drafting committee member or an outside question writer (a law school professor). Questions (and analyses, discussed later) are prepared according to NCBE’s MEE Item Writer’s Guidelines, which detail such things as length, style conventions, analysis content and format, and suggested grading percentages. In no case has the final draft of an administered question ever looked like the first draft, because each question goes through multiple stages of review and revision.
When a newly commissioned question is received, it is assigned to a “shepherd,” a drafting committee member (who is not the author of the question and who is a content expert in the question’s subject area). The shepherd is responsible for leading the discussion and editing of the question through the first meeting at which it is reviewed by the entire drafting committee. At that first meeting, the entire committee determines whether the question is appropriate for a future test. If so, the committee offers some initial editorial suggestions for the shepherd to review outside the meeting. In addition, the committee decides whether to collectively review the question again or, following the shepherd’s edits, to move it forward to one or both of the next test development steps.
Multistate Essay Examination (MEE)
- Six 30-minute essay questions covering up to 12 areas of substantive law: Business Associations (Agency and Partnership; Corporations and Limited Liability Companies), Civil Procedure, Conflict of Laws, Constitutional Law, Contracts, Criminal Law and Procedure, Evidence, Family Law, Real Property, Secured Transactions, Torts, and Trusts and Estates (Decedents’ Estates; Trusts and Future Interests). The particular areas covered vary from exam to exam. (Visit ncbex.org/exams/mee/preparing/ for a complete MEE Subject Matter Outline.)
- Produced by NCBE since 1988.
- Administered by 40 user jurisdictions as part of the bar examination.
- Developed by NCBE; graded by jurisdictions.
- Purpose: to test the examinee’s ability to (1) identify legal issues raised by a hypothetical factual situation; (2) separate material which is relevant from that which is not; (3) present a reasoned analysis of the relevant issues in a clear, concise, and well-organized composition; and (4) demonstrate an understanding of the fundamental legal principles relevant to the probable solution of the issues raised by the factual situation. The primary distinction between the MEE and the MBE is that the MEE requires the examinee to demonstrate an ability to communicate effectively in writing.
- 2019–2020 MEE Drafting Committee: 6 members from 5 jurisdictions (Georgia, Iowa, Minnesota, New York, North Carolina), including 1 appellate judge and 5 faculty members from 4 different law schools.
Outside Expert Review and Pretesting
Once the drafting committee determines that a question is ready to move forward, it is sent for outside expert review to a law professor or an established practitioner and is also pretested by newly licensed lawyers. The reviewers are given guidelines to follow in preparing their reviews, including questions to answer about the analysis.
Pretesters are identified and invited by bar administrators across the country to take the MEE under exam-like conditions using an application that records how much time is spent on each question. They complete a survey about each question, rating it on clarity of facts and call of the question, difficulty, bias, and appropriateness in general. They also participate in a debriefing session with the bar administrator from their jurisdiction, who then prepares a memo that the drafting committee reviews at its next meeting.
The drafting committee carefully considers the results of the outside expert review and pretesting, which always lead to significant improvements to the question. The committee then either assigns the question to a future exam form or places it in a test-ready category for eventual assignment to a test form.
Further Review After Assignment to a Test Form
Because a test form is typically assembled 18 months before its administration, the drafting committee will typically review a question at least two more times following the question’s assignment to a test form. Then, a few months before the scheduled test’s administration, the test form with all six questions is sent to the MEE/Multistate Performance Test (MPT) Policy Committee for review and comment. The drafting committee may make further changes to the questions based on these comments.
By now, it should be clear that questions are not simply drafted shortly before a test administration. It’s a much longer process with numerous quality-control steps built into the making of the final product.
Developing the Summary and Analysis
For each MEE question, the drafting committee also drafts a summary and an analysis. The analysis is provided to the jurisdictions to assist graders in grading the MEE; it is illustrative of the discussion that might appear in an excellent answer to the question. As much care goes into the drafting of the summary and analysis as goes into the drafting of the question.
The summary is a shortened version of the analysis and represents the drafting committee’s judgment of what a high-quality answer could be. The analysis, however, includes much more information than the summary because it has the additional goal of educating the graders assigned at the jurisdiction level about the subject matter being tested. Furthermore, the analysis provides graders with a detailed discussion of the issues to help them appropriately rank-order answers and often makes suggestions to graders as to what discussion in an actual answer is worthy of credit.
Preparation of the summary and analysis is an absolutely essential part of quality control. It ensures that a question’s fact pattern fully supports the conclusion and rationale reflected in the summary and analysis. And if, in the early stages of test development, the question and the summary/analysis are out of sync, it signals that changes in the facts or analysis must be made.
It is important that the graders fully understand the facts and analysis of the question so that they are able to grade essay answers in a consistent manner. Accordingly, following each MEE/MPT administration, NCBE holds a Grading Workshop to support jurisdiction graders in their grading of the MEE and the MPT; at these workshops, drafting committee members act as facilitators during hands-on grading sessions.
The Fruits of a Hardworking Drafting Committee’s Labor
Perhaps the most impressive thing I’ve seen over my years on this drafting committee is the intensity and devotion with which committee members approach their work and the pride that they take in ensuring that MEE questions are of the highest possible quality. The time commitment of the faculty members, judges, and lawyers serving on the committee is enormous. In addition to the time throughout the year that each committee member spends drafting and editing questions (which itself is substantial), the committee meets twice a year for two consecutive 8- to 9-hour days.
Furthermore, the committee’s commitment to quality necessitates an enormous amount of wordsmithing to the question, the summary, and the analysis. I’m very proud to say that this process (as tedious as it sometimes feels during a meeting) has paid off.
At the core of every successful project are people. It is the dedicated committee members and NCBE staff with whom I am honored to work who make the MEE test development process a success.
Sheldon F. Kurtz is the David H. Vernon Professor of Law at the University of Iowa College of Law, where he teaches the first-year Property Law course as well as upper-class courses on Trusts and Estates and Health Law. He also holds an appointment in the University of Iowa Carver College of Medicine. He has served on NCBE’s MEE Drafting Committee since 1988 and as chair since 1990.
Drafting MPT Items: Guiding Principles and Collaboration
By Alexander W. Scherr
Drafting items (or questions) for the Multistate Performance Test (MPT) requires a special set of abilities: knowledge of the law, careful writing, imagination and creativity, and good judgment. The MPT Drafting Committee includes law professors and lawyers from diverse practices who have worked with young lawyers and law students. The drafting process for an MPT item takes about two years and a high degree of collaboration. This article describes the format and the drafting process for an MPT item. It ends by articulating some guiding principles for and reflections on the work of drafting a good item.
The Format of an MPT Item
An MPT item asks examinees to complete a legal task in writing based on a universe of facts and law included in the MPT materials. Each MPT item has two parts: a File and a Library.
The File starts with a memo that contains instructions to the examinee from an assigning attorney, often followed by directions on how the examinee should format the written product. The balance of the File contains facts presented in diverse formats, including transcripts of hearings or interviews, results of law firm investigations, newspaper articles, or excerpts from contracts, pleadings, or other documents. Drafters avoid maps, graphics, or complex forms, which are difficult to replicate in those versions of the test created for examinees who have received ADA accommodations for visual disabilities.
The Library contains the law necessary to accomplish the task. Libraries often include selected statutes and cases but may also contain regulations, ethical opinions, Restatement provisions, and secondary sources. Drafters use either federal or state sources, as appropriate, making sure to select well-accepted legal principles.
Examinees must use these materials to complete a task that a young lawyer in practice might perform: an objective memo, a persuasive brief, a contract or other transactional document, a proposed judicial order for a judge, a client letter, or a pleading, among others.
All the necessary facts appear in the File; all the necessary law appears in the Library. For this reason, the MPT is not a test of the examinee’s knowledge of substantive law. Instead, it tests the examinee’s ability to read carefully, to sort and organize facts, to identify relevant legal principles, and to integrate facts and law to accomplish the task set in the memo through careful organization and writing.
Multistate Performance Test (MPT)
- Two 90-minute tasks designed to test fundamental lawyering skills in a realistic situation. (Visit ncbex.org/exams/mpt/preparing/ for a complete list of MPT Skills Tested.)
- Produced by NCBE since 1997.
- Administered by 47 user jurisdictions as part of the bar examination. (Jurisdictions may select one or both MPT items to include as part of their bar examinations; jurisdictions that administer the Uniform Bar Examination use both MPT items.)
- Developed by NCBE; graded by jurisdictions.
- Purpose: to test an examinee’s ability to use fundamental lawyering skills in a realistic situation and complete a task that a beginning lawyer should be able to accomplish. The MPT is not a test of substantive knowledge. Rather, it is designed to evaluate certain fundamental skills lawyers are expected to demonstrate regardless of the area of law in which the skills are applied.
- 2019–2020 MPT Drafting Committee: 8 members from 7 jurisdictions (District of Columbia, Georgia, Minnesota, New Mexico, New York, Tennessee, Texas), including 3 practicing attorneys and 5 faculty members from 5 different law schools.
The Drafting Process
Drafting an MPT item typically takes about two years and includes several stages of drafting, critique, editing, and review, including at least three edits by the drafting committee and critical assessments from a wide range of sources.
The First Draft and First Committee Review
An MPT item starts with an individual drafter on the committee, who identifies an area of the law, a specific scenario within that area, and a task for the examinee to perform. The drafter produces both an initial draft and a point sheet. This point sheet describes the item in detail and outlines the issues and grading points that jurisdictions will eventually use in grading the item.
NCBE staff members also arrange a pretest of the item, with 8–10 newly licensed lawyers selected by bar administrators taking the MPT item under exam-like conditions and providing feedback about the item. The bar administrator also prepares a report summarizing the feedback from the pretesters about the item.NCBE also appoints a separate member of the committee as a “shepherd,” with responsibility to lead discussion and editing of the item. Drafter and shepherd discuss the item early in the process, making sometimes extensive edits to the item before it is sent to the full committee. All committee members review the item and come prepared to comment at the next semiannual drafting meeting. At that meeting, the shepherd presents the draft, identifies significant areas for group discussion, and leads a line-by-line review of the item.
Outside Expert Review, Pretesting, and Policy Committee Review
After revisions in the first meeting, NCBE staff members submit the item to outside expert reviewers: one law professor and one practitioner with expertise in the pertinent area of law. NCBE staff members also arrange a pretest of the item, with 8–10 newly licensed lawyers selected by bar administrators taking the MPT item under exam-like conditions and providing feedback about the item. The bar administrator also prepares a report summarizing the feedback from the pretesters about the item. Experts and pretesters sign confidentiality forms to prevent disclosure of the item or its topic.
NCBE staff members send the pretest answers, the pretest report, and the expert reviews to the drafting committee, which reviews these materials and grades the pretest answers before the next meeting. At that meeting, a new shepherd prepares a list of edits suggested by these materials, which the committee incorporates as appropriate.
When items are assigned to a test form, they receive another review by NCBE’s Multistate Essay Examination (MEE)/MPT Policy Committee, consisting of bar examiners from MEE and/or MPT user jurisdictions. The policy committee comments come back to the drafting committee for a third review of the items and further editing before the items are finalized.
Developing the Grading Materials
Once the third review is complete, NCBE attorney editors take the final steps to prepare the question for the bar exam. They prepare grading materials that include the point sheet, several summaries of grading weights and points, and a model answer. During this process, NCBE attorney editors often work with the drafter or earlier shepherds of the item to finalize these materials, which sometimes produces additional edits before the item appears on an upcoming exam.
Components of an MPT Item
The File consists of source documents containing all the facts of the case. The specific assignment the examinee is to complete is described in a memorandum from a supervising attorney. The File might also include transcripts of interviews, depositions, hearings or trials, pleadings, correspondence, client documents, contracts, newspaper articles, medical records, police reports, or lawyer’s notes. Relevant as well as irrelevant facts are included. Facts are sometimes ambiguous, incomplete, or even conflicting. As in practice, a client’s or a supervising attorney’s version of events may be incomplete or unreliable. Examinees are expected to recognize when facts are inconsistent or missing and are expected to identify potential sources of additional facts.
The Library may contain cases, statutes, regulations, or rules, some of which may not be relevant to the assigned lawyering task. The examinee is expected to extract from the Library the legal principles necessary to analyze the problem and perform the task. The MPT is not a test of substantive law; the Library materials provide sufficient substantive information to complete the task.
Six Guiding Principles for Drafting MPT Items
The following six principles guide the drafters and editors of MPT items.
1. The item should be realistic.
Items should present facts and require the performance of tasks that address problems that a lawyer will likely see in practice. Most items include both relevant and irrelevant facts, testing the examinee’s ability to sort and winnow. The legal sources reflect real-world majority or minority approaches to the scenario posed by the item.
2. The task should pose challenges that occur in real practice.
The task should be one that a newly licensed lawyer would encounter in practice. The practice contexts for items may vary, including work in private firm, nonprofit, governmental, and judicial practices. But the item’s subject matter and its assigned task should offer a fair test of examinees’ ability to perform work they will see early in their careers.
3. The item should require answers that cover a range of difficulty.
Each item typically includes several different grading points, which should range in difficulty from easier to more difficult. Almost every item has one or two relatively easy points as well as points that only better examinees will see and address. This practice should lead to answers that allow graders to rank-order examinees reliably by the quality of their performance.
4. Items should cover a variety of tasks.
Over time, the items presented on exams should present tasks drawn from a wide range of practice areas. Items should not focus solely on writing memos, but should call for work characteristic of litigation, transactional, and judicial practices. NCBE’s MEE/MPT Program Director monitors the bank of completed items and encourages drafters to create items that fill in any gaps in coverage.
5. The item should fairly reflect skills gained during law school.
Items should fairly reflect the training that examinees will have received in law school. Assessing the tax consequences of a complex corporate reorganization would, for most law students, fall outside the abilities they develop in law school. By contrast, writing a persuasive argument or assessing a contractual provision would fall within the average law student’s training.
6. An examinee should be able to complete an answer within 90 minutes.
The pretest process helps the committee to determine whether the item can be read and completed within the 90-minute time span of an MPT. Pretesters record how long it takes them to complete an item and provide feedback on the time pressure they experience. This helps the committee assess whether the item needs rewriting to increase its readability and whether the item has too many or too few issues for the examinee to address.
Three Observations about the Drafting Process
Three aspects of the drafting process stand out from my nearly 15 years of drafting MPTs. First, collaboration is central to the process. Initial drafters rely on feedback from the shepherds and the committee that draws on a wide range of knowledge and perspectives; item editors play a critical role as constructive readers. NCBE fosters this collaboration with a web of useful resources: outside experts, pretest participants, bar examiners, copy editors, and experienced NCBE staff members.
Second, because it is a written exam, the MPT cannot test skills that do not involve written performance: interviewing a client, questioning a witness, negotiating a settlement or a contract, or managing the flow of a trial. That said, MPT items serve as useful proxies for testing related skills, including specifically the ability to deal with facts and to solve problems within the limits of the law and the range of client goals and needs.
Finally, MPT items differ importantly from essay and multiple-choice tests, in that they test problem-solving in a plausible real-world setting. Rather than testing memory or acquired knowledge, MPT items test how well examinees accomplish a set of goals with materials that they first encounter in the exam setting. In so doing, the MPT tests the examinee’s capacity to exercise judgment in serving a client and thus plays a unique role in assessing whether an examinee has the ability necessary for admission to the bar.
Alexander W. Scherr is Associate Professor of Law and Veterans Legal Clinic Director at the University of Georgia School of Law, where he directs a clinic serving veterans and teaches Evidence, Mental Health Law, and other courses. He has served on NCBE’s MPT Drafting Committee since 2005.