The Testing Column: Did UBE Adoption in New York Have an Impact on Bar Exam Performance?

This article originally appeared in The Bar Examiner print edition, Winter 2019-2020 (Vol. 88, No. 4), pp 34–42.

Highlights of a Study Directed by the New York State Court of Appeals

By Andrew A. Mroch, PhD, and Mark A. Albanese, PhD

In adopting the Uniform Bar Examination (UBE), the New York State Court of Appeals directed the New York State Board of Law Examiners (NYSBLE) to study the impact of the change to the UBE on candidate bar exam performance. The NYSBLE requested assistance from NCBE in conducting the study, which NCBE provided as part of its service mission as a not-for-profit corporation.

The study covered the two bar exam administrations immediately before UBE adoption (July 2015 and February 2016) and continued through the July 2017 administration, resulting in one February administration post-UBE adoption (February 2017) and two July administrations post-UBE adoption (July 2016 and July 2017). In addition to the overall impact of UBE adoption, the study addressed potential differential effects by gender and race/ethnicity.

The New York State Court of Appeals released the study results in a publicly available report on August 20, 2019, providing a rich trove of information on the background characteristics and performance of candidates taking the bar exam in New York between July 2015 and July 2017.1

This article briefly highlights several findings of the report relative to what happened before and after UBE adoption in New York regarding three topics:

bar examination performance
candidate background characteristics: pre-law-school undergraduate grade point average (UGPA), Law School Admission Test (LSAT) score, and law school grade point average (LGPA)
the relationships between candidate bar exam performance and their background characteristics (UGPA, LSAT score, and LGPA)2

What Data Were Used for the Study?

Two samples of New York bar exam data were analyzed.

Domestic-educated NYSBLE sample: The first sample, referred to as the domestic-educated NYSBLE sample, included candidates who had received a JD degree from an American Bar Association–approved law school in the United States.3
School-based sample: The second sample, referred to as the school-based sample, was a subset of the domestic-educated NYSBLE sample and included candidates for whom law schools throughout the United States provided their UGPAs, LSAT scores, and LGPAs. To facilitate a meaningful analysis, only those candidates whose law schools provided such data for at least 25 candidates were included in the school-based sample.

The LGPAs provided by law schools used various systems. The most common was the 4-point system corresponding to A = 4, B = 3, and so on, but some schools used a 100-point system and an assortment of other approaches. In order to appropriately analyze LGPAs from different schools, LGPAs were scaled in two ways to ensure comparability: (1) to range from 1 to 4 (resulting in the 4-point LGPA) and (2) to account for school-level differences in selectivity4 (resulting in the index-based LGPA). All analyses that included LGPAs were conducted separately using each method of scaling LGPAs.

Table 1 shows the numbers and percentages of candidates included in the two samples at each bar exam administration. Compared to the total number of domestic-educated candidates in the NYSBLE sample, the percentages of candidates represented by the school-based sample were relatively low for the February exams (22.8% and 30.5%) and for July 2015 (27.7%) compared to July 2016 (62.0%) and July 2017 (55.4%).

In addition to having a relatively small percentage of candidates represented in the school-based sample, the February results were sufficiently unstable that they were excluded from this summary. For July, there were differences in the percentage of candidates represented across years, but the numbers were sufficiently large that the school-based sample was still useful for studying candidates across July exams.

Bar examination scores also required adjustments in order to enable appropriate comparisons. Scores on the prior New York bar exam were on a 1,000-point scale but were converted in this study to the 400-point UBE scale to facilitate comparisons across exams. (See sidebar)

Table 1. Numbers and percentages of candidates in the New York UBE study samples

New York UBE study sample	February 2016 administration	February 2017 administration	July 2015 administration	July 2016 administration	July 2017 administration
Domestic-educated NYSBLE sample–%	100.0%	100.0%	100.0%	100.0%	100.0%
Domestic-educated NYSBLE sample–(n)	(2,346)	(2,370)	(7,513)	(7,292)	(6,776)
School-based sample–%	22.8%	30.5%	27.7%	62.0%	55.4%
School-based sample–(n)	(534)	(723)	(2,084)	(4,520)	(3,753)

The data used for the results presented in this article are indicated in bold.

The New York Bar Exam, Pre- and Post-UBE

The UBE consists of

the Multistate Bar Examination (MBE), weighted 50% of the total score, and
a written component consisting of six Multistate Essay Examination (MEE) questions, weighted 30% of the total score, and two Multistate Performance Test (MPT) questions, weighted 20% of the total score.

Scores on the UBE are on a 400-point scale. A passing score in New York on the 400-point UBE scale is a score of at least 266.

The New York bar exam prior to UBE adoption consisted of

the MBE, weighted 40% of the total score,
a written component consisting of five New York–developed essay questions, weighted 40% of the total score, and one MPT question, weighted 10% of the total score, and
50 New York–developed multiple-choice questions, weighted 10% of the total score.

Prior to adoption of the UBE, the passing score was 665 on a 1,000-point scale. This passing score corresponds to a 266 on the 400-point UBE scale.

What Were the Results of the Study?

Bar Exam Performance

Figures 1 and 2 show bar exam performance and pass rates across the period of the study.

Between July 2015 and July 2017, before and after UBE adoption in July 2016, bar exam performance and pass rates in New York increased, on average. For example, the pass rate for domestic-educated candidates in the NYSBLE sample was 72.5% in July 2015, 75.1% in July 2016, and 78.0% in July 2017 (Figure 2a).5
Males tended to score slightly higher than females, on average, across Julys, with the difference between males and females in the domestic-educated NYSBLE sample widening slightly in July 2016 upon UBE adoption before narrowing in July 2017 (Figure 1a).
Similar patterns of bar exam performance and pass rates were observed for the school-based sample (Figures 1b and 2b).
Candidates grouped by race/ethnicity showed similar differences in bar exam performance and pass rates across July exams. Bar exam performance and pass rates tended to increase for each group, particularly when comparing July 2015 to July 2017 (see Figures 1c and 1d for bar exam performance for both sample groups; see Figures 2c and 2d for pass rates for both groups). An exception was that the Black/African American group had mean bar exam scores that increased slightly and pass rates that decreased slightly between July 2015 and July 20166 and mean bar exam scores and pass rates that subsequently increased between July 2016 and July 2017, more than the other groups.

Because the composition and characteristics of candidates taking the bar exam may change across years, more information is needed to determine the extent to which the overall improvement in average performance on the bar exam in New York was due to the UBE versus other factors. This is where studying additional information, such as candidate background characteristics like UGPA, LSAT score, and LGPA, can help to better contextualize changes in bar exam performance and put them in perspective.

Figure 1. Mean bar exam scaled scores by gender and race/ethnicity for domestic-educated NYSBLE sample and school-based sample, July administrations, 2015-2017

(1a) Domestic-educated NYSBLE sample

(1b) School-based sample

(1c) Domestic-educated NYSBLE sample

(1d) School-based sample

Figure 2. Pass rates by gender and race/ethnicity for domestic-educated NYSBLE sample and school-based sample, July administrations, 2015-2017

(2a) Domestic-educated NYSBLE sample

(2b) School-based sample

(2c) Domestic-educated NYSBLE sample

(2d) School-based sample

Candidate Background Characteristics

Findings of the study on candidate background characteristics by gender are shown in Figure 3:

Of the three candidate background characteristics studied (UGPAs, LSAT scores, and LGPAs), UGPAs and LGPAs tended to remain constant or increase across the three July exams for both females and males (Figures 3a, 3e, and 3g).
Mean LSAT scores decreased slightly between July 2015 and July 2016 before increasing in July 2017 (Figure 3c).
Average values for background characteristics tended to differ by gender. Females tended to have higher mean UGPAs than males for groups taking each bar exam (Figure 3a).
This pattern was reversed for LSAT scores (Figure 3c) and both the 4-point and index-based LGPAs (Figures 3e and 3g), where males tended to have higher means than females.
Differences between males and females decreased between July 2015 and July 2017 for each background characteristic (Figures 3a, 3c, 3e, and 3g).

Findings of the study on candidate background characteristics by race/ethnicity are also shown in Figure 3:

Average values for candidate background characteristics tended to differ according to candidates’ race/ethnicity (Figures 3b, 3d, 3f, and 3h).
Each background characteristic between July 2015 and July 2016 tended to remain constant or increase for Asian/Pacific Islander, Black/African American, and Hispanic/Latino groups (although it can be seen in Figure 3f that the mean 4-point LGPA did dip slightly for the Black/African American group in July 2016 compared to July 2015).
For the Caucasian/White group, each background characteristic between July 2015 and July 2016 tended to remain constant or decrease.
In July 2017, mean background characteristics tended to increase for each group, with the exception of the Hispanic/Latino group, which had similar mean UGPAs (Figure 3b) and lower mean 4-point LGPAs in July 2017 (Figure 3f) compared to July 2016.

The pattern of mean UGPAs, LSAT scores, and LGPAs was generally consistent with the average performance on the bar exam between July 2015 and July 2017, where performance tended to increase.

Figure 3. Mean UGPA, LSAT score, and LGPA (4-point and index-based) by gender and race/ethnicity, July administrations, 2015-2017

(3a)

(3b)

(3c)

(3d)

(3e)

(3f)

(3g)

(3h)

The Relationship Between Bar Exam Performance and Candidate Background Characteristics

UGPA, LSAT score, 4-point LGPA, and index-based LGPA each had a relatively strong, statistically significant positive relationship with bar exam score, where a positive relationship indicates that an increase in background characteristic is associated with an increase in bar exam score. UGPA had the weakest relationship with bar exam score, and index-based LGPA had the strongest relationship, followed by LSAT score and then 4-point LGPA. Each relationship would be considered moderately strong to strong.7

One way of illustrating the relationship between the background characteristics and bar exam performance is to show how candidates at different levels of background characteristics performed on the bar exam. Figure 4 shows mean bar exam scores for candidates grouped by the three background characteristics (UGPA, LSAT score, or LGPA). For example, Figure 4a plots mean bar exam scores for candidates with UGPAs below 2.50 on the far left, then candidates with UGPAs between 2.50 and 2.69, and so on. Candidates with UGPAs below 2.50 had mean bar exam scaled scores between 254 and 267 depending on the year, and candidates with UGPAs above 3.89 (far right) had mean bar exam scaled scores between 303 and 316. Mean bar exam scaled scores increased as UGPAs increased from left to right across the figure, showing a moderately strong positive relationship.

Mean bar exam scaled scores also increased as LSAT scores and LGPAs increased (Figures 4b, 4c, and 4d). Another way of summarizing the positive relationships in these figures is that they illustrate that candidates with lower UGPAs, LSAT scores, or LGPAs tended to score lower on the bar exam, and those with higher UGPAs, LSAT scores, or LGPAs tended to score higher. The upward shift in the three lines corresponding to the three different years of scores indicates that as the years progressed, bar exam scores increased.

This study illustrated that for a bar exam like the one in New York, which already used the MBE and one MPT question on its exam prior to UBE adoption, the effects of UBE adoption were at most small and likely positive.Figure 4 also illustrates that the differences in lines relating the candidate background characteristics and bar exam scores across the three years were relatively similar for the three background variables shown in 4a, 4b, and 4c. What is notable is that the differences in the lines almost disappeared in 4d, which relates index-based LGPAs to bar exam scores. The index-based LGPAs can be considered LGPAs adjusted for differences in law-school-level UGPAs and LSAT scores. Thus, the increase in mean bar exam scores across the three years noted earlier and seen in the upward shift in the lines across the three years in figures 4a, 4b, and 4c was mostly removed when using index-based LGPAs. The lines are particularly close for the bar exam score of 266 where New York sets its passing score. The full report and appendices provide additional analyses that further reinforce that most of the increases observed in average bar exam scores across the three years could be accounted for by a combination of UGPAs, LSAT scores, and LGPAs. In other words, improvement in the UGPAs and LSAT scores of candidates on entry to law school and their subsequent performance in law school (LGPAs) accounted for most of the improvement in bar exam scores across the three July exams.

Figure 4. Correlations between mean UGPA, LSAT score, and LGPA (4-point and index-based) and mean bar exam scaled score, July administrations, 2015-2017

(4a)

(4b)

(4c)

(4d)

Conclusions

Mean bar exam scores and pass rates on the bar exam in New York increased, on average, after UBE adoption, and the improvement in performance was explained in large part by improvements in the UGPAs, LSAT scores, and LGPAs of candidates taking the New York bar exam. The improvement in bar exam performance in New York after UBE adoption was likely not attributable to the UBE. In addition, the UBE did not have sustained or adverse effects on candidates in New York compared to the prior bar exam. Differences in pass rates and average bar exam scores on the UBE observed across groups defined by gender and race/ethnicity were also observed prior to UBE adoption in New York.

This article provides only a glimpse of the results from the study that NCBE conducted for the NYSBLE. The full report addresses the topics described here in more detail and addresses other topics, including the performance of repeat test takers who did not pass the bar exam on their first attempt and MBE performance in New York compared to all other jurisdictions. This study illustrated that for a bar exam like the one in New York, which already used the MBE and one MPT question on its exam prior to UBE adoption, the effects of UBE adoption were at most small and likely positive.

Notes

The press release, Executive Summary, full report, and appendices are available on the New York State Board of Law Examiners’ website: Impact of Adoption of the Uniform Bar Examination in New York, https://www.nybarexam.org/UBEReport.html. (Go back)
Results for each topic were further analyzed by gender and race/ethnicity. For a more complete description of the data and extensive analysis of the samples of data, see the full report and associated appendices at https://www.nybarexam.org/UBEReport.html. (Go back)
In New York, roughly 30% of candidates in July and 40% of candidates in February received their legal education outside the United States. (Go back)
School-level UGPAs and LSAT scores were used to adjust LGPAs such that index-based LGPAs as a group from a more selective school (based on UGPAs and LSATs at that school) would be higher than index-based LGPAs from a less selective school (based on UGPAs and LSATs at that school). Otherwise put, if two candidates from different law schools have the same LGPA, the candidate from the more selective school would generally have the higher index-based LGPA. (The index referred to is one that was computed for this scaling process based on each candidate’s LSAT score and UGPA.) Index-based LGPAs were placed on a scale that ranged from roughly 1 to 15 with most values typically near 10 and a range typically falling between roughly 7 and 13. (Go back)
Average bar exam scores were 283.04 in July 2015, 286.85 in July 2016, and 290.98 in July 2017. (Go back)
The increase in mean bar exam score but decrease in pass rate may seem counterintuitive but has to do with shifts in the distributions of scores between July 2015 and July 2016 for the Black/African American group. (See Figure 4.2.29, Distributions of July Bar Exam Scores, Black/African American Candidates, New York State Board of Law Examiners Sample, on page 150 of the full report.) (Go back)
Correlations between UGPAs and bar exam scaled scores were 0.46, 0.40, and 0.45 in July 2015, 2016, and 2017, respectively. The corresponding correlations between LSAT scores and bar exam scores were 0.56, 0.57, and 0.57. The 4-point LGPA correlations were 0.65, 0.61, and 0.61, while the index-based LGPA correlations were 0.76, 0.75, and 0.75. (See Table 3.7.1, Correlations among UGPA, LSAT Scores, LGPA, MBE, Written Scores and Bar Exam Scores, School-Based Sample, on page 109 of the full report for additional details.) (Go back)

Andew A. Mroch, PhD, is Senior Research Psychometrician for the National Conference of Bar Examiners.

Mark A. Albanese, PhD, is the Director of Testing and Research for the National Conference of Bar Examiners.

[apss_share]