Share this on Twitter Share this on Facebook Share this on Linked In Share this by Email
MCLE Self-Assessment Test

Better Predictors of Effective Lawyering

By Marjorie Shultz, J.D. and Sheldon Zedeck, Ph.D.

Marjorie Shultz
Sheldon Zedeck

By a show of hands, how many employers would like a reliable crystal ball when selecting new associates for their firms, to help determine how likely the candidates are to be effective lawyers? Wow … that’s a lot of hands.

Recent research conducted at the University of California, Berkeley, offers methods to appraise a wide range of individual competencies related to lawyering. The newly designed methods evaluate a much larger range of abilities than traditional grade and test-score assessments do; lawyers need classroom-type smarts, but they also need a lot of other skills not taught in school: problem solving; advocacy; creativity; self-development; negotiation; practical judgment; stress management; interviewing and counseling, just to name a few. 

In the research, conducted between 1999 and 2008, more than 2,200 lawyers participated during Phase I in individual interviews, focus groups and surveys to determine what factors lawyers believed important for professional effectiveness. Participants also supplied real-life examples to create scales attuned to actual lawyers’ behaviors so that an individual’s professional performance could be rated in terms of those factors.

Based on this information, researchers developed a new test battery that assesses the identified competencies. During the Phase III validation process, approximately 1,100 graduates of Berkeley Law and Hastings College of the Law (graduating between 1975 and 2006, practicing in a variety of fields and settings) took the new test, and granted access to their academic credentials. After taking the new test battery, those who participated also provided contact information for two peers and two supervisors who could evaluate their current work performance on 26 effectiveness factors identified earlier in the research. 

Researchers next compared each test-taker’s scores on the new test battery with the ratings by self, peer and supervisors of on-the-job performances. Correlations were significant enough to demonstrate that a wide array of lawyering skills can be predicted. The new test battery showed significant correlations (validity) in predicting 24 of the identified 26 factors whereas traditional academic indicators showed significant correlation with only eight of the 26, with several of those correlations being negative. For example, higher academic grades and test scores appeared predictive of lower competence in networking and business development. The most highly predictive segments of the new test battery can be completed online in less than an hour.

The best point in the pipeline to administer the new tests would be at application to law school. The research was initially directed toward expanding what admission officers consider in choosing among applicants. Based on this initial exploratory research, inclusion of professional effectiveness predictors would help align the de facto gateway to the legal profession with production of lawyers who are more effective, especially in the early years of practice. If legal employers use similar tests in making hiring decisions, better legal service for clients and society should result.

Use of professional effectiveness tests could also have another significant impact: it could meaningfully increase racial and ethnic diversity in the profession. As expected, based on experience in employment where these methods were pioneered, the Shultz-Zedeck research results were – for all practical purposes – race and ethnicity neutral. Several decades of experience with similar methods in employment decisions have shown that selection systems relying primarily on academic grades and tests create significant adverse impacts on underrepresented minorities; by contrast, merit-based selection methods that assess competencies actually required on the job greatly reduce or even eliminate such adverse impacts.

Even in a state as diverse as California, racial and ethnic diversity in the legal world has only increased at a glacial pace. That may be true partly because of deeply entrenched selection traditions that mainly pay attention to academic indicators. Expanding the types of assessments used in choosing among law school applicants, and in hiring and promoting lawyers, could make a real difference for diversity. 

The research discussed here was exploratory, aimed to demonstrate a new approach to selection for the legal profession. Further research and validation are important; nevertheless, the results of this initial, empirical study are sufficiently strong to suggest that two important goals of the legal profession – improved effectiveness and greater diversity – could be advanced by adding these new methods to those by which we currently choose people for training and practice.

We urge these new methods as “additions” because academic skills remain relevant in both education and employment. Users of professional effectiveness testing could combine it with academic indicators in a variety of ways: allocating varied weights to different types of information; using grades and test scores to set a floor for cognitive performance and considering professional prediction tests for decisions among those who make the cut; or choosing only to emphasize some from among the 26 professional factors according to needs, goals, and so forth. 

The legal professions of other countries are already using work-competency testing, and other professions in the United States are exploring the potential for such testing in admission to their professional training programs. For example, US medical schools recently announced plans to broaden selection criteria among applicants by adding communication and leadership skills to the scientific knowledge that has historically been the focus of MCAT testing. The Graduate Management Admission Council is currently researching which competencies are important to success in MBA programs and subsequent employment, and whether those can be better predicted by inclusion of tests other than those encompassed in the GMAT.

The law should not linger at the back of the 21st century curve in introducing newer and more comprehensive methods of selection and advancement within its professional ranks.

To learn more about this research and its potential applications, please contact Marjorie Shultz (510)642-1921 or Sheldon Zedeck (510)642-7130 For a more detailed description of the research, see 36 Law & Social Inquiry 620 (2011).

Marjorie M. Shultz is a professor emerita of the UC Berkeley School of Law, and she has written and consulted in the fields of contracts, health law and bioethics, race and gender and legal ethics. She is co-author of “White-Washing Race; The Myth of A Color Blind Society” (U.C. Press, 2003).

Sheldon Zedeck, Ph.D., is a professor emeritus of the psychology department at UC Berkeley. He retired in 2010 after more than 42 years at Cal, including almost four years as Vice Provost for Academic Affairs and Faculty Welfare.

Any opinions and/or viewpoints contained in this article belong solely to the author(s) and are not necessarily the opinion of the California Bar Journal or the State Bar of California or its staff and employees.