It’s October. The seasons are changing. The air is growing crisper. And people in the United States are beginning to take more interest in Major League Baseball as the World Series fast approaches. In celebration of the season, we invite legal professionals to revisit “Moneyball,” the 2011 sports drama starring Brad Pitt and directed by Bennett Miller.
Pitt plays Billy Beane, who, it should noted upfront, is neither a lawyer nor an AI expert. He is however the general manager of the 2002 Oakland A’s, a lifelong student of the game who struggled to consistently produce a winning team in a small baseball market. Beane didn’t have the budget to compete for players like teams in larger markets like New York or Boston. The A’s could develop players, but they couldn’t retain them when they became stars. They did, however, have access to the vital statistics.
Baseball is a sport with a century of data behind it and benchmarks like a player’s batting average are known by even casual fans. What Billy and the A’s did was use analytics and different key performance indicators to win. A player’s batting average is a great metric, but it doesn’t account for other factors like the ability to get on base so on-base percentage is better. Getting on base leads to more runs scored. And more runs scored means winning more games.
When losing a star player like Jason Giambi to the New York Yankees or Johnny Damon to the Boston Red Sox, conventional wisdom would say the team would have to replace two stars. But Beane, recognized the A’s needed to replace the production of the players they lost. In aggregate, they needed to get on base as much as the prior year. Using analytics, the A’s would go on to set an American League record by winning 20 straight games in a row and also made it to the World Series. But the way they did it was the bigger story.
So what can “Moneyball” teach us about benchmarking AI in legal?
The Stanford study on benchmarking GenAI solutions this past summer moved the conversation forward regarding the usefulness and impact of GenAI solutions. The study was not without some controversy that also helped in generating awareness to an important topic: How do we measure the results of GenAI on the legal industry.
The Stanford study tested leading research products on their ability to create answers to questions related to caselaw research. A correct answer was one that accurately reflected the current state of the law. An answer that did not reflect the current state of the law was considered a hallucination. The result? One in six queries hallucinated.
The definition of hallucination in the study is great for benchmarking. But does a hallucination as defined in the study always equal a bad outcome? What if the answer moved your research in the right direction and then you were able to formulate a Boolean search that answered your question? Another run crossing home plate.
And what about associates using traditional research solutions? Has anyone benchmarked their legal research skills to see how often their conclusions do not reflect the current state of the law?
The key points are:
- Benchmarks are important, and the right benchmarks for your goals are more important than what is easy to measure.
- Benchmarks on new approaches need to reflect the context of the effectiveness of current approaches.
- Outcomes are more important than benchmarks.
Outcomes are always interesting. The goals of two organizations can differ. And what counts as winning or a positive outcome at one level may be different at another level of an organization.
An entertaining television advertisement that viewers recall, is considered a winner in the advertising world. But what if viewers can’t remember the name of the advertiser? What if there is no discernable uptick in sales activity as a result of the ad campaign? Recall of an ad can be an example of a vanity metric — something that that is perhaps easy to measure but doesn’t support decisions that a business or law firm should make. The same pitfalls can apply to measuring the efficacy of GenAI solutions. Is what we are measuring aligned with outcomes for the firm?
To be sure, goals and outcomes can change over time. Billy Beane came up with a winning strategy to confront the realities of being a general manager in a small market. Circumstances have changed: On September 26, 2024, the Oakland A’s played their last game in Oakland as they prepared for an eventual move to Las Vegas, a much bigger market with its own unique challenges.
Next month, I’ll explore different use cases for legal GenAI and relate the performance of tools to positive outcomes. Said another way, I’ll explore how to identify getting on base to score runs to win games with legal GenAI.
Ken Crutchfield is Vice President and General Manager of Legal Markets at Wolters Kluwer Legal & Regulatory U.S., a leading provider of information, business intelligence, regulatory and legal workflow solutions. Ken has more than three decades of experience as a leader in information and software solutions across industries. He can be reached at [email protected].