Raghu Ramanathan, who in February was named president of the Legal Professionals segment within Thomson Reuters, overseeing all its products for the legal profession, believes there should be open benchmarking on legal AI products, he told me during an interview earlier this week.
“I do believe the industry needs open benchmarks on AI performance in legal assistants,” he said. “I welcome it. Personally, I think the more well-funded, well-conducted research there is – even multiple ones so we can compare and contrast – it’s better for the industry, it’s better for the customers.”
His remarks come on the heels of a study by researchers at Stanford University that found that Thomson Reuters’ AI legal research product, AI Assisted Research for Westlaw Precision, delivered hallucinated results nearly a third of the time, and at twice the rate of the comparable Lexis Nexis product, Lexis+ AI.
The study’s authors said the results point to “the need for rigorous, transparent benchmarking and public evaluations of AI tools in law.”
But Ramanathan, who was formerly a vice president at SAP, said he was surprised by the results of the study, in that they do not comport with the company’s own internal testing or with what he is hearing from customers.
In our conversation earlier this week, we talked about Ramanathan’s background, his priorities as president, and his views on the market.
What follows is a transcript of that conversation, which I have condensed and edited for style and continuity.
Robert Ambrogi: Tell me about your background.
Raghu Ramanathan: I come from the tech industry. I used to work at SAP, an enterprise software company, for the last 19 years. Most recently, I was head of the platform division, where we were developing and selling solutions for application development, application integration, everything around data management, AI, ML, all of that. I built that from a $250 million business to a $2.2 billion business over the course of the last five years.
Prior to that, I had studied engineering, so I did electronics, telecommunication, computer science, and engineering. But I went into the business side and I worked as a consultant and was an associate partner with McKinsey and Company. I was part of the strategy practice helping technology companies, as well as telecom and media companies.
What you might notice is that I haven’t directly worked in the legal industry. Other than the fact that my father is a lawyer, I don’t have any direct connection to the industry. But I have spent a lot of time in the professional services industry. Consulting, you can argue, has some parallels to the legal industry. Plus, in my time at SAP, I was running our professional services business unit, which took care of technology and software solutions for all professional services, including law firms.
Ambrogi: Given that you hadn’t worked in the legal industry, what was it that attracted you to this position?
Ramanathan: A lot of it was around AI and the opportunity to completely transform an industry. I had worked a lot in AI across a broad set of industries. My experience was that the technology was great, but use cases were hard to make stick. You would come up with one use case for one company, but it was not transportable to another company. It was a case of technology seeking a solution. The sense that I had is that in the legal industry, the use cases are pretty clear. Lots of a lawyer’s work is about reading, writing, and analysis, and this is where I think gen AI is most progressed.
Ambrogi: Let me ask the flip side of that question. You said why you wanted to go to Thomson Reuters, but why did Thomson Reuters want you to go to them?
Ramanathan: As you know, they are transforming from being a content-oriented company to a content-enabled technology company. I think that what was attractive to them were two things. One, my deep background in running technology companies, enterprise software, SaaS software as a service, and cloud companies. That background is what they thought was needed for the future.
The second part is that I’m not purely coming from a technology company. I have my consulting background. I did run a professional services group. I’m very familiar with professional services. So while I may not be deep in legal, I do know what’s happening in the consulting industry, in the IT services industry, in the audit tax industry, and I believe there are parallels in terms of the evolution of these other professional services companies and how that might impact law firms. In fact, the funny thing is, I wrote a white paper about eight or nine years ago titled ‘Professional Services on the Brink of Disruption.’ I predicted what happens in the next 20 years in professional services companies.
Ambrogi: Well, the legal industry’s been on the brink of disruption for a long time. It just can’t get over that brink.
Ramanathan: What we are realizing, as we launch our AI products, is that the level of support we need to provide our customers is very different from our classical products like Westlaw or Practical Law. They were more intuitive, straightforward to use in a self-service mode. What we’re finding with AI is our customers need a lot of orientation and training on how to use it properly. That’s where the parallels with the SaaS world come in. One of the areas I’m looking into is customer success. I do believe for AI offerings you need a strong customer success function to invest in the customers, to help them use those products and generate real value.
Ambrogi: Does that mean one-on-one customer support or does that mean that you’re launching initiatives to help customers in a larger sense understand these technologies?
Ramanathan: It’s a cultural transformation. It starts with a a philosophy and culture that everybody needs to buy into, which is that we measure our success and we say we are successful only if our customers are using the solution the way it’s intended to be used and are deriving value and satisfaction from it. I think there will be organizational investments that we need to make, investments in technology and process systems that we need to make, compensation incentives we need to change. And there will be an element of one-to-one support for our larger customers. Our larger customers will have a named person who they can rely on to say, ‘Hey, come and sit with us and help us understand how to use this more.’
There will be e-learning digital solutions to enable the rank and file of our customers to benefit from it as well. The big change for me is that, in customer success, we say it’s about the end user, it’s about the practitioner, it’s less about the managing partners, it’s less about the chief innovation officers or digital officers. It is about the individual associate, the individual lawyer, about understanding what’s blocking them from using and generating value from these technologies.
Ambrogi: As president of the legal professionals segment, you oversee all legal products in the Thomson Reuters portfolio. What’s your primary mission? Is it to develop the products? Is it to grow the market?
Ramanathan: Number one for me is helping the industry transform. This is our true north. What does it mean? It’s about making sure that we are evangelizing and leading this AI charge, coming up with products which are really adding value to the industry, which are cutting edge.
One of the things I believe is really attractive as part of that, Bob – and I spent a lot of time with customers in the last months testing it and it seems to resonate – is this concept of universal legal assistant. The idea is that AI is not there to replace you as a lawyer, but it’s going to be side by side with you as a lawyer, as your assistant, as your paralegal. Everybody needs to have one by their side as they do their day-to-day work. So it’s making that vision come alive, evangelizing that vision. It is making sure that the products are fit for that purpose.
Also, it is making sure that we can deliver value by partnering with the industry to help with change management. With our large customers, we believe we can spend some time with them, educating them, coaching them on what good change management looks like, how to get there, partnering with them on figuring out business models, revenue models. We’re taking a much broader perspective as a transformation partner rather than just a technology vendor.
Ambrogi: The company has talked about its plans to deploy CoCounsel across products and segments. Is that primarily how you achieve that goal of that AI assistant?
Ramanathan: Exactly right. CoCounsel becomes this universal legal assistant. The way we are developing our AI is based on a finite number of skills, like summarization, drafting, et cetera, but the whole idea is that the user can be agnostic about which product delivers which skill. That’s for us to figure out in the backend. The whole idea is that it’s a very simple, universal solution that will leverage the entire breadth of the product portfolio and content portfolio of Thompson Reuters.
Ambrogi: There has been a lot of talk over the last couple of weeks about the Stanford study that came out on hallucinations in AI legal research products. What’s your take on that? Do you feel that it in any way derails the momentum around gen AI or causes you to go in a different direction in terms of your own thinking about generative AI in legal?
Ramanathan: I do believe the industry needs open benchmarks on AI performance in legal assistants. I welcome it. Personally, I think the more well-funded, well-conducted research there is – even multiple ones so we can compare and contrast – it’s better for the industry, it’s better for the customers.
Specifically, I would say the results of the study are surprising because it’s incongruent with the feedback that I’m hearing day in, day out with our customers. I was with a customer last week and they were telling me that it used to take them something like 300 minutes to do this research, and now it just takes them three to five minutes. They see concrete value in the product.
So, I find the results incongruent with what I think is happening on the ground, the feedback that’s actually coming from the customers. Now, I’m not an expert. I haven’t dived deep into the methodology. Of course we will look into it. We will see what there is to learn for us. We will see how we can partner so that these studies can become more representative, more effective, to bridge the divide between customer feedback and what we are seeing in that report. But it hasn’t diminished my optimism and my view of the efficacy of such AI models.
On a day-to-day basis, because of the study, we are getting more questions from customers. Not all customers, but a fraction of our customers, are saying, ‘Hey, can we take a step back and talk about it?’ And I would say it’s not just for us, but broadly for the market. It creates some questions about how effective are AI models in the legal space for all vendors. As a result of the study, the industry has to do a bit more proving to the customers to show that it actually works. The burden of proof has gone up a little bit. And that’s okay. In the long term, that’s healthy.
Ambrogi: Are you suggesting that Thomson Reuters itself will be providing more benchmarking information or that you’ll be supporting independent benchmarking?
Ramanathan: We may end up providing some benchmarks ourselves because, as a vendor, we want to do that testing anyway, and we have some rigorous testing, and we can make some of that public. But what I really meant is we want to encourage the formation of more industry-leading open consortiums to do it. To the extent that we can support such consortiums, provide them our tools, provide them any information, be transparent, that’s what we want to do.
I don’t think the industry is going to trust any individual vendor coming and saying, ‘Here’s my benchmark.’ I think there should be open benchmarks conducted by a consortium of respected universities, law firms, and law associations. I think that’s the way to go.
Ambrogi: To shift gears, I want to talk more about the market. I sense that there’s been a perception in recent years that Thomson Reuters has catered more heavily to the large firm market. That’s obviously where the deep pockets are in terms of being able to sell products. I know you’ve only been there four months, but do you have any thoughts on what segments of the market you need to be focusing on? Also, with specific reference to the solo and smaller firm market, are you looking to enhance how you serve that segment?
Ramanathan: Large law firms are less than half of our revenues. We can get you the exact numbers, but I think it’s about 40% of our revenues. The majority of our revenue comes from mid and small law firms. In a funny way, if I look at the conversations we’re having with customers, it’s easier with a small firm, because for them, the business case seems more straightforward. They look at the cost of hiring a paralegal to help them out with all these activities, and for them, it’s a straight saving to the bottom line if they don’t have to hire someone. So the conversations have been easier around generative AI assistants and solutions with the smaller firms. With the larger law firms, you get into a much larger-scale transformation. And the business case has other angles than just saying you’re cutting spend on someone.
I also think a key component of the puzzle is the government side, including the courts and the judiciary. One focus of ours is to make sure that the courts are not just keeping pace, but are perhaps even leading in terms of directing the legal community around AI usage. We’re having some very promising conversations here – there are some very promising conversations where the judiciary seems to be in some cases quite visionary and they’re willing to go that extra mile. When I talk about customer success, I also mean the judiciary, not just law firms. It’s also helping the judicial system and helping them to use AI properly.
Ambrogi: That’s interesting to hear. I’ve been at a couple of legal AI-related conferences where there has been a contingent of judicial officials trying to figure out how they can use this technology. One of the things that’s driving that interest on their part is the access to justice crisis and the fact that courts are overwhelmed by self-represented litigants. Is that at all part of the conversations that you’re having with courts and something you think that Thomson Reuters can play a role in addressing?
Ramanathan: Absolutely. It’s interesting because, from what the team tells me, before I joined, most of the questions coming to us from the judiciary were all defensive. How do I spot a hallucinated case? They were trying to react to the news. But it’s changed a lot. Twelve months forward and they’re now looking at AI as a driver of, exactly as you said, speedier justice, access to justice. It’s no longer defensive thinking. They’re looking at it much more proactively.
Ambrogi: Products such as Westlaw or Practical Law have, traditionally, been focused on helping lawyers get an understanding of what the law is relevant to an issue or a question. Is that the best use case for generative AI in legal? Or is it in more generative types of tasks like document drafting or summarization? Where do you see gen AI best serving the legal profession?
Ramanathan: It’s like asking which of your children is better? While technologically you need to have skill-based AI features like summarization, drafting, research, et cetera, from a practical perspective, I find that these use cases are blurred. A real-life use case needs a bit of research, a bit of drafting, a bit of summarization, a bit of document intelligence, a timeline, all of that. The family is going to win over the individual children.
Ambrogi: I know we’re about out of time. Anything else that you wanted to say that we haven’t discussed?
Ramanathan: Just one last thought to leave with you, Bob. Where we’re looking to develop our thinking is in how we convince individual practitioners – the associates, the lawyers – that AI is going to help them, and how do we help them embrace the change, as opposed to it coming top down in law firms. We’re going to spend a lot of time with our customers on these questions and challenges.