AI: It talks! A conversation with Johannes (Jan) Scholtes on what the legal industry needs to know about AI before embracing it.

Artificial intelligence and the emerging technology surrounding it (hello ChatGPT) is the tech industry’s next frontier. On this episode of The Legal Helm, Bim talks with AI specialist, Jan Scholtes, Chief Data Scientist at IPRO, a leading eDiscovery application. They look at how this new technology can be applied to legal, how best to incorporate it into your firm’s landscape, and where this new functionality is heading. It’s an engaging conversation you don’t want to miss. Enjoy!

Your host

Bim Dave is Helm360’s Executive Vice President. With 15+ years in the legal industry, his keen understanding of how law firms and lawyers use technology has propelled Helm360 to the industry’s forefront. A technical expert with a penchant for developing solutions that improve business systems and user experience, Bim has a knack for bringing high quality IT architects and developers together to create innovative, useable solutions to the legal arena.

Our guest

Johannes (Jan) C. Scholtes is the Chief Data Scientist at IPRO, which provides legal eDiscovery tools and software to law firms and corporations. Additionally, he holds the Extraordinary Chair in Text Mining from the Department of Advanced Computer Sciences at the Faculty of Science and Engineering of the University of Maastricht (Netherlands). There, he teaches courses on Artificial Intelligence and law. Jan specializes in text-analytics, information retrieval and natural language processing with application specializations in legal technology, regulatory overview, law enforcement, and intelligence.

Transcript  

Bim: Hello, Legal Helm listeners! Today I am excited to be talking with Jan Schultz, Chief Data Scientist at IPRO, which provides legal e-discovery solutions to law firms and corporations. As well as his role as Chief Data Scientist, Jan holds the Extraordinary Chair in text mining from the Department of Advanced Computer Sciences at the University of Maastricht in the Netherlands, where he teaches AI and natural language processing. And if that wasn’t enough, he also teaches legal technologies to lawyers, at the Ledon University. Today I’m talking to Jan about the application of AI technology in the legal space.

Jan, thank you very much for taking time out of your busy schedule to join us today. I’m really excited to have you on the show.

Jan: Thank you, thank you. Very happy to be here.

Bim: So, Jan, could we start at the beginning and have you explain your role as chief data scientist at IPRO and what that means and what do you get up to day-to-day?

Jan: That’s a very good question and a very good start. I’ve been involved with IPRO and also the company that was acquired by IPRO, XLA for almost 30 years. In the last 10 years my role was primarily to look further down the horizon and find new technology that was stable, innovative and that could help our clients. Typical e-discovery and information governance bottlenecks. This was typically technology that was brand new, state-of-the-art and would not be implemented in our product for the next three to five years, some even longer.

What is very important when you implement this brand-new technology, especially in the legal space, is to test this technology to see whether it’s stable, transparent, reliable… It’s very important that it’s legally defensible. I learned the hard way if you create black box technology and you use it in legal applications, that’s very dangerous. You have to be transparent and you have to be able to explain your technology. And most important, I believe is that you create trust with the end users: the lawyers, the legal professionals. They must trust the technology. It’s hard for them to fully understand the technology because they often do not have the mathematical or other skills that are required to understand it. But there are many different ways you can make them feel they trust it and that they understand what’s going on. Only then will they use it.

So that’s my role. Responsible AI, responsible data science, legal responsibility, making sure that we apply good science. We don’t take shortcuts and we understand what we’re using and what we’re selling.

Bim: Fantastic. Thank you for that. You mentioned a very key word which is trust. When you combine that with AI, especially now more than ever, it seems to be one of the potential challenges of AI and AI application. I’m very interested to get your opinion on that and what kind of things can we do from a governance perspective to build that trust with the technology to enable it to be as powerful as it needs to be, as a tool that we can leverage in the legal industry?

Jan: There’s a whole field called explainable AI. XAI: explainable artificial intelligence. That’s one of the most important fields these days. I’ve been in this business for almost 30 years, and 30 years ago, natural language processing didn’t work. We tried everything. The speech recognition, machine translation, the quality was not very impressive.

Today we have a different situation. For the last five years, great technology that outperforms humans. But we have a problem. We do not fully understand what these models know, what they don’t know, how they work, why they take certain decisions, why they do not take other decisions. This is why a couple of years ago we started a brand-new research field in artificial intelligence, called explainable AI. And explainable AI tries to explain how these models work, not in mathematical terms or in a hundred thousand-dimensional feature spaces because most people don’t understand that. But in a similar way to how we humans explain complex situations or complex decisions to each other.

For instance, if you go to, you go to a bank and you ask for a mortgage and you don’t get the mortgage, then the bank is not going to tell you how their decision process works. What they do is they take, for instance, a counter example approach and they say, “well, if your salary would’ve been higher, then you would’ve gotten the mortgage.” We accept that and we understand that. What we’re doing now is we went back to the psychology of explanation: How do you explain something to humans and how do humans explain complex situations to each other? Then we use these techniques and these ideas to create extensions to these very complex artificial intelligence models.

For instance, one of the things we can do is change parameters, change inputs, and then see how the system responds. Or we can use counterexamples. We bring it back to some really basic features. Instead of looking at a hundred thousand or a hundred million parameters, we just look at the most important parameters.

So we transform the space into a latency space with lower dimensionality. It sounds a little bit complex, but instead of looking at a lot of variables, you only look at the most important variables and how the decision of the model changes when you change these parameters.

It’s the like driving a car. When you hit the gas, you want to go faster, you don’t want to brake. If you hit the brakes, you want to slow down. That’s how these systems should behave. You change the input, it should change accordingly. That’s the start of people understanding a decision and then providing more transparency to make sure that the systems take similar decisions in similar cases. That creates trust.

You should be transparent as an AI vendor. You should not use a black box approach.

If you hide behind proprietary technology that’s not going to fly, especially in our space. We need to do this. And in Europe with the Artificial Intelligence Act, this is all about explainability and it’s about transparency and creating trust.

I believe that’s very good because I don’t believe in black boxes. I don’t believe in proprietary algorithms, especially not for sensitive topics like legal and also medical applications. There’s a ton of Hollywood movies made about what can happen if we end up in such a situation.

Bim: Totally. I love that link between the human psychology and the technology. That makes total sense to me. It leads me nicely to talk about the hot topic of the moment, which is GPT and ChatGPT and what that is doing to not just the legal industry, but pretty much changing the way that we think about artificial intelligence and bringing it to the mainstream. For the benefit of the audience, could you give your view and explanation as to why this technology from an AI perspective is so significant? And what are you seeing in terms of the impact of the technology in the legal industry?

Jan: Those are two very good questions. I actually wrote a couple of books and articles about this. You can find links to them on my LinkedIn profile. They offer much more detail.

We can talk about this for several hours. Let me start telling why this is important and why this was a watershed moment for us in AI, especially the natural language processing part of AI.

Like I said, for 30 years we tried to teach a computer natural language, human language, with translation, dialogues, synthetic analysis, semantic analysis, those type of applications. And we never really succeeded. All of these systems were sub-optimal. The reason they were sub-optimal is that they all took a shortcut. None of these systems implemented enough functionality to deal with all the special tricks and all the special aspects of natural language.

With natural language we start at the bottom. We deal with punctuation. That’s pretty easy for a computer to understand and we deal with words which we call tokens in computer science. On top of that, you have grammar syntax. But syntax can be organized in different order and you can have what we call ambiguity. Ambiguity is the biggest problem in language. Something can have multiple meanings. We humans immediately understand the meaning of a sentence. For instance, there’s a great sentence, “I shot an elephant in my pajamas.” The computer has trouble understanding whether I’m running around in my pajamas shooting an elephant or whether there’s an elephant in my pajamas that I’m shooting.

Of course, the last one is ridiculous. And because we all apply our natural linguistics, we don’t see all these crazy ambiguities that computers see. On top of that, if you’re going to look at semantics, the meaning of something, there’s even more ambiguity. And if you go a step further, like pronouns or co references, you see that natural language has a lot of relationships. Most of those are actually long-term relationships that could be in the same sentence, but one could be at the beginning, the other at the end, or other way around, Or it could be in other sentences or maybe a sentence, couple of paragraphs earlier.

Those long-term relationships we never dealt with. We never dealt with anything more than basic semantics. But in 2017, when Google published a revolutionary paper called Attention is All You Need, a five or eight page paper that described a mechanism called self-attention, multi-headed self-attention. And that mechanism was where the linguistic algorithm was able to find not only basic relationships, like punctuation and syntax and semantics, but also very complex semantic relations, like speech act, intention, pragmatics, and also certain long-term relationships.

This model was initially used by having an name encoder and a decoder for machine translation. The encoding was, for instance, the English language and the decoding was the German language. We build up to this model step by step, but this was really revolutionary. They called these models transformers. The moment these transformers were used by Google, the quality of Google Translate went sky-high. Also, the Google search engine became much better. But Google used it in a different way. They used it to understand what people were asking and then convert it into keyword searches, and then returned like traditional hits.

This revolution is ongoing for six years. It’s really interesting to see that Google started it and Microsoft is running away with it from a commercial point of view. The reason why this is so important and why I consider this to be a watershed moment is that these models are now able to generate human language. We can no longer distinguish from computers or from other humans, and that’s, that’s a major achievement. We were never able to do that. We can also have a natural conversation with a computer system.

You know the Turing test. I believe this era is almost the end of the Turing test because we can now have conversations with computers. It’s very hard to determine who’s the human and who’s the computer. That’s a major achievement. The other major achievement is that these models from ChatGPT are the first models that are staying within ethical norms and values, ethical boundaries, right? There were a lot of other models, Lambda, but also the model from Facebook from ADA and of course the old chatbot from Microsoft. They didn’t stick to ethical boundaries. These models do.

The way how we achieve this by a methodology which we call reinforcement learning. Reinforcement learning is the same technology used for AlphaGo which was used a couple of years ago. Another breakthrough from DeepMind from Google where the world champion go was defeated by these computer algorithms. What they did at OpenAI is they had humans chat with the computer for many, many months. Then the humans told them, “well, this is a good answer, this is a bad answer”. This was all stored in the model.

This is also what they used. You probably read a story about Kenya where they actually had people have really obnoxious discussions and topics that you don’t want to have people in the United States talk about in order to fine tune a model. This is a go and this is a no-go. Those are the two things that are really important. We can now make a model that behaves itself, doesn’t start talking about all kind of crazy things immediately. And we have a model that we can no longer distinguish from humans. These two problems we solved.

What we didn’t solve is this is just a language model, right? The way this is created is from the transformers. It’s not the encoder and the decoder. It’s only the decoder. All it can do is talk in a human way. It has no clue what it’s talking about because for that you need a encoder. You need to feed it, you need to drive it, with something which makes sense, which are the prompts. Microsoft took a step by now saying, “okay, we’re going to drive it with what comes out of our search engine.” They take text traces from the search engine and use that as a prompt. Sometimes that goes right, but if you do that in two long sequences after each other, you get these really interesting articles that we read in the New York Times this week about ChatGPT falling in love with the reporter. People are suddenly suffering from the Eliza effect and start assigning sentience and consciousness to these models, which they don’t have. It’s just a statistical generation process. So, we’re getting there. But these models just generate language. It’s very dangerous to use them in applications where you need factuality.

That brings us to the next question you had: What’s their future in the legal space?

Depending on what kind of text you want to generate, you can use ChatGPT as it is now or you have to wait for other models because in order to have these models stick better to the facts, we need to have something in front of it. We need to have either a knowledge base or some kind of graph neural network or some kind of semantic network that understands what’s going on and that understands, there are semantic relationships between all the objects you want to talk about. Then we can generate some really good text. The other way to get equality up is another example we saw last week with Harvey from Ellen and Ovary, where the model is trained vertically with a lot of legal information. I don’t know the details because they haven’t disclosed anything, but from the contract templates they have at Allen and Ovary and by training the model with this type of information, vertical training, it gets better at this type of applications. Because the original ChatGPT was trained with whatever stuff from the internet, including really bad discussions deep down from the dark internet. We made a major step forward.

You know, I’m super excited. This is one of the most exciting times in my life. Finally, after 30 years, this all works. We still don’t know exactly what it knows and what it doesn’t know, but we’re working on that. Now we need to find a way to make sure these models stick better to the truth, to the factuality. That’s why we have to make some major steps. Microsoft, in my opinion, did a very interesting experiment by integrating this with Bing. but they took a huge risk. They take text from Bing without fully understanding the meaning of the text and they use that as a prompt to generate a conversation. That’s where things go wrong. Suppose you have two different individuals with exactly the same name. ChatGPT and Bing are not able to distinguish them from each. When you search, you immediately notice that if you’re looking for John Doe and it’s not a John Doe you’re looking for, you have to look for the other John Doe and you’re going to add additional keywords to find only that individual.

Now that type of stuff is what Bing and ChatGPT cannot do. It’s very good and very detailed questions and it’s reformulating documents it finds on the internet. But within those documents, we should add an additional step where we structure the text better according to the meaning. Then it’ll be a killer application. I’m sure Microsoft and OpenAI are working on that.

I think for the legal space, we’re going to see a lot of these, let’s call them co-pilots, helping people draft contracts or draft e-discovery responses. Similar to co-pilots in the medical space and Microsoft already uses at GitHub, where they claim that developers have like 40% more productivity by using these tools to generate code for them.

I truly believe in the future of these tools as co-pilots, legal co-pilots, that are helping us generate templates and texts and then it’s the human who decide whether it’s right or wrong. There’s still too much risk that these models start to hallucinate and don’t stick to the facts given as an example like GPT being so broad in terms of what it’s been trained on and what it can do.

Like you say, focusing on a particular area of knowledge and training on a particular area of knowledge. If a law firm today is looking at implementing some level of technology that involves this to solve a particular problem from a practical perspective, is that the starting point to contain it so that it’s really focused on one particular key area and control the outcome as such? I think for me, the biggest risk is it can answer a question, but you don’t know whether factually it’s correct or not. Ultimately that introduces risk, especially when I think about some of the examples that you mentioned that are key in terms of enabling junior lawyers, for example, to use it as an aid as they’re learning how to be a legal expert, using it as a knowledge source and then being trained to use that as a basis of information that may not be correct.

Bim: Would you say that as a practical kind of journey, should firms be taking a step back and saying, “what do we want it to solve?” Then narrowing the scope of that? How do you think people should approach it?

Jan: Lawyers do many different tasks. Chat alone is not a search engine. An e-discovery for the large part is a data sorting problem. Information governance, the data part, early case assessment is also a data sorting problem. That’s not what GPT is designed for. ChatGPT is designed to have a conversation or to generate text. It could be great to generate draft letters or draft templates. It is already used a lot on the internet to generate content for websites or to improve search engine optimization. It is also used for writing sales pitches. If you have a product, then the system comes up with a creative sales pitch.

Now with sales pitches and marketing, if I’m a little bit cynical, you don’t have to always completely stick to the facts, right? There’s a couple of those applications in the legal space. If you’re a criminal defense lawyer and you have to be creative: Why was your client at that particular moment at the bank that was robbed and why was he holding that bag with money, well, he was just there and he tried to help the guy who was actually robbing the bank, but of course he’s not guilty. We’ve read these stories in the newspapers. They can maybe use GPT to create some creative aspects. Actually, there’s a company that’s already doing it, to help you to write a letter to object against the fine. It’s the same company that’s also wanted to use the lawyer in courts. They tried to actually have ChatGPT as a lawyer which I understand from reading that the US courts didn’t allow.

One of the great examples is write letters why you don’t agree with a fine and all those applications. You can already use ChatGPT but if you want to write a more complex legal contract, right now lawyers can use templates. GPT models are trained on texts like these templates. So, if you only train it on legal templates, it’ll actually find the statistical relevance of certain words and phrases and the context. And because these models are so long, they can recognize or remember a lot of context. It’s actually a complex statistical method to reproduce the templates. It’s a little bit more flexible than the templates.

The templates you have to look for a particular template check. GPT can generalize. If it sees all the templates, it can also create culminations of those templates. It can also start hallucinating and therefore you always need to have a human being to validate everything it comes up with. But the risk is with these models is that they generate this text with such a authoritative tone that you really have to concentrate to read it to see if it’s nonsense or not. So, there’s a big risk if lawyers are going to use this for contract generation. They are going to read over errors or they don’t see like more higher level intentional errors and that’s because the text is so perfect that you think, “oh, wow, the computer knows what it’s talking about.” I think that’s where it can help us. Maybe we’ll see similar productivity numbers, like with the GitHub. They’re co-pilots, but everything needs to be checked.

I do not believe right now that these models can be used as search engines alone. They can definitely not be used because they are not search engines. They also train with data until 2021, November, 2021. They have no awareness of anything happening before that. But if you combine it with a search engine, for instance a legal search engine or a case law engine, you can use it to have a more natural conversation instead of keying in keywords. You’re going to have a conversation with the computer now. What is really interesting here is it’s more like a philosophical question. Do you know what the average query length is for keywords from Google?

Bim: I do not.

Jan: It’s about 1.2 words. Most people find what they are looking for with 1.2 words.

I don’t know if you’ve seen the examples from Bing and from Allen Overy, when they ask questions. These are highly detailed questions where you really need to sit down and key in a bunch of phrases. I’m not sure if people are going to do that. On the other hand, the model works so well because the questions are so detailed, If you look at all the examples that are used by Microsoft and Allen Overy, there are examples with a lot of very detailed, very specific situations. [I’s like, a legal situation in India with something in the US and then maybe some GDPR in it. I’m sure that somewhere on the internet there is a document on exactly that case. Then it’s just going to reproduce that document by using the statistical models of the decoder and with the risk that it may also go off track, which we call hallucination. and that’s not going to work. We need to build a knowledge graph. Maybe a semantic network, a relation network where the meaning of all the words also in case law is represented, right? What’s the crime? What’s the applicable law? What does the law say? What is the effect? What’s the evidence factor? Are there special circumstances or other considerations? All of those are in a verdict and the words that are used in one part of that verdict have a completely different role than the words that are used. For instance, this is what the law says, because the law says it doesn’t mean that the suspect actually did it, right?

So, you need to understand what the role of those words is and then I believe you can drive the generational language much better. We’re halfway there and we are going to get where we want to go. Five years from now, 10 years from now, search will be very different, but we need to work on it because the current models are not reliable enough. The biggest problem is that people don’t understand why these chatbots in combination with search engines return certain answers. Microsoft includes a couple of citations, but not all of them. I do not believe that smart people are going to take a statistically generated answer for granted. They want to understand why the computer came to that conclusion and why the computer thinks this is the case. That’s something that hasn’t been addressed at all.

We’re taking a very big risk in the AI community because we’ve had AI summers before. In 1970 with the perceptron in 1989 with back propagation. But we also had AI winters. Really serious AI winters because in the summers we always blew up expectations and then the computers couldn’t do what we expect and people got disillusioned. That’s a big risk. Now it finally works. It can do a couple of things amazingly well, but we should be very careful not to now say,” okay this is ChatGPT, it’s going to solve all the problems in the world.” Because it isn’t. We should be very careful not to hype this too much and then create another AI winter for ourselves. That’s the biggest risk I think now for us in the science.

Bim: Agreed. I think it’s the start of a very long journey. It reminds me of when I was learning how to drive, for example, and getting my license and figuring out how to get from one destination to another. In my day, we used to use a map, right? A physical map to get from one destination to another. We didn’t have the luxury of, of GPS to guide us to different locations. I look at my nieces and nephews these days who have their licenses and they literally can’t get out of town without some form of electronic device to get them there. It’s pretty interesting to see, in the context of what we are talking about with ChatGPT evolving to a point where it is able to answer certain questions that we then become dependent on, particularly going back to that early example of the lawyer journey of a lawyer. They’re developing their skills, they’re developing their knowledge, and they’re starting to retain some of those things and then apply those principles to the practice of law. Do you see any danger there? Any risks of becoming too dependent on GPT being able to answer those rudimentary questions? That they miss out on an element of learning and growth as an individual who then be, is supposed to go on to become an expert in their field?

Jan. Absolutely. One of the reasons why we learn to write is learning to write helps us to learn to structure our thoughts. It is still very important to learn how to write, but there are a number of legal tasks that are done now by junior lawyers or interns which we are cognitively not really suited as humans, like a lot of repetitive simple tasks. A good example is reduction or anonymization. That’s something you don’t want to do manually. Same for finding for problems in a data room by putting junior lawyers with binders and markers in the basement and having them got page by page.

That’s not going to work. We see that in those type of applications, people maybe start with 80% quality, but very quickly they get bored and distracted and they fall back to 30% quality. For certain tasks where hundreds of thousands or millions of documents have to be reviewed or you need to analyze data or you need to generate a standard contract, these types of tools can help us. But for the strategic thinking, the tactical thinking, we still need humans.

I’ve been involved in helping law firms present how advanced their law firm was to a group of students they wanted to recruit. I was part of the recruiting process where they said, “In our firm we use technology for all the boring work and you’re only going to do really interesting strategic legal work here.” Of course, that’s only the case for the really top law firms. but for many law firms these days, it’s very hard for them to find people that are willing to work 1800 hours, billable hours a year doing boring work. I believe that the legal industry is the industry with the highest level of burnouts. I think we can help the industry by making work more interesting, helping them retain employees, and help their employees and partners do really interesting work and provide the service to the clients that has value.

Some of the biggest law firms do use technology and they are empowered by technology. The only reason why they’re not doing more legal work is that the National Bar Associations don’t allow them to do it. But that’s going to change. The whole industry, I believe, is going to change. We can benefit by embracing technology in a responsible legally defensible way and then become happier and provide better services. And at the end of the day, probably also make more money because a lawyer empowered with technology is worth a higher hourly rate rates than a lawyer who does everything with a fountain pen and manually in a library. I think more and more the top law firms already understand this. I think the mid echelon will have to otherwise I I don’t see a very bright future for them.

Bim: I remember talking to somebody recently who was trying to sell a paperless performer solution a managing partner. The reason why he was on the fence about buying the product was because there was no print button and he needed to print them out, put them on his desk to run through them one by one. The person who was trying to sell them the solution was like, “I think you’re missing the point of what the product’s about.” I think you’re right. If we can get them past that, that’s one of the challenges that we’ve seen in the legal industry for many years really. Although they may be adopting technology, the actual rollout of the technology at the lawyer level seems to be a barrier to really making an impact. I loved reading about what you do in terms of teaching when they’re at the kind of student level. You’re teaching students who are learning the practice of law about how AI might impact them and how the technology pieces come into play. I’m really interested to hear a little bit more about that. What’s the reaction when you are teaching some of those subjects? Do you see resistance? Do you see fear of this thing is going to take my job?

Jan: For technology to become adopted it typically takes 20 years. If you go back to television, radio, cell phones, personal computers, 20 years from the introduction to full-blown acceptance by the whole society. Because of the law’s partner model, it takes two times longer. It takes 40 years in the legal industry. It is changing. They are using text processors. On the other hand, I recently got my hands on a Word Perfect T-shirt, which I was very proud of. My children had no idea what it was. I said, “This is the most famous word processor.” I got it at Legal Tech New York because Word Perfect is still selling to lawyers.

Anyway, the legal industry is indeed a little bit slow, but you also see this in the legal education. If you look at universities, professors, associate professors, school professors, they are still very much of the old school, especially in the scientific academic legal universities. The students on the other hand, they are like, you know, digital natives. They work with technology all day and they do not understand why their curriculum does not involve more legal tech courses. Not courses on what the lawyer thinks of technology. There are enough courses for that, but courses where technology helps them as a legal professional to do their job better.

I was approached by law students who ask me, “Can I please do an internship at your company because I want to learn this. Because I feel that the skills I learn at the university are useless. I feel that I’m out of a job probably in 20 years from now if I’m not more equipped with technology.” That made me think, “okay, maybe we should start a course.” I actually visited a lot of universities and some of the bigger universities. The people higher up in the chain are very scared for this and they’re pushing it back. But the younger universities teach it. For instance, in Maastricht, we had a more than 150 law students signing up for this course. We’re now developing a master course, Responsible use of Artificial Intelligence. A lot of interest there but not from the professors, not from the deans, but from the students.

I strongly believe the reason why I started teaching, I only do it one day a week, is that, in our company, we were not able to find people with the right skills. The moment I decided to go to the university, I teach them, and then hopefully I can find employees with the right skill set. I’ve always thought from the interest of the student and the interest of what we need as a company, what would we need as a society in skillset? What’s interesting now is that the government lawyers in the Netherlands are really embracing technology because they understand it can make them more agile, more productive, help them to take better decisions. Because on the one hand there’s a lot of discussion about bias and algorithm, but on the other hand, computers are very consistent. If you address the bias, computers will actually have less bias than human beings and they are really pushing it. Where on the other hand, corporate lawyers and law firm lawyers, they are the slow adopters where you would kind of feel it would be the other way around. Typically government’s not the first adopter with a certain type of technology. but there’s other dimensions like, business models, and those type of things. But we see that it’s changing.

You need to embrace it as a law firm. Maybe currently your partners make a lot of money, but that’s not a very sustainable business model. At the end of the day, if you are providing a service that is too expensive, too slow, and not good enough, there’s never been a business model in the history of human beings that survived. Technology is essential. I understand that lawyers are afraid of technology or they don’t understand technology or that they don’t trust technology. My goal is to help them to address those concerns. I can help them understand it. I can make it transparent. I can help them trust it. I cannot help them changing their business model. I cannot help them if there’s another agenda. An agenda to be efficient, provide a good service at the right time, those things sometimes get intermixed. I always try to understand what people’s agenda is. People want to be helped. I can help them. Do you want to continue working as you work? Fine. You know, good for you.

Bim: I think the future is pretty exciting when you see that young talent coming up the ranks that are technology savvy and growing up in, in this generation of AI. I think it’s going to be pretty interesting to see what kind of problems they solve with the technology as, they kind of use their skills in that area. So, yeah, very interesting to see how that develops and impacts like future firm growth.

I want to switch gears a little bit and talk about IPRO. One of the products that I’ve read about is Live EDA (Live Early Data Assessment). Could you give our audience a little feel of what that’s all about and what problem it solves?

Jan: That’s a very good question. E-discovery is very reactive—or e disclosure as you call it in the U.K. It’s very reactive. You wait until there’s a situation and then you have this process. You need to follow the process and do it right. What we see is that all of these e-discovery tasks are done, data that’s out there in the company. There’s an organization you may have heard of, called The AIIM, the Association of Intelligent Information Management. Works very closely with AMA, which is a record management association.

I was on the Board of AIIM in 2009, 2010. We tried to propose companies and organizations to implement information governance principles If you no longer need certain data, get rid of it. The problem with e discovery is pretty simple. If you have a gigabyte of data, it doesn’t matter what’s in the data, you get a gigabyte bill. If you have a terabyte of data, you get a terabyte bill.

I wrote a post in block summary in 2012 with AIIM, where I said we have a new application of Moore’s Lowell. Instead of doubling CPU power every 18 months and doubling storage capacity for the same price every 18 months, we now also see that our legal bill doubles every 18 months. That’s, exactly what happens since those days.

At IPRO, we take the approach that instead of waiting for e-discovery, why don’t you use the same technology to go upstream and determine what’s out there, remove what you no longer need. If you do want to keep certain data for knowledge management purposes, fine. But make sure there are GDPR or privacy compliant redact or pseudonymized or anonymize whatever personal data is in there. Organize it, structure it so it can actually be found, and it can be reused. By doing that, you highly reduce the cost of e discovery because you see that only 5% of all the data you generate, maybe even less if you include duplicates, is relevant.

When I was at XY Labs, I was a strong advocate of that whole process. I think I even called it the dark side of Big Data in 2010. Companies were not ready for it. They said, “okay, we don’t have the money. (2008, 2009, big crisis.) Why should I invest in something that’s not going to give me a problem for the next five years?” Now we’re 10 years or 13 years down the line and data has grown and grown and grown. There’s no reason to throw it away because every 18 months our storage space doubles. It’s not like Iron Mountain, the warehouse is full and we need to start retaining data because it no longer fits in our physical space. The space is infinite. We need to start implementing retention management. And there’s a lot of data that you don’t have to keep, that has no value, that has no knowledge. It’s just junk. So why not throw it away?

Now with the technology that we have, the same technology we use for e-discovery, we can also use this on this type of data. What we see is that in e-discovery you have to follow a very rigid process, very strict process. There is no room for errors. You have to be very careful what kind of technology you use. You have to do it right in information governments. You have more flexibility, you have more time. The deadlines are not as strict as an e-discovery. If something goes wrong, you can fix it. So you can also take more risk with respect to the type of technology that you use.

On the other hand, you have to use more advanced technology because the data sets are larger even than at the discovery space. That’s what we believe in. Like I said, 13 years ago, the market wasn’t ready for it. Now there are many companies that are absolutely ready for it not only from e-discovery point of view, but also from a regulatory point of view or privacy point of view. We’ve all seen the fines that corporates and hospitals got for medical information flying around. If there’s a cyber breach, you’re in deep trouble. if there’s data all over the place, then you have to start informing all these people.

Next you get an avalanche of data, subject access requests, which are all like any discovery in itself. You know, disaster, serial litigation problem. So, be smart. Clean up the house. It’s not that much effort and it doesn’t cost that much. It’s a principle of good records management. We used to do that. You guys in England invented red tape. You have the public records office. You have very strict records management guidelines. Very good, but companies no longer have records management. There’s no records manager anymore. There’s no archivist anymore. So, record management’s everybody’s individual task. As a result, nobody does it. The government is still a good records manager so the government has much less problems with information governance. But corporate and especially fast grow and tech companies, it’s not that much work. The costs are minimal compared to the cost you’re going to run into when you are involved in like some serious e discovery or regulatory investigations.

Bim: I think we could spend a lot of time talking about data and data management and governance. It’s a whole topic on its own. You should come back for another episode where we can focus on data. I think that’s a big challenge and a big area that’s relevant to so many different types of business out there.

Jan, I must ask this question. Extraordinary Chair in Text Mining. That is quite a title. Care to explain that?

Jan: Like I said, 15 years ago we decided we needed better educated employees. So XY Labs decided to found the Chair for Text Mining and Search Engines. In the Netherlands, we called that an Extraordinary Chair. So, it’s funded by the company, but the chair’s completely independent and can do whatever scientific research it does. I’s fully focused on teaching and graduation projects, PhD projects, master projects… so that’s what I do one day a week. Over time, I also became responsible for information retrieval. And since last year, I also became responsible for the advanced natural language processing, for which I volunteered. Because when I started in this business 30 years ago, I started in natural language processing. I started in machine translation, speech recognition, and I got really demotivated because it didn’t work. I didn’t feel that the algorithms and the methodologies we used were ever going to work so I backtracked to search and information retrieval and text analytics because that did work. That’s what I’ve been doing for a long time. But now we see that suddenly overnight, last six years, these natural language processing algorithms finally work. And they do not just work, but they outperform humans. So I volunteered to fully embraced it, took over that course in the master last year. I have to say it’s great fun.

I was in the middle of teaching transformers and how they work in the mathematics behind the transformers in December when OpenAI released ChatGPT. There was so much excitement. I remember that Thursday morning it was released Wednesday evening and Thursday at the neural information processing systems in New Orleans. The next morning my students were all excited. A lot of noise and they were giving me demos. “Did you see this?” And we had a number of tutorials in the course where they had to program their own chatbots. Then in these tutorials, I would ask them questions why certain methodologies work better than others. They were actually answering those questions with using ChatGPT.

You may have read where I actually took the draft exam for my course and put it to GPT. It passed with a B-minus or A- minus, It was the most viral post I ever posted on LinkedIn.

So yeah, for us in data science and AI, these are great times. Now we have to make sure that it’s applied responsibly and that we’re not going to overhype it. We have to make sure that people understand what these models can do and what they can do. We all need to have a little bit self-discipline and hold back a little bit. I’m not sure everybody is capable of doing that. It’s super and I’m very happy to do this at university still. Working with students on one hand and then working with corporates on the other hand, you can make sure that things are done well in both worlds.

Bim: Agreed. Like you said, it’s such a fantastic time to see all this stuff evolve. I do believe that education is key. It’s great to have people like yourself sharing your knowledge and experience to also contain the beast and make sure that we’re doing it at the right pace in the right way.

I have a couple of wrap up questions. My first I ask all my guests: I you could borrow Dr. Who’s time machine, I think they play Dr. Who in, in Netherlands, if not, then think of Back to the Future instead. if you could go back to Jan at 18 years old, what advice would you give him?

Jan: I build my own Macs, I, I love pinball machines. So, I wouldn’t do anything different. I have no regrets. If I’m allowed to go forward, that will be interesting. I think these times are so exciting. I don’t want to miss them. I tell my students how grateful they should be that they are their age in this moment in time. I would love to be 25 again and be in the middle of all this. Because what’s going to happen next now we’ve solved this language problem? You may have heard of Whisper, which is OpenAI’s speech recognition. You know, Dall-E, the image generation. Now, what’s going to happen in the next version of GPT? That’s all going to be combined, like it’s combined in our heads. 2023 is going to be a super exciting year in AI.

Bim: Really exciting. I agree. Jan, any closing thoughts or advice for legal professionals in our audience?

Jan: Don’t be afraid. Embrace it. You need to understand it. You have to understand it. Don’t accept technology you don’t understand, you don’t trust, but look for the right sources to help you to understand it and to trust it. Don’t take the ostrich in the sand approach. You cannot ignore it. If you’re like 65 years old and you’re going to retire very soon, then you can probably continue working for another couple of years the way you always worked. But if you want to stay relevant and be relevant 10 years from now, you have to get yourself up to speed and there’s more than enough resources.

Also, at Oxford University, Cambridge University, and in the United States, Georgetown Stanford, Berkeley, and MIT are providing excellent education. A lot of these courses are open courses that you can follow. If you happen to be in the Netherlands, Maastricht University or great universities where they can teach you this kind of stuff.

Bim: Fantastic. Great advice and tips. I really appreciate that. Jan, how do people reach out to you? Contact you if they want follow up with you.

Jan: LinkedIn is the easiest. Just Google my name. I’d love to engage with you.

Bim: Wonderful. Thank you again for your time today. It’s been fascinating talking to you. I hope you do come back for another episode when we can talk more. I think we could have spent another hour talking about this stuff. I love learning from you and hear hearing this stuff.

To my listeners, if you liked what you heard today, please like, subscribe and spread the word. It really makes a difference. Thank you very much.