Thank you. Good morning, or good afternoon, everyone. Sorry, it's the afternoon already. So I'm going to get started. I'm Melissa Hollis. I, as mentioned, worked at Principal. I work at Principal. And I'm going to talk about AI attribute intelligence. And before I begin, I wanted to read the fine print, and that is that the views expressed here are those of the employees. Us, and not necessarily those a principle. We have a short agenda today. I'm going to start us off setting the stage. Then Zach is going to talk about the solution and walk through that. And then Justin's going to finish it up by talking about the value of scaling. So with regard to setting the stage, I thought of an experience that I had recently, and I thought I would share kind of what that looked like. But I was planning a road trip, six days, five nights, three states, and a part of that is really finding all the restaurants along the way as well. So I wanted something that was somewhere between Chicago and Two Rivers, Wisconsin. So I found this really interesting place right off of I-41. It's called the Mars Cheese Castle. If anybody's ever been there. Okay, excellent. Awesome. I thought it was a really good choice. It looked really interesting. I clicked on the link for the menu and it took me to a PDF. And this PDF, the first part of it was an order form. And it suggested, download the form, print it out, fill it in, fax it back to them. Call with your credit card number to confirm, and then your order could be picked up, because I was really looking for a place where I could order ahead of time and then just pick it up as we drove through. And I thought, well, this probably is not going to be the best option for me, because I don't think that I have ever faxed anything in my life. However, I am very familiar with food delivery apps, right, which are much different and have a much different interface. So in a food delivery app, a lot of the decision making or the thinking is done for you. There's a lot of constraints on what you can and cannot put into those apps. So you often will pick out an entree, and there may or may not be check boxes to, you know, add extra sauce or remove the onions or something like that. Sometimes that decision is not available to you, and so that's what you have to go with. This is much different from the Mars Cheese Castle, where you could put in, you handwrite, you know exactly what you want, maybe what you want to add, what you want to remove. But with that comes a lot of interpretation on their end, right? They're looking at your facts that has come through, maybe deciphering your handwriting, figuring out exactly what you want to order, and how much that's going to cost if you've added a lot of things to that. And then eventually, it will get into their system, go into the kitchen, and then get completed. So 2 very different experiences, both from the front end and the back end of this. So then I thought about it more, and I thought about how a lot of the data that we receive as an organization looks a lot more like that fax than what would come out of the food delivery app. Right? We don't have a lot of constraints maybe to what information is coming into our walls. And a lot of that is because we also want to be really good a business partner. We want to be good business partners. We don't have a lot of requirements on how you interact with us. We just want you to be able to interact with us in whatever capacity is possible. So we do, and I'm sure many of you also receive a lot of external information in a lot of different ways. It can be in a spreadsheet, a Word document, PDF, email, text. It could be a spreadsheet inside of a Word document. It can be all sorts of different ways. But you're really then having to take that additional time to really interpret and translate that information that you've received in order to fit it into the systems that you're using. And so although you may be receiving information in different ways, there's often generally some patterns in what you do receive. So I'll use as an example, like date of birth. This might be a column that you're expecting to be receiving, and it may be called date of birth, it might be called DOB, it might be called age, and it might be a month, day, year, or a day. year, month, day, year, either way, you get the point. And it may just be the age. So there's a lot of different ways that it may come in, but in general, there's some patterns that you're able to look for. And ultimately, then you have teams or individuals that are reading this information, they're translating it, they're maybe remapping it, and then eventually getting into those systems. This process is not something that is easy to pick up, maybe. There's a lot of nuances in what is happening. It also is prone to errors, and ultimately it does not scale very well. So this can look a lot of different ways from the outside. It's often considered maybe a processing delay because you have the information, why can't you just process it faster? But really it's often more of that interpretation or translation delay that's causing the delay. And this is not a problem just for principal or for our industry, for financial services. I believe it happens everywhere. Anywhere you have information that's coming into you from an external partner. It can also even happen internally if you have systems that don't talk to each other well, anything like that. But anytime where there's a lot of data coming in, There's the potential for this translation or interpretation problem. So then that begs the question of why are humans still acting as translators between messy inputs and rigid systems? And to answer that question, I'm going to turn it over to Zach, who's going to talk about our solution. Thank you, Lisa. I hope we're hearing my voice. And the clicker is working. Sweet. Melissa gave a great example, a real-life situation of when messy inputs lead to downstream delays. The solution we're going to speak to today is kind of a framework, a tiered engine that's able to take a variety of inputs. It really doesn't matter what type of file type that we iterate on. and improve over time through various use cases to try and tackle this in a way that is augmenting that interpretation layer. We're not trying to completely remove that judgment from the human per se. We still need to make sure that the inputs are correct before they make any changes or affect a customer. But we can really augment that with a stack of technology. When I use the word chaos here, it may be a better synonym is compounding variability. For my stats nerds out there, each one of these different parameters just adds degrees of freedom for the system that make it more complex. It compounds with each different type of file. That you're trying to use different structures, different formats, all those things are just making an exponential problem, because every time there's a variability in any one of these, it creates that dimension of difference across all of them, so we kind of have to take these one step at a time and kind of attack them in a way where... Not only do they impact each other, but they're all kind of handled separately, so that depending on what type of file type it is, what type of data attribute we're working with, we understand it throughout the entire flow, and so that we can get the correct values. Overloaded values and context is another tricky piece of the chaos. Overloaded values is when Maybe a specific data attribute, I think age and date of birth is a good example, where you can have multiple pieces of information within the same data attribute. So somebody may give you their date of birth, you require age, it could be a sentence in an unstructured document that contains multiple pieces of information where context comes into play. on how that should be interpreted. And we kind of have to juggle all of these things before we even get to the data itself. So we kind of split apart the problem and tackle it with specific tools for each of them before we even do any of the extraction. So this is where we start to talk about the intake of documents. This is unglamorous and boring work, honestly. I mean, whether it's an Excel file, PDF, JSON document, text file, image, God forbid, you really need to make sure you understand what type of file that is before you start anything downstream, because that'll affect what type of modules or processes you apply to that specific document to do the extraction later on. I mean, when it comes to PDFs, is there certain types of scraping or pre-processing we have to do? If it's semi-structured documents, even Excel documents, are we having to handle multiple sheets within an Excel document? Are we having to find that a table is not in the top left corner of your spreadsheet for some reason a customer has given it to us and it's in the middle of the document for some reason. Is it an embedded spreadsheet within a Word document? These are some of these structural manipulations that need to take place before you even start to look at the attributes and start to look for extraction patterns. And all of them are kind of separate. So, like I said, this is very iterative. When we take on use cases with this framework, you know, you may have an understanding of the universe and what types of files you expect to receive. That could change over time. And so you want to make sure that you're building upon this framework with new use cases so that as you take on additional ones. You've kind of built up the knowledge base to handle a wide variety, check that file type, handle it appropriately, and kind of save that off for later use. But then we move on to the extraction page, which is where I'll be spending the most of my time. We created this tiered engine. I think of it as a cake, think of it as anything that's tiered. It doesn't really matter. But the idea here is to break apart, like I've said multiple times, individual components that can be handled by different tools and bring the right tool to the problem instead of a shiny hammery. LLM AI model to do all of these things, really fine-tune the tech stack so that it's handling the right problem with the most efficient, cost-effective, risk-mitigating set of technology possible. What this looks like is kind of taking a lot of the behind-the-scenes things that you see when you Use documents in your favorite chat bot or co-pilot, and kind of breaking them apart and doing them sequentially and using the less sophisticated, simpler technology when you can, and only relying on some of the more sophisticated, high-cost, risky stuff when it can't be solved by some of the more simple stuff. And so document and text operations, think of PDFs. Maybe it gets overlooked quite frequently, but there's a lot of formatting or metadata within a PDF document. Think of things like bolded text, color, location on a document. These are all things that could indicate what the data is. Is it useful? Is it the type of data input I'm looking for? And getting using this text operations is to kind of get that structure right, so that you at least are in a good place to go ahead and do some of the actual extraction, which is the following ones. So basic natural language processing, this would be like pattern searches. You know, if you're looking for a phone number, at least here in the United States, it's maybe 1 digit, but then we can expect 334. with some maybe hyphens in between. Some of these regex patterns, really straightforward, programmatic, deterministic, right? If I see that pattern, I'm pretty darn confident that it's a phone number. And so you're kind of taking care of some things that you can with really straightforward processing. You can also, like I said, lean on some of the metadata you got from the document operations. If you understand a given form, it is set up in a way where you have the definition of the attribute, so date of birth on the left-hand side, bolded, and then maybe the actual data point itself on the right-hand side, or if it's a spreadsheet with your columns and rows. You can use some of the basic document and text operations to inform the natural language processing. If we can't quite get, and maybe I'll breeze over real quickly, but you can also do some dictionary searching and mapping for keywords and phrases. I'll gloss over that because the next slide dives in a little bit deeper. But if you can't get through it. get to it with deterministic or straightforward programming or code, you can take maybe one layer of sophistication down into machine learning. Here, maybe your attributes can come in slightly different than expected, date of birth, DOB, that type of thing, slight variations of an attribute first name, F underscore name. Those slight variations are semantic nuances within maybe the attribute name, but they're close enough that a machine learning model would be able to pick up on, all right, based on my training data, I would expect that this unseen, unknown data attribute aligns to the class of data attribute that I'm trying to detect. This is the next step mainly because it is something that we can train ourselves, understand since a machine learning model. We have accuracy metrics. It's probabilistic in nature. It's not just outputting tokens. It is making a prediction. And that's something that we can monitor and track over time for performance. It's something that we can also use in a feedback loop. And I'll talk about that in the next slide as well. Named entity recognition is an unnatural language processing concept. Maybe the type of attribute you're going for is a type of thing, an organization, a person, a location. Those types of attributes are things that machine learning models can predict as well. Specifically unstructured text, you can look for that type of part of speech within a sentence, pronouns and whatnot, and you can use that to inform what types of data attributes would align to that. If we can't do it with those things, yes, we do need to lean on the shiny hammer, specifically when you've got sentence structure with some sort of logic to determine the data attribute and you can't grab it directly from the document itself. That's when we leverage a large language model. We'll talk about this a little bit deeper in the next slide as well. But breaking this out again and not just relying on maybe a chat bot to do this, we have complete control over how we use the large language model and RAG architecture. So what type of chunking or pre-processing should we do To a document, before we even ask that question of what data is in this document, we can save money and time, which I'll talk about next, and then calculations. There may be an instance where you've extracted maybe a set of different attributes, but the actual true data point you're going after is a calculation between the different attributes. That's something we'd also handle outside of the use of large language models. Very key here at the bottom two, once we've run through all this and we've gotten all the attributes we can from a file, we really want to do some data validation steps. For example, age can't be negative, right? So you can do some of those really straightforward things to make sure that after I've identified where that attribute is, Pull out the exact values themselves - do they fall within our expected values for that given data attribute? You might be saying, again, great, why not just use the strongest tool for all of these things? And I'll maybe make that a little bit more clear with my double click down into some of these things. Dictionary intelligence falls into that bucket of basic natural language processing. It's something that is really informed by domain and business knowledge. So this is interaction with stakeholders who have been maybe doing some of that interpretation themselves of, okay, here's the form that came in, here's the data I need from it, here's how it needs to look for the system. We can take some of those known terms known data attribute names and turn that into a dictionary. And that's a living, breathing dictionary where we use that to curate new examples of potential values within that dictionary to search for and have that feedback loop to say, is this a good one to include, not include? When we include it, is it getting the right attributes out? Is it not? It's trackable, auditable, all that stuff, and really high confidence here, because we're just saying, "Hey, here's the list of different attributes for first name that we've seen. Here's is it in this document? If so, go ahead, grab it. If it's not, go ahead and try and use some of the more sophisticated techniques. We use that. dictionary to then train our machine learning model. So we give it a list of, here's training data, here's what the different variations of first name are, here's a set of variations of first names you've never seen before, and a whole bunch of different data attribute names. Try and predict the data attribute names that align to first name and get better and better at that. This is, again, good. because it relies on more mathematical probabilities. It's really straightforward. It's converting that text into an embedding. That embedding encases some of the semantic information around that text and can carry that over into training. You can also use voting mechanisms here where multiple of these really lightweight, cheap, we're not paying for tokens, we're paying for a small amount of compute to make a prediction. We can use multiple of these lightweight small language models or even just machine learning models on embeddings to vote for which data attribute and then aggregate those predictions together to be very confident and reduce the risk of getting Getting misses. It's also a confined scope. When you set up these machine learning models, they're just making a prediction from a set universe, so it can't go off, maybe as some of you have experienced with chatbots, and start just generating completely unrelated text, right? It has to predict within a specific set of data attributes you are looking to find within a A given form or document. Lastly, I kind of hinted on the last slide that we Lee on these, but again, and maybe another reason why this approach is risk adverse, efficient, effective, as well as makes a lot of sense from a cost standpoint. As we heard the keynote speaker start to talk about cost of using these tools in the future, may get kind of crazy. Even when we lean on these tools, we can do so because of the pre-processing and other tech stack pieces we've used. We can do so in a smarter way, right? Instead of sending that entire document to the LLM and say, hey, here's the data attributes I need. Well, I've done that parsing and chunking to realize maybe instead of this 300 page abstract that you need to now use as input tokens, I just need a paragraph from that, right? Because I've identified here's the keywords or phrases, a portion of that document that correlates to the data attribute that I'm looking for. That's where that keyword searching and some of that document processing on the front end comes. Kind of comes comes back in later to interpret. We also have, you know, strategies around which model we use can be lightweight and cheap, even from an LLM perspective. Question answering and rag solutions don't really require the latest and greatest machine or large language models, and we have parameter selection. going on as well. Chunking and different rag implementations allow you to kind of manipulate how much overlap and how much different sections you want to look at within a document. Again, we kind of try and avoid that by looking for the specific section with the document that would make the most sense to find that data attribute in. And then your your stereotypical prompt engineering and instructions you can provide to the model. Still here, very crucial that we want to make sure that we're validating any of the data that we pull out at the end of the day before providing it to a human for review or a system for ingestion. I do want to call out these important considerations. I just did one of them at the end of that last slide, but we're making sure that human is involved. Feedback loops on the beginning, making sure that we're using their domain knowledge to guide our dictionaries, which is really the cornerstone of the rest of the tech stack. And then we want to have validation and quality checks. Going on for some of those edge cases or examples of ones that do leverage maybe the machine learning or the large language models, we want to provide some level of confidence score accuracy in the past for that data attribute on a source of truth. So they kind of have a feel for how much they should rely on or just take this as what it is instead of maybe reviewing and going back to that source document and validating themselves. We do have defined boundaries on all of this work, privacy considerations as well, making sure that we're not using anything outside of the bounds that privacy and risk and governance would be upset with us about. Making sure that anything's redacted or filtered that shouldn't be used. And making sure that any data that we have access to is getting removed from people's access when it's unneeded. And of course, it's a highly governed and regulated industry, so we have a bunch of risk assessment, validation checkpoints, and making sure that we're following those governance standards. I spoke a lot for a long time, but with that, I'm going to pass it to Justin to talk about the value of scaling. Appreciate it. All right. I am going to beg forgiveness. I woke up with the worst sinus infection this morning. So we're all going to walk through this together. I'm going to do something I promised I wasn't going to do, and I'm going to basically read from my presentation, but just give me a little bit of freedom. It'll be like the Michael Jordan flu game of reading an AI scaling speech. We'll see how it goes. So we've also talked already a little bit about scaling in general. I mean, we're no Mars cheese castle with one type of ordering process, but Principal has lots of different challenges. We're a large federated organization with lots of different data sources, and we're known to be easy to do business with. for good or for better or for worse. As Melissa mentioned, we generally accept all incoming data. And we need to be able to make the data ready for the company to use in the best way possible. So that's where this process comes in. That being said, we click on the notice, you know, one, this is a one-off problem. This happens across the area. So every one of our BUs has those, business units has those problems. Different teams, different data sources, different business needs, the same translation work is showing up over and over and over again. So that reputation, that repetition showed us that we, this isn't just a one-time use case problem, it's a gap in our capabilities at principle, which we acknowledged. And it's important to know that we, here, we just want to predict pause. We don't use AI just to use AI at principle. But when we see that, we say, We use it where it's appropriate, and it's where it speeds our processes, and again, with governance and human in the loops all the time. So instead of trying to solve each request independently using my esteemed colleague Zach Diger's process, we shifted to solve this problem at a bigger scale. So how do we do that? What it really became was moving from individual solutions Doing this every time and doing this process every single time to really working towards a shared attribute intelligence capability, something that the teams can reuse rather than rebuild. So, what's in the shared attribute intelligence capability in that platform? I can't give away the entire secret sauce of everything that we do. But it involves shared workflows, standardized codes, reusable attributes, and lightweight AI agents that help bind it all together. A part of standardizing the workflow is not just the data itself. We're standardizing how the attributes are defined, extracted, validated, and measured, so new use cases can move forward faster without sacrificing consistency. What that leads to is the shift, the teams where they spend their time. It doesn't remove anyone from the loop. Just less effort goes into the one-off mechanics and wiring of every one of these processes. More effort goes into data quality, confidence, and reuse. And those are really the things at principle that go to scale. Those are the ones that provide value at scale. So the key idea here is simple. We're not just solving single problems, we're building a repeatable pattern that makes the next 10 problems easier, safer, and to handle. So just to go where we're at, one of the biggest limitations we saw is just how long did I add anything new to the process. A small new attribute, a new field. In the beginning, every new attribute we added to the process turned into a small engineering and ML project, which slowed everything down. What we're doing here moving forward as we scale it is we're trying to change that model. The new attributes are really added through configurations, not custom builds. That means we're defining what the attribute is and how it should behave without rewriting the engine that we're using every single time. So, like just an example, the old way, you know, to add payment frequency, we would have write new parsing logic, we'd have new regex or ML code, we'd create new deployment, we'd add new tests, new maintenance paths, we'd revisit the code again when there's new variations that appear. Every attribute became a mini project. When we're looking at the new one. We're really talking about configuring the new attribute. We're looking at, so if we take payment frequency again, we're talking about all the allowed values of payment frequency, monthly, quarterly, annual, singles. And we're talking about also adding in what are synonyms for that? What are signals for that? What are signals for that type of variable? So build monthly. Annual premium at principal, there's a myriad of ways to talk about how premiums and payments and stuff like that, so we always make sure that we have all the cinnamons available. The extraction rules, though, we also look at that too, so he was talking about that here, and this one was looking at the various places that it could exist, and that's going to be based on... Patterns that we know already that exist, so it's going to go through those various patterns and see which one is the best pattern to look for for that particular one. So it's going to look in structured fields if it's present there first, then it's going to go to the free text sections and add it. It's all based on what we know has happened in the past with those various fields. It's been part of the engineering that goes into it, and then... For each of those ones, we're also going to have the confidence and the validation tests that go into that one, so they're automated and ready to use for each one. And then we're also going to flag ambiguous cases for review, so we can spend more time looking at those types of cases than we do engineering in the 1st place. So under that new scaling, we're not making a new pipeline. There's no new deployment. There's no custom code path unless it's truly something that we've never experienced before. It's the same engine that handles policy type and now handles payment frequency because we are telling it what to look for and not how to rebuild itself. A key part of that is every attribute follows the same definition pattern. We're being explicit about what we're detecting, how it's extracted, how confidence is measured. The consistency there is what allows us really to scale this whole process, not just given the speed. And I also wanted to make sure we reference this again. We're very intentional about using rules first or your ML first, we will make sure that that's built into the whole process. At the heart, we really want to follow the guide path that Zatkalik laid out. If something can be handled deterministically, we're going to do that. AI really is only layered where there's vulnerabilities or ambiguity that requires it. And it keeps everything really predictable, and it controls for risk. Also means scaling. When we talk about this one, we want to scale this one. We want to make sure that this one works both across structured and free text data. So we have the same process that works for both. And that's part of our scaling of this one. So we don't need to have different approaches depending on the input type. Same engine, same guardrails, same onboarding motion. So the end result is that we add new attributes, then it becomes fast, repeatable, low risk. That's what makes it usable at scale. One of the things we wanted to make sure when organizations try to move faster with data, control is usually one of the first things that gets compromised. People skip validation, they introduce special cases. They very frequently hard code exceptions just to get something to go live. We want to make sure that we're not doing that in this instance. What we built intentionally is designed to avoid that trade-off. When new data source comes in, it follows the same repeating path, same onboarding path. Consistency really allows us to introduce speed without introducing chaos in this whole process. We have a very intentional governance privacy policies that we follow. We have a very intentional data validation process that we follow, and we make sure that whatever we're doing to scale doesn't change that. So Is that instead of building new pipelines every time, we're just reusing what already exists? Um... This means that we have created expansions that require very little new engineering, even as the volumes and complexities of the data gets increased. And we just make sure that we're putting the control baked into it, not adding it later, all that stuff, the current confidence scores, it's all part of standard flow. practical outcome for us is that we can really move a lot faster with confidence. And we adopt new data sources quickly, scale the platform over time, which is what we're doing right now. We're still in the middle of kind of scaling the whole thing. And we move, we maintain consistency and trust with our stakeholders. It's really designed to move fast, responsibly. That's where I'm at. Need to sit down. Awesome, we started a little early, but we did want to leave time for any questions deliberately, so I'll stand up, but... Appreciate it, and yeah, open for questions. This isn't a question to say, you don't just have to experience it. Well, yeah, and I do want to be clear. I am still planning on going to the Mars Cheese Castle. It did look really tempting, and I've been thinking about cheese curds ever since. Any other questions? Behind you. So, let's do it here. Do you process handwritten notes, do OCRs? Yeah, yeah, good question. I didn't go that deep into it, but that check file type you have, it's an image, we'll do OCR for sure. We haven't had yet a use case of handwritten notes. They're out there for sure, especially if you think about our claims area with doctors and stuff like that. That would be a bridge that when we reach it would be more challenging. But yeah, if it's an image, we will do stereotypical OCR to get the text off of that image. Thank you. Yeah. So, maybe I enjoyed one. Doesn't feel the full picture here, but you guys get in. Resume a lot of. Customer submissions, and... You're filing all those submissions through this engine to process all that data. Is that? Yeah, I think maybe a good way to think about it is there's bespoke siloed areas that are taking different types of customer inputs, having a human Maybe they have an Excel macro that does this a little bit or something like that. Or maybe they are truly pulling up a PDF, reading through it, pulling up an Excel file next to it, and making a spreadsheet live that then goes downstream for further analysis. So the idea here is to create a framework that can kind of adapt to each of those different types of use cases. and speed up that and kind of augment is the word I think we kind of like to use, that interpretation step. So it's not just that manual. It has a level of technology behind it that helps speed that up and kind of moves us ever so slowly as confidence gets higher to what you're kind of describing where. All customer inputs for a specific function are getting handled almost automatically, yeah. How do you handle the types of PDFs? Do they come from different sources? Yeah, there's a lot of good, actually surprising, open source PDF manipulation tools, readers, and we experimented with a wide variety of them. Kind of have two that we really love. I'm sorry, I'm not remembering off the top of my head. But yeah, we, Python code does a lot of that work, really. Open source frameworks for scrubbing, like I said, some of that metadata off the PDF documents. If I've identified, it's a PDF. But you're exactly right. Some, like if you open a PDF, sometimes you'll see some. You know, you can click a check box. That's a different type of PDF. Or you can highlight the text or you can't highlight the text. All of those are little intricacies that we try to do the most least sophisticated thing. If it's a true PDF, we can use those open source packages to scrub pretty much everything off of them. Thank you. We don't have any other questions. Any other closing comments for us? No, I just appreciate the time. I will, I think we'll all be at the facilitated discussion potentially. Come with any questions that maybe didn't come to mind quite yet. Yeah, I appreciate being here and for all of you giving us your attention. Thank you.