Good afternoon, everyone, and then occur. This is Lee Wei. So, we both thought that we were going to have forty-five minutes for the session. Turns out we're going to have only half of that, so we're going to go real quick here. Hope that's OK. So we both are part of LSS, or Life Care Services. We are an operator and owner of senior living communities. We operate around 120 senior living communities across the country. So as you can see here, we have a breadth and depth of the services that we provide for our residents and our communities. Today we're going to focus on what we do from a sales and marketing perspective. As you can see here, we operate around... 120 communities, that's a lot of prospects, a lot of sales counselors, lots of conversations. So essentially a lot of nodes in our CRM, which is Salesforce. So whenever I say Salesforce, just think of it as your CRM system. So as a company, we have four key strategic priorities. As you can see, the innovation and AI is a key priority for our company, just like I'm sure most of you. This specific capability that we have built really falls into The AI foundational, high-level imperative. I want to spend some time on this slide. So this is our typical prospect journey. That 3 to 18 months that you see here, that's not a typo. That's how long it takes. Sometimes it takes a few years, because this is a key decision that someone takes in their life. So, essentially, what we sell is, it's not a skew, it's not a subscription, it's really where... A prospective resident goes to spend the next chapter of their life, so it's a big decision, right? So that carries a lot of weight. So Salesforce often has the information, but understanding that takes a lot of work. So just like I said, because the sales cycle is long, it might be two, three, four months since we have talked to that specific prospect. So as soon as they call up, our sales counselors or sales reps have Maybe 10, 15 seconds to really refresh and retrain themselves about that relationship, right? We have the data, but how do you make sure that's useful? So this is a real record from our CRM system. What you see here is all these different nodes from our different activities and conversations, right? And this is actually typical for our sales force. A lot of our prospect records, we have the information, as you can see, But it's really wrong, right? I mean, how do we make sure we can make this useful for our sales counselors? So the key knowledge lives in notes, but it's scattered over a period of months and years. It's written by different people with different types of writing and verbiage. Some just say LM for left message. Some actually write two, three paragraphs, right? So, it's different. So, how do you make sure we can make this searchable and quickly understandable for our sales trips? So our mantra is a salesperson should be able to get to and understand these notes in seconds, not in minutes, but in seconds, right? So typically, yeah, it's our average number of notes for each record, each prospect or account, as the case may be, it's around 47 nodes per account on an average, right? And 15 seconds is not enough to get there, unless you are like a speed reader in which we are hiring, so please apply. If there's one thing that we want all of you to take from our session today, it's that this is really, this isn't about AI. Now, with due respect to the organizers here, this is much more than just AI, right? So this is really a cross-functional initiative. So between our sales folks here, the sales counselor, sales leadership operations, our data and technology here. So I'm part of the technology team. I lead the CRM at LCS. Levi is part of the data science team. We had a lot of other teams. This was truly a cross-functional project. And we involved even our sales folks From day one, and we'll go through that, really trying to understand what's benefit, what's beneficial, what is really helpful, to understand from their end what is helpful, rather than what we thought was helpful. So, from a feedback process, we really started out with... Really showing what the possibilities are, we actually... gave three different kinds of, hey, do you want to have a heads up display or do you want this kind of three paragraphs of summaries or something in between? So we got that feedback. And this was way before Levi and the team actually started creating the model, right? So once we did that, We showed some of the results, and this is where we want to really highlight some of this that you don't have to do everything, build everything, and then show. We actually showed some of these results from the model within spreadsheets. Hey, this is how it's going to look like. Yes, at the end of the day, we will have it in your CRM. You can see it just like you see the prospect record. But we wanted to make sure what the model is giving back was accurate and complete. So for that, to get that feedback, we just used spreadsheets and showed them. We don't have to go all fancy. And then finally, trying to understand, hey, what are the specific data points or piece of information that would be beneficial to them? So it's a two, three sentence summary, for sure. But then we also extracted certain specific information, like financial information, what are their personal details, constraints, who are the decision makers, et cetera. Now think of this, right? Going back to that slide with the 47 notes, it's not easy to get to this information quickly, right? So this is what we are giving them. I'll turn it over to Levi to talk through how the model was actually built. Thank you, Jennifer. Okay, real quick, about more of the technical aspect of the project. So we have a problem. It takes too long for our sales reps to understand a lead. There's too many notes. It's difficult to pull out important information. We have, we know that AI is very good at taking large amounts of unstructured data. and summarizing it, right? And we have our requirements. We met with our sales experts at the beginning to get exactly what they're looking for in a summary. So around the problem, the AI solution and requirements, we built this pipeline. Some of the more technically inclined in here will recognize this as an ETL. process, basically, and that's precisely what it is with a little AI flavor to it. Quickly, we have source data. This is where our raw data lives in our data lake. We pull it in, we structure it so that the AI can better understand it and better make use of it to create the summaries. Then we Inject it into a prompt, and then we... And then we can go back. Yeah. Inject it into a prompt, send it out to the LLM endpoint, it generates a summary, comes back, we put it in Salesforce and it's ready for our sales team, and then we have that continuous improvement loop at the bottom. Next slide. And just a little more detail on each step. This is, we got to go grab the right information, right? So we know what we're creating, so we know which information to grab. I would say about this, this isn't only what is summarizing, but it's also information that will give context to the LLM to create the summaries. After that, you get the raw data, and you have to structure it so the LLM will understand it. Activities, for example, we're not just throwing in all the sales activities. There's tons of sales activities, and a lot of it is just noise. We have like templates that exist in the notes, email templates, event invites, things like that. That would just be noise, would add no value, and it would be more expensive. That's just more tokens you have to send out. So got to filter all that out. And then you structure it. We have the date, name of the activity, associated note, and that's one field, and it's ready to inject into that prompt. And we do that. For every, can you go back, please? We do that for every field, every piece of information we get. And then, once it's structured, then it's ready to put into the prompt. So a prompt, this is the centerpiece of the project. This is what I like to think of it as contract. This is how the shape and the quality of the summary, this is what dictates it at all. I like to think it's a contract, so it's telling explicitly exactly what the output will be, the rules it's got to stay within, and the expectation of the output. I have like, it's 3 layers, basically, rules and instructions. So the instructions tell it exactly what it's creating, the data it's going to pull for each piece of information, that we need at the end in that final summary. You recall there were specific pieces of information they were wanting to pull out, and so we explicitly define each of those. Output schema, we have it exported in a, we have it come back from the LLM as a JSON, structured JSON. And the reason for that is so we can send that, save that to our data lake in the way we want to. Otherwise, it would just return as just a ball of text. And then live data, that's the data we just created in the last step, and we insert that to the prompt and it's ready to go to the endpoint. This is the most technically simple of the process, so I'll go into a little bit about how we chose the model. So there's dozens of models, each with varying capacity and wildly varying costs. So we thought, well, how we're going to figure out which one to use. This is our first LLM sort of project at scale. So we said, well, it's going to be something to do with quality and cost. It needs to be good enough to do the job we're wanting it to do. At the same time, we have to keep in mind the cost. So we estimated the number of summaries we'd need over a year. We knew the output token, input token costs, so we were able to estimate total costs over a year. And then quality was a little more difficult. Ideally, what you would do is create a bunch of summaries and get that out in front of your experts. They would label it for you, and then you would know the quality of the various models you were looking at. Our salespeople did not have that time, so we came across this concept of LLM as a judge. And essentially what it is, is you use an LLM to label the output of another LLM. And we did that on several criteria. Was it missing any data? Is it accurate? Was it helpful? And was it easy to read? We found that on the two aspects of is it missing anything and was it accurate, it did spectacular. Probably, I'd say even better than a human could do. On the other two more subjective measures, it was not so great. Is it easy to understand and was it helpful? But we thought we found it to be directionally correct, so if it scored higher. Say, if it was an 8 for is it easy to understand? And another model scored a four, we could say, well, the eight was better. To our surprise, we found that an open source LLM model, I think it was LLama Maverick, was just as good as even the most recent Opus model, and it was much more cost-effective. So that was a surprise. So once we get the summaries, we send them into Salesforce, and this is the final product, what they see in Salesforce. If you recall the wall of text, That Dentacar showed, it's turned into this, so it went from the, you know, minutes to understand to seconds. Yeah. And the all-important feedback loop, of course, the feedback loop is very important for our project. You see at the bottom, those two fields down there, that's on the summaries, about every summary in Salesforce. And is it accurate? That's basically a label. This is a good summary, it's a bad summary. And then we provide a text box. If it's not good, precisely what the issue might be, and then we can correct the prompt, and then it just is a circle improvement circle. Next slide. And then once we have the project running, it's a matter of measuring the success of the project. We broadly think of this in three buckets. Adoption, so of the leads getting summaries, what percentage are getting used? Return, repeat usage, if they've used the summary, how many are returning. And we hope that increases over time. It takes some time for a solution like this to get into the workflow, but hopefully that continues to grow over time. The middle one, workflow impact, that's the biggest measure, the problem was it took too much time to understand a lead. So is the time decreasing to understand a lead? So that's very important to measure. Workflow fit, does it fit into the workflow easy? Handoff effectiveness, if it's easier to understand the lead, you would expect it to be easier to hand off that lead across the sales team. So are we seeing more team sales, for instance, or more sales reps touching the same lead, essentially? Confidence in next action, if they use some if they use the summary, are they more confident in their next sales activity? So we'd like to know that. And then finally, quality and outcomes. I've talked about quality before. We'll probably continue to use the LLM as judge system to ensure those prompts remain as good as they can be. Better notes, we're hoping they make the connection that Better notes equal better summary. More detailed notes equal better summary. So hopefully see better notes and more volume. And business outcomes, obviously, we'd like to see if there's any impact on sales, move-ins, conversion rates, if it's faster to understand a lead. It might be faster to get to those conversions. and time between activities. You know, if it's faster to understand the lead, maybe it's faster to get, you know, to the next activity. And to conclude, So we had a clear problem. It was difficult to understand a lead. We had a pretty solid AI solution, and we implemented it, and it went from minutes to understand the lead to seconds, and we think that's where AI will fit in. with our organization in the future, not necessarily to replace judgment or replace workflow, but to reduce that friction between a good salesperson and them doing an even better job. So with that, we are done. Thank you. Thank you, guys. Really interesting. My question is, Salesforce is a large company that's doing a lot of AI investment. How quickly was what you guys are doing be basically part of Salesforce? I mean, right? Yes, so Salesforce, as you may know, they have an AI platform called Agent Force, right? It's their platform. They are learning about this tool. They do have a summarization tool. Now, like Levi explained, yeah, we had to choose between whether we use native Salesforce functionality, which actually also has a cost. It's not free. It's an additional license, right? And we have to use tokens and all that. So that was the kind of due diligence we did. So the functionality is there, but is it the right way for this specific capability or the use case? It's what we tried out. Now, there are other use cases or capabilities that we know we are going to use Salesforce, the embedded AI functionality within Salesforce, right? If it's sales coaching is a great example, or the SDR, so there's an SDR agent within Salesforce. So those are things that, yeah, we're not probably going to build it out, right? We're going to just use the functionality that Salesforce gives. The other thing with this is... This is not, so we had some leeway in terms of, this is not real time in the sense that. what the model output gives, we do that on a nightly basis. So we didn't have to, it's not like someone opens up the record and then at the same time we have to figure out and give that summary back, right? That calculation has already happened, right? We're just showing them at that moment in time. So because of that, yeah, we could actually use less expensive ways of doing it. So it's always that balance of doing that due diligence, right? Hey, is it better to kind of build this, which is what we did in our case, or buy it off the shelf? Theh. Levi and Danikar, thanks for talking about your case study. Can you give us an idea of like your team size, how long this project took, maybe once you decided which model you were going to use or off the shelf and just give those types of details? The. So our data science team that Lee Wei is part of, there are four? Yeah, we have 4 data scientists. My team, which is the Salesforce team, we have six people, like 5 business analysts, system analysts, and one developer. But for the most part, it was Lee Wei who worked on it. from the model perspective. And then we spent, I would say, a couple of days bringing it into Salesforce. We only had some of the pipelines built with our warehouse. But it took us, I would say, two to three months from the inception to rolling this out. And the reason for that is, like I mentioned, this was really a cross-functional project. So we actually involved our sales folks from day one. So it was a point of, hey, showing them, getting their feedback, going back, iterating, correcting the model, showing them again. So we actually had, I don't know, weekly, many, many rounds of those feedback loops. And then that's how we came up with the solution. Right.