Success Story #1 – Prospect Summarization in CRM using AI to Enable Sales Conversion
Business Systems and Data
The session content will consist of a PowerPoint deck and also live demo of the Salesforce application showcasing how the users utilize the capability provided.
We will also talk through data ingestion using Snowflake & Databricks, model development using LLMs and surfacing of the outputs in Salesforce application.
We’ll talk through the project delivery process including iterative development based on end user feedback to fine tune the model outputs in the form of AI-generated summaries and also understanding where in the sales process the new functionality fits in the best.
We’ll talk through how the project was a collaborative effort between the Data Science, Data Analytics and Marketing & Sales Systems teams at LCS.
We collaborated with the business stakeholders and end users to understand the answers to the following questions as we developed the solution. This resulted in an optimum solution as well as increased adoption.
- What is the best way for the Summary support the Sales Process?
- When would a Summary have the biggest impact?
- Where would you expect to see the greatest impact from the Summary over time?
- What does acceptable risk look like?
Key Takeaways
- Use Gen AI to propel revenue growth
- Meet your users where they are!
- Make it easy and intuitive for users to utilize the capability
Session Recording
Session Data
Transcript from Summit:
Click any timestamp to jump to that moment in the video.
life care services senior living crm salesforce prospect summarization
We are going to be covering success story number one, which is going to be about prospect summarization in CRM using AI to enable sales conversion. And we'll be hearing from Levi Sperry and Dena Carr first. And I'll introduce the second speaker after they're done. Thank you. Good afternoon, everyone. I'm Dinakar. This is Levi. So we both thought that we were going to have 45 minutes for the session. Turns out we're going to have only half of that. So we're going to go real quick here. Hope that's okay. So we both are part of LCS, our Life Care Services. We are operator and owner of senior living communities.
senior living communities sales and marketing salesforce crm strategic priorities innovation
We operate around 120 senior living communities across the country. So as you can see here, we have a breadth and depth of the services that we provide for our residents and our communities. Today we're going to focus on what we do from a sales and marketing perspective. As you can see here, we operate around 120 communities. That's a lot of prospects, a lot of sales counselors, lots of conversations. So essentially a lot of nodes in our CRM, which is Salesforce. So whenever I say Salesforce, just think of it as your CRM system. So as a company, we have four key strategic priorities. As you can see, the Innovation and AI is a key priority for our company, just like I'm sure most of you.
sales cycle prospect journey 3 to 18 months senior living decision sales counselors
This specific capability that we have built really falls into the AI foundational high-level imperative. I want to spend some time on this slide. So this is our typical prospect journey. That 3 to 18 months that you see here, that's not a typo. And that's how long it takes. Sometimes it takes a few years, because this is a key decision that someone takes in their life. So essentially what we sell is, it's not a SKU, it's not a subscription. It's really where a prospective resident goes to spend the next chapter of their life. So it's a big decision, right? So that carries a lot of weight. Salesforce often has the information, but understanding that takes a lot of work. So just like I said, because the sales cycle is long, it might be two, three, four months since we have talked to that specific prospect.
crm notes scattered data unstructured information sales activities data accessibility
So as soon as they call up, our sales counselors or sales reps have maybe 10, 15 seconds to really refresh and retrain themselves about that relationship, right? We have the data, but how do you make sure that's useful? So this is a real record from our CRM system. What you see here is all these different nodes from our different activities and conversations, right? And this is actually typical for our sales force. A lot of our prospect records, we have the information, as you can see, but it's really wrong, right? I mean, how do we make sure we can make this useful for our sales counselors? So the key knowledge lives in nodes, but it's scattered over a period of months and years. It's written by different people with different types of writing and verbiage. Some just say LM for left message.
47 notes average 15 seconds speed reading cross-functional initiative sales efficiency
Some actually write 2, 3 paragraphs, right? So it's different. So how do we make sure we can make this searchable, and quickly understandable for our sales reps. So our mantra is a salesperson should be able to get to and understand these notes in seconds, not in minutes, but in seconds. So typically, yeah, it's our average number of notes for each record, each prospect your account, as the case may be, it's around 47 nodes per account on an average, right? And 15 seconds is not enough to get there, unless you are like a speed reader in which we are hiring, so please apply. If there's one thing that we want all of you to take from our session today, it's that this is really This isn't about AI.
cross-functional project user involvement sales counselors data science team feedback process
Now, with due respect to the organizers here, this is much more than just AI, right? So this is really a cross-functional initiative. So between our sales folks here, the sales counselors, sales leadership operations, our data and technology here. So I'm part of the technology team. I lead the CRM at LCS. Levi is part of the data science team. We had a lot of other teams. This was truly a cross-functional project. and we involved even our sales folks from day one. And we'll go through that, really trying to understand what's benefit, what's beneficial, what is really helpful, to understand from their end what is helpful rather than what we thought was helpful. So from a feedback process, we really started out with really showing what the possibilities are. We actually gave three different kinds of, hey, do you want to have a heads-up display or do you want this kind of three paragraphs of summaries or something in between?
spreadsheet prototyping model validation iterative feedback data accuracy financial information
So we got that feedback. And this was way before Levi and the team actually started creating the model. So once we did that, we showed some of the results. And this is where we want to really highlight some of this, that you don't have to do everything build everything and then show. We actually showed some of these results from the model within spreadsheets. Hey, this is how it's going to look like. Yes, at the end of the day, we will have it in your CRM. You can see it just like you see the prospect record. But we wanted to make sure what the model is giving back was accurate and complete. So for that, to get that feedback, we just used spreadsheets and showed them. We don't have to go all fancy. And then finally, trying to understand, hey, what are the specific data points or piece of information that would be beneficial to them? So it's a two, three sentence summary for sure. But then we also extracted certain specific information, like financial information, what are their personal details, constraints, who are the decision makers, et cetera.
etl pipeline unstructured data ai summarization sales requirements technical architecture
Now think of this, right, going back to that slide with the 47 notes, It's not easy to get to this information quickly, right? So this is what we are giving them. I'll turn it over to Levi to talk through how their model was actually built. Thank you, Dinikar. Okay, real quick, about more of the technical aspect of the project. So we have a problem. It takes too long for our sales reps to understand a lead. There's too many notes. It's difficult to pull out. important information. We know that AI is very good at taking large amounts of unstructured data and summarizing it, right? And we have our requirements. We met with our sales experts at the beginning to get exactly what they're looking for in a summary. So around the problem, the AI solution, and requirements, we built this pipeline. Some of the more technically inclined in here will recognize this as an ETL process, basically, and that's precisely what it is, with a little AI flavor to it.
data lake data structuring prompt injection llm endpoint salesforce integration
Quickly, we have source data. This is where our raw data lives in our data lake. We pull it in, we structure it so that the AI can better understand it and better make use of it to create the summaries. Then we inject it into a prompt, and then we, and then we, can you go back? Yeah. Inject it into a prompt, send it out to the LLM endpoint, it generates a summary, comes back, we put it in Salesforce and it's ready for our sales team, and then we have that continuous improvement loop at the bottom. Next slide. And just a little more detail on each step. This is, we've got to go grab the right information, right? So we know what we're creating, so we know which information to grab. I would say about this, This isn't only what it's summarizing, but it's also information that'll give context to the LLM to create the summaries. After that, you get the raw data and you have to structure it so the LLM will understand it.
data filtering noise reduction token optimization cost management activity structuring
Activities, for example, we're not just throwing in all the sales activities. There's tons of sales activities and a lot of it is just noise. We have like templates that exist in the notes, e-mail templates, event invites, things like that. would just be noise. would add no value, and it would be more expensive. That's just more tokens you have to send out. So got to filter all that out, and then you structure it. We have the date, name of the activity, associated note, and that's one field, and it's ready to inject into that prompt. And we do that for every... Can you go back, please? We do that for every field, every piece of information we get. And then once it's structured, then it's ready to put into the prompt. So a prompt, this is the centerpiece of the project. This is what I like to think of it as a contract. This is how the shape and the quality of the summary, this is what dictates it at all.
prompt engineering contract design output schema json format rules and instructions
I like to think it's a contract, so it's telling explicitly exactly what the output will be. The rules, it's got a stay within and the expectation of the output. It's 3 layers, basically, rules and instructions. So the instructions tell it exactly what it's creating, the data it's going to pull for each piece of information that we need at the end in that final summary. You recall there were specific pieces of information they were wanting to pull out. And so we explicitly define each of those. Output schema, we have it exported in a, we have it come back from the LLM as a JSON, structured JSON. And the reason for that is so we can send that, save that to our data lake in the way we want to. Otherwise, it would just return as just a wall of text. And then live data, that's the data we just created in the last step, and we insert that to the prompt and it's ready to go to the endpoint.
model selection cost estimation llm as judge quality evaluation token costs
This is the most technically simple of the process, so I'll go into a little bit about how we chose the model. So there's dozens of models, each with varying capacity and wildly varying costs. So we thought, well, how are we going to figure out which one to use? This is our first LLM. sort of project at scale. So we said, it's going to be something to do with quality and cost. It needs to be good enough to do the job we're wanting it to do. At the same time, we have to keep in mind the cost. So we estimated the number of summaries we'd need over a year. We knew the output token, input token costs. So we were able to estimate total cost over a year. And then quality was a little more difficult. Ideally, what you would do is create a bunch of summaries and get that out in front of your experts. They would label it for you, and then you would know the quality of the various models you're looking at.
llm as judge llama maverick opus model open source accuracy evaluation
Our salespeople did not have that time, so we came across this concept of LLM as a judge. And essentially what it is, you use an LLM to label the output of another LLM. And we did that on several criteria. Was it missing any data? Is it accurate? Was it helpful? And was it easy to read? We found that on the two aspects of, is it missing anything? And was it accurate? It did spectacular. Probably, I'd say, even better than a human could do. On the other two more subjective measures, it was not so great. Is it easy to understand? And was it helpful? But we thought, we found it to be directionally correct. So if it scored higher, say, if it was an 8, four, is it easy to understand? And another model scored a four, we could say, well, the eight was better. To our surprise, we found that an open source LLM model, I think it was Llama Maverick, was just as good as even the most, you know, recent Opus model.
salesforce integration summary display user feedback accuracy labels prompt correction
And it was much more cost effective. So that was a surprise. So once we get the summaries, we send them into Salesforce, and this is the final product, what they see in Salesforce. If you recall the wall of text that Denikar showed, it's turned into this, so it went from minutes to understand to seconds. Yeah. And the all-important feedback loop, of course, the feedback loop is very important for our project. You see at the bottom, those two fields down there, that's on the summaries, every summary in Salesforce. And is it accurate? That's basically a label. This is a good summary, it's a bad summary. And then we provide a text box. If it's not good, precisely what the issue might be, and then we can correct the prompt. And then it just is a circle, improvement circle. Next slide. And then once we have the project running, It's a matter of measuring the success of the project.
adoption metrics workflow impact return usage handoff effectiveness conversion rates
We broadly think of this in three buckets. Adoption. So of the leads getting summaries, what percentage are getting used? Return, repeat usage. If they've used the summary, how many are returning? And we hope that increases over time. It takes some time for a solution like this to get into the workflow. But hopefully that continues to grow over time. The middle one, workflow impact, that's the biggest measure. The problem was it took too much time to understand a lead. So is the time decreasing to understand a lead? So that's very important to measure. Workflow fit, does it fit into the workflow easy? Handoff effectiveness, if it's easier to understand a lead, you would expect it to be easier to hand off that lead across the sales team. So are we seeing more team sales, for instance, or more sales reps touching the same lead, essentially?
quality monitoring llm as judge better notes move-ins conversion rates
Confidence in next action, if they use the summary, Are they more confident in their next sales activity? So we'd like to know that. And then finally, quality and outcomes. I've talked about quality before. We'll probably continue to use the LLM as judge system to ensure those prompts remain as good as they can be. Better notes, we're hoping they make the connection that better notes equal better summary. More detailed notes equal better summary. So hopefully see better notes and more volume. And business outcomes, obviously we'd like to see if there's any impact on sales, move-ins, conversion rates, if it's faster to understand a lead, it might be faster to get to those conversions. And time between activities, you know, if it's faster to understand the lead, maybe it's faster to get, you know, to the next activity. And to conclude, so we had a clear problem.
friction reduction ai philosophy workflow enhancement salesperson enablement not replacement
It was difficult to understand a lead. We had a pretty solid AI solution and we implemented it and it went from minutes to understand the lead to seconds. And we think that's where AI will fit in with our organization in the future, not necessarily to replace judgment or replace workflow. but to reduce that friction between a good salesperson and them doing an even better job. So with that, we are done. Thank you. Thank you, guys. Really interesting. My question is, Salesforce is a large company that's doing a lot of AI investment.
salesforce agentforce build vs buy licensing costs nightly batch not real-time
How quickly was what you guys are doing be basically part of Salesforce? I mean, right? Yes, so Salesforce, as you may know, they have a AI platform called Agentforce, right? It's their platform. They are learning about this tool. They do have a summarization tool. Now, like Levi explained, yeah, we had to choose between whether we use native Salesforce functionality, which actually also has a cost. It's not free. It's an additional license, right? And we have to use tokens and all that. So that was the kind of due diligence we did. So the functionality is there, but is it the right way for this specific capability or the use case? is what we tried out. Now, there are other use cases or capabilities that we know we are going to use Salesforce, the embedded AI functionality within Salesforce, right?
batch processing nightly updates not real-time pre-calculated summaries cost optimization
If it's sales coaching is a great example, or the SDR, so there's an SDR agent within Salesforce. So those are things that, yeah, we're not probably going to build it out, right? We're going to just use the functionality that Salesforce gives. The other thing with this is This is not, so we had some leeway in terms of this is not real time in the sense that what the model output gives, we do that on a nightly basis. So we didn't have to, it's not like someone opens up the record and then at the same time we have to figure out and give that summary back, right? That calculation has already happened, right? We're just showing them at that moment in time. So because of that, yeah, we could actually use less expensive ways of doing it. So it's always that balance of doing that due diligence, right? Hey, is it better to kind of build this, which is what we did in our case, or buy it off the shelf?
team size 4 data scientists 6 salesforce team 2-3 months timeline project duration
I have a question. Yep. Thanks. Levi and Danikar, thanks for talking about your case study. Can you give us an idea of like your team size, how long this project took, maybe once you decided which model you were going to use or off the shelf, and just give those types of details? So our data science team that Levi is part of, there are four? Yeah, we have 4 data scientists. My team, which is the Salesforce team, we have Six people, 5 business analysts, system analysts, and one developer. But for the most part, it was Levi who worked on it from the model perspective. And then we spent, I would say, a couple of days bringing it into Salesforce.
weekly feedback iteration loops sales involvement day one engagement model corrections
We only had some of the pipelines built with our warehouse. But it took us, I would say, two to three months from the inception. to rolling this out. And the reason for that is, like I mentioned, this was really a cross-functional project. So we actually involved our sales folks from day one. So it was a point of, hey, showing them, getting their feedback, going back, iterating, correcting the model, showing them again. So we actually had, I don't know, weekly, many, many rounds of those feedback loops. And then that's how we came up with the solution.
We are going to be covering success story number one, which is going to be about prospect summarization in CRM using AI to enable sales conversion. And we'll be hearing from Levi Sperry and Dena Carr first. And I'll introduce the second speaker after they're done. Thank you.
Good afternoon, everyone. I'm Dinakar. This is Levi. So we both thought that we were going to have 45 minutes for the session.
Turns out we're going to have only half of that. So we're going to go real quick here. Hope that's okay. So we both are part of LCS, our Life Care Services.
We are operator and owner of senior living communities. We operate around 120 senior living communities across the country. So as you can see here, we have a breadth and depth of the services that we provide for our residents and our communities. Today we're going to focus on what we do from a sales and marketing perspective.
As you can see here, we operate around 120 communities. That's a lot of prospects, a lot of sales counselors, lots of conversations. So essentially a lot of nodes in our CRM, which is Salesforce. So whenever I say Salesforce, just think of it as your CRM system.
So as a company, we have four key strategic priorities. As you can see, the Innovation and AI is a key priority for our company, just like I'm sure most of you. This specific capability that we have built really falls into the AI foundational high-level imperative. I want to spend some time on this slide.
So this is our typical prospect journey. That 3 to 18 months that you see here, that's not a typo. And that's how long it takes. Sometimes it takes a few years, because this is a key decision that someone takes in their life.
So essentially what we sell is, it's not a SKU, it's not a subscription. It's really where a prospective resident goes to spend the next chapter of their life. So it's a big decision, right? So that carries a lot of weight.
Salesforce often has the information, but understanding that takes a lot of work. So just like I said, because the sales cycle is long, it might be two, three, four months since we have talked to that specific prospect. So as soon as they call up, our sales counselors or sales reps have maybe 10, 15 seconds to really refresh and retrain themselves about that relationship, right? We have the data, but how do you make sure that's useful?
So this is a real record from our CRM system. What you see here is all these different nodes from our different activities and conversations, right? And this is actually typical for our sales force. A lot of our prospect records, we have the information, as you can see, but it's really wrong, right?
I mean, how do we make sure we can make this useful for our sales counselors? So the key knowledge lives in nodes, but it's scattered over a period of months and years. It's written by different people with different types of writing and verbiage. Some just say LM for left message.
Some actually write 2, 3 paragraphs, right? So it's different. So how do we make sure we can make this searchable, and quickly understandable for our sales reps. So our mantra is a salesperson should be able to get to and understand these notes in seconds, not in minutes, but in seconds.
So typically, yeah, it's our average number of notes for each record, each prospect your account, as the case may be, it's around 47 nodes per account on an average, right? And 15 seconds is not enough to get there, unless you are like a speed reader in which we are hiring, so please apply. If there's one thing that we want all of you to take from our session today, it's that this is really This isn't about AI. Now, with due respect to the organizers here, this is much more than just AI, right?
So this is really a cross-functional initiative. So between our sales folks here, the sales counselors, sales leadership operations, our data and technology here. So I'm part of the technology team. I lead the CRM at LCS.
Levi is part of the data science team. We had a lot of other teams. This was truly a cross-functional project. and we involved even our sales folks from day one. And we'll go through that, really trying to understand what's benefit, what's beneficial, what is really helpful, to understand from their end what is helpful rather than what we thought was helpful.
So from a feedback process, we really started out with really showing what the possibilities are. We actually gave three different kinds of, hey, do you want to have a heads-up display or do you want this kind of three paragraphs of summaries or something in between? So we got that feedback. And this was way before Levi and the team actually started creating the model.
So once we did that, we showed some of the results. And this is where we want to really highlight some of this, that you don't have to do everything build everything and then show. We actually showed some of these results from the model within spreadsheets. Hey, this is how it's going to look like.
Yes, at the end of the day, we will have it in your CRM. You can see it just like you see the prospect record. But we wanted to make sure what the model is giving back was accurate and complete. So for that, to get that feedback, we just used spreadsheets and showed them.
We don't have to go all fancy. And then finally, trying to understand, hey, what are the specific data points or piece of information that would be beneficial to them? So it's a two, three sentence summary for sure. But then we also extracted certain specific information, like financial information, what are their personal details, constraints, who are the decision makers, et cetera.
Now think of this, right, going back to that slide with the 47 notes, It's not easy to get to this information quickly, right? So this is what we are giving them. I'll turn it over to Levi to talk through how their model was actually built. Thank you, Dinikar.
Okay, real quick, about more of the technical aspect of the project. So we have a problem. It takes too long for our sales reps to understand a lead. There's too many notes.
It's difficult to pull out. important information. We know that AI is very good at taking large amounts of unstructured data and summarizing it, right? And we have our requirements. We met with our sales experts at the beginning to get exactly what they're looking for in a summary.
So around the problem, the AI solution, and requirements, we built this pipeline. Some of the more technically inclined in here will recognize this as an ETL process, basically, and that's precisely what it is, with a little AI flavor to it. Quickly, we have source data. This is where our raw data lives in our data lake.
We pull it in, we structure it so that the AI can better understand it and better make use of it to create the summaries. Then we inject it into a prompt, and then we, and then we, can you go back? Yeah. Inject it into a prompt, send it out to the LLM endpoint, it generates a summary, comes back, we put it in Salesforce and it's ready for our sales team, and then we have that continuous improvement loop at the bottom.
Next slide. And just a little more detail on each step. This is, we've got to go grab the right information, right? So we know what we're creating, so we know which information to grab.
I would say about this, This isn't only what it's summarizing, but it's also information that'll give context to the LLM to create the summaries. After that, you get the raw data and you have to structure it so the LLM will understand it. Activities, for example, we're not just throwing in all the sales activities. There's tons of sales activities and a lot of it is just noise.
We have like templates that exist in the notes, e-mail templates, event invites, things like that. would just be noise. would add no value, and it would be more expensive. That's just more tokens you have to send out. So got to filter all that out, and then you structure it. We have the date, name of the activity, associated note, and that's one field, and it's ready to inject into that prompt.
And we do that for every... Can you go back, please? We do that for every field, every piece of information we get. And then once it's structured, then it's ready to put into the prompt.
So a prompt, this is the centerpiece of the project. This is what I like to think of it as a contract. This is how the shape and the quality of the summary, this is what dictates it at all. I like to think it's a contract, so it's telling explicitly exactly what the output will be.
The rules, it's got a stay within and the expectation of the output. It's 3 layers, basically, rules and instructions. So the instructions tell it exactly what it's creating, the data it's going to pull for each piece of information that we need at the end in that final summary. You recall there were specific pieces of information they were wanting to pull out.
And so we explicitly define each of those. Output schema, we have it exported in a, we have it come back from the LLM as a JSON, structured JSON. And the reason for that is so we can send that, save that to our data lake in the way we want to. Otherwise, it would just return as just a wall of text.
And then live data, that's the data we just created in the last step, and we insert that to the prompt and it's ready to go to the endpoint. This is the most technically simple of the process, so I'll go into a little bit about how we chose the model. So there's dozens of models, each with varying capacity and wildly varying costs. So we thought, well, how are we going to figure out which one to use?
This is our first LLM. sort of project at scale. So we said, it's going to be something to do with quality and cost. It needs to be good enough to do the job we're wanting it to do. At the same time, we have to keep in mind the cost.
So we estimated the number of summaries we'd need over a year. We knew the output token, input token costs. So we were able to estimate total cost over a year. And then quality was a little more difficult.
Ideally, what you would do is create a bunch of summaries and get that out in front of your experts. They would label it for you, and then you would know the quality of the various models you're looking at. Our salespeople did not have that time, so we came across this concept of LLM as a judge. And essentially what it is, you use an LLM to label the output of another LLM.
And we did that on several criteria. Was it missing any data? Is it accurate? Was it helpful?
And was it easy to read? We found that on the two aspects of, is it missing anything? And was it accurate? It did spectacular.
Probably, I'd say, even better than a human could do. On the other two more subjective measures, it was not so great. Is it easy to understand? And was it helpful?
But we thought, we found it to be directionally correct. So if it scored higher, say, if it was an 8, four, is it easy to understand? And another model scored a four, we could say, well, the eight was better. To our surprise, we found that an open source LLM model, I think it was Llama Maverick, was just as good as even the most, you know, recent Opus model.
And it was much more cost effective. So that was a surprise. So once we get the summaries, we send them into Salesforce, and this is the final product, what they see in Salesforce. If you recall the wall of text that Denikar showed, it's turned into this, so it went from minutes to understand to seconds.
Yeah. And the all-important feedback loop, of course, the feedback loop is very important for our project. You see at the bottom, those two fields down there, that's on the summaries, every summary in Salesforce. And is it accurate?
That's basically a label. This is a good summary, it's a bad summary. And then we provide a text box. If it's not good, precisely what the issue might be, and then we can correct the prompt.
And then it just is a circle, improvement circle. Next slide. And then once we have the project running, It's a matter of measuring the success of the project. We broadly think of this in three buckets.
Adoption. So of the leads getting summaries, what percentage are getting used? Return, repeat usage. If they've used the summary, how many are returning?
And we hope that increases over time. It takes some time for a solution like this to get into the workflow. But hopefully that continues to grow over time. The middle one, workflow impact, that's the biggest measure.
The problem was it took too much time to understand a lead. So is the time decreasing to understand a lead? So that's very important to measure. Workflow fit, does it fit into the workflow easy?
Handoff effectiveness, if it's easier to understand a lead, you would expect it to be easier to hand off that lead across the sales team. So are we seeing more team sales, for instance, or more sales reps touching the same lead, essentially? Confidence in next action, if they use the summary, Are they more confident in their next sales activity? So we'd like to know that.
And then finally, quality and outcomes. I've talked about quality before. We'll probably continue to use the LLM as judge system to ensure those prompts remain as good as they can be. Better notes, we're hoping they make the connection that better notes equal better summary.
More detailed notes equal better summary. So hopefully see better notes and more volume. And business outcomes, obviously we'd like to see if there's any impact on sales, move-ins, conversion rates, if it's faster to understand a lead, it might be faster to get to those conversions. And time between activities, you know, if it's faster to understand the lead, maybe it's faster to get, you know, to the next activity.
And to conclude, so we had a clear problem. It was difficult to understand a lead. We had a pretty solid AI solution and we implemented it and it went from minutes to understand the lead to seconds. And we think that's where AI will fit in with our organization in the future, not necessarily to replace judgment or replace workflow. but to reduce that friction between a good salesperson and them doing an even better job.
So with that, we are done. Thank you. Thank you, guys. Really interesting.
My question is, Salesforce is a large company that's doing a lot of AI investment. How quickly was what you guys are doing be basically part of Salesforce? I mean, right? Yes, so Salesforce, as you may know, they have a AI platform called Agentforce, right?
It's their platform. They are learning about this tool. They do have a summarization tool. Now, like Levi explained, yeah, we had to choose between whether we use native Salesforce functionality, which actually also has a cost.
It's not free. It's an additional license, right? And we have to use tokens and all that. So that was the kind of due diligence we did.
So the functionality is there, but is it the right way for this specific capability or the use case? is what we tried out. Now, there are other use cases or capabilities that we know we are going to use Salesforce, the embedded AI functionality within Salesforce, right? If it's sales coaching is a great example, or the SDR, so there's an SDR agent within Salesforce. So those are things that, yeah, we're not probably going to build it out, right?
We're going to just use the functionality that Salesforce gives. The other thing with this is This is not, so we had some leeway in terms of this is not real time in the sense that what the model output gives, we do that on a nightly basis. So we didn't have to, it's not like someone opens up the record and then at the same time we have to figure out and give that summary back, right? That calculation has already happened, right?
We're just showing them at that moment in time. So because of that, yeah, we could actually use less expensive ways of doing it. So it's always that balance of doing that due diligence, right? Hey, is it better to kind of build this, which is what we did in our case, or buy it off the shelf?
I have a question. Yep. Thanks. Levi and Danikar, thanks for talking about your case study.
Can you give us an idea of like your team size, how long this project took, maybe once you decided which model you were going to use or off the shelf, and just give those types of details? So our data science team that Levi is part of, there are four? Yeah, we have 4 data scientists. My team, which is the Salesforce team, we have Six people, 5 business analysts, system analysts, and one developer.
But for the most part, it was Levi who worked on it from the model perspective. And then we spent, I would say, a couple of days bringing it into Salesforce. We only had some of the pipelines built with our warehouse. But it took us, I would say, two to three months from the inception. to rolling this out.
And the reason for that is, like I mentioned, this was really a cross-functional project. So we actually involved our sales folks from day one. So it was a point of, hey, showing them, getting their feedback, going back, iterating, correcting the model, showing them again. So we actually had, I don't know, weekly, many, many rounds of those feedback loops.
And then that's how we came up with the solution.