1
00:00:00,614 --> 00:00:10,614
We are going to be covering success story number one, which is going to be about prospect summarization in CRM using AI to enable sales conversion.

2
00:00:11,174 --> 00:00:15,494
And we'll be hearing from Levi Sperry and Dena Carr first.

3
00:00:16,974 --> 00:00:19,254
And I'll introduce the second speaker after they're done.

4
00:00:19,694 --> 00:00:19,974
Thank you.

5
00:00:33,254 --> 00:00:34,494
Good afternoon, everyone.

6
00:00:34,494 --> 00:00:35,094
I'm Dinakar.

7
00:00:35,254 --> 00:00:35,974
This is Levi.

8
00:00:39,414 --> 00:00:43,414
So we both thought that we were going to have 45 minutes for the session.

9
00:00:43,414 --> 00:00:45,494
Turns out we're going to have only half of that.

10
00:00:45,494 --> 00:00:47,414
So we're going to go real quick here.

11
00:00:47,974 --> 00:00:48,614
Hope that's okay.

12
00:00:51,734 --> 00:00:56,054
So we both are part of LCS, our Life Care Services.

13
00:00:56,534 --> 00:00:59,814
We are operator and owner of

14
00:01:00,214 --> 00:01:01,454
senior living communities.

15
00:01:01,454 --> 00:01:05,094
We operate around 120 senior living communities across the country.

16
00:01:11,844 --> 00:01:21,604
So as you can see here, we have a breadth and depth of the services that we provide for our residents and our communities.

17
00:01:22,084 --> 00:01:26,404
Today we're going to focus on what we do from a sales and marketing perspective.

18
00:01:27,524 --> 00:01:30,724
As you can see here, we operate around

19
00:01:31,814 --> 00:01:32,974
120 communities.

20
00:01:33,174 --> 00:01:38,094
That's a lot of prospects, a lot of sales counselors, lots of conversations.

21
00:01:38,094 --> 00:01:43,174
So essentially a lot of nodes in our CRM, which is Salesforce.

22
00:01:43,574 --> 00:01:47,654
So whenever I say Salesforce, just think of it as your CRM system.

23
00:01:51,014 --> 00:01:54,854
So as a company, we have four key strategic priorities.

24
00:01:55,414 --> 00:01:56,974
As you can see, the

25
00:01:58,214 --> 00:02:04,774
Innovation and AI is a key priority for our company, just like I'm sure most of you.

26
00:02:05,494 --> 00:02:13,614
This specific capability that we have built really falls into the AI foundational high-level imperative.

27
00:02:13,614 --> 00:02:18,694
I want to spend some time on this slide.

28
00:02:19,174 --> 00:02:22,294
So this is our typical prospect journey.

29
00:02:23,174 --> 00:02:26,854
That 3 to 18 months that you see here, that's not a typo.

30
00:02:27,214 --> 00:02:29,094
And that's how long it takes.

31
00:02:29,174 --> 00:02:35,654
Sometimes it takes a few years, because this is a key decision that someone takes in their life.

32
00:02:35,894 --> 00:02:40,614
So essentially what we sell is, it's not a SKU, it's not a subscription.

33
00:02:41,014 --> 00:02:47,174
It's really where a prospective resident goes to spend the next chapter of their life.

34
00:02:47,814 --> 00:02:49,494
So it's a big decision, right?

35
00:02:49,494 --> 00:02:51,494
So that carries a lot of weight.

36
00:02:55,174 --> 00:03:00,614
Salesforce often has the information, but understanding that takes a lot of work.

37
00:03:01,094 --> 00:03:08,934
So just like I said, because the sales cycle is long, it might be two, three, four months since we have talked to that specific prospect.

38
00:03:09,414 --> 00:03:20,134
So as soon as they call up, our sales counselors or sales reps have maybe 10, 15 seconds to really refresh and retrain themselves about that relationship, right?

39
00:03:20,374 --> 00:03:22,934
We have the data, but how do you make sure that's useful?

40
00:03:25,334 --> 00:03:28,214
So this is a real record from our CRM system.

41
00:03:28,614 --> 00:03:35,014
What you see here is all these different nodes from our different activities and conversations, right?

42
00:03:35,094 --> 00:03:40,614
And this is actually typical for our sales force.

43
00:03:41,294 --> 00:03:47,694
A lot of our prospect records, we have the information, as you can see, but it's really wrong, right?

44
00:03:47,694 --> 00:03:51,894
I mean, how do we make sure we can make this useful for our sales counselors?

45
00:03:54,614 --> 00:04:00,814
So the key knowledge lives in nodes, but it's scattered over a period of months and years.

46
00:04:01,734 --> 00:04:06,934
It's written by different people with different types of writing and verbiage.

47
00:04:07,174 --> 00:04:10,454
Some just say LM for left message.

48
00:04:10,614 --> 00:04:13,254
Some actually write 2, 3 paragraphs, right?

49
00:04:13,574 --> 00:04:14,614
So it's different.

50
00:04:14,934 --> 00:04:17,814
So how do we make sure we can make this

51
00:04:18,534 --> 00:04:22,694
searchable, and quickly understandable for our sales reps.

52
00:04:27,254 --> 00:04:36,254
So our mantra is a salesperson should be able to get to and understand these notes in seconds, not in minutes, but in seconds.

53
00:04:38,694 --> 00:04:45,734
So typically, yeah, it's our average number of notes for each record, each prospect

54
00:04:45,854 --> 00:04:53,814
your account, as the case may be, it's around 47 nodes per account on an average, right?

55
00:04:54,374 --> 00:05:04,374
And 15 seconds is not enough to get there, unless you are like a speed reader in which we are hiring, so please apply.

56
00:05:08,054 --> 00:05:15,254
If there's one thing that we want all of you to take from our session today, it's that this is really

57
00:05:15,814 --> 00:05:17,094
This isn't about AI.

58
00:05:17,174 --> 00:05:22,854
Now, with due respect to the organizers here, this is much more than just AI, right?

59
00:05:23,094 --> 00:05:25,174
So this is really a cross-functional initiative.

60
00:05:25,654 --> 00:05:32,374
So between our sales folks here, the sales counselors, sales leadership operations, our data and technology here.

61
00:05:32,374 --> 00:05:35,014
So I'm part of the technology team.

62
00:05:35,254 --> 00:05:37,174
I lead the CRM at LCS.

63
00:05:37,254 --> 00:05:39,014
Levi is part of the data science team.

64
00:05:39,574 --> 00:05:41,414
We had a lot of other teams.

65
00:05:41,414 --> 00:05:43,894
This was truly a cross-functional project.

66
00:05:44,454 --> 00:05:48,454
and we involved even our sales folks from day one.

67
00:05:48,734 --> 00:06:00,454
And we'll go through that, really trying to understand what's benefit, what's beneficial, what is really helpful, to understand from their end what is helpful rather than what we thought was helpful.

68
00:06:03,094 --> 00:06:09,334
So from a feedback process, we really started out with really showing what the possibilities are.

69
00:06:09,574 --> 00:06:10,214
We actually

70
00:06:11,334 --> 00:06:20,854
gave three different kinds of, hey, do you want to have a heads-up display or do you want this kind of three paragraphs of summaries or something in between?

71
00:06:21,334 --> 00:06:22,494
So we got that feedback.

72
00:06:22,494 --> 00:06:27,494
And this was way before Levi and the team actually started creating the model.

73
00:06:29,014 --> 00:06:33,334
So once we did that, we showed some of the results.

74
00:06:33,494 --> 00:06:38,374
And this is where we want to really highlight some of this, that you don't have to do everything

75
00:06:39,334 --> 00:06:40,694
build everything and then show.

76
00:06:40,694 --> 00:06:45,174
We actually showed some of these results from the model within spreadsheets.

77
00:06:45,494 --> 00:06:46,854
Hey, this is how it's going to look like.

78
00:06:47,414 --> 00:06:50,454
Yes, at the end of the day, we will have it in your CRM.

79
00:06:50,854 --> 00:06:54,294
You can see it just like you see the prospect record.

80
00:06:54,854 --> 00:06:59,814
But we wanted to make sure what the model is giving back was accurate and complete.

81
00:07:00,134 --> 00:07:05,254
So for that, to get that feedback, we just used spreadsheets and showed them.

82
00:07:05,654 --> 00:07:07,254
We don't have to go all fancy.

83
00:07:08,174 --> 00:07:16,934
And then finally, trying to understand, hey, what are the specific data points or piece of information that would be beneficial to them?

84
00:07:17,494 --> 00:07:20,614
So it's a two, three sentence summary for sure.

85
00:07:20,934 --> 00:07:32,774
But then we also extracted certain specific information, like financial information, what are their personal details, constraints, who are the decision makers, et cetera.

86
00:07:32,934 --> 00:07:36,854
Now think of this, right, going back to that slide with the 47 notes,

87
00:07:37,254 --> 00:07:40,214
It's not easy to get to this information quickly, right?

88
00:07:40,214 --> 00:07:42,134
So this is what we are giving them.

89
00:07:44,334 --> 00:07:48,134
I'll turn it over to Levi to talk through how their model was actually built.

90
00:07:50,294 --> 00:07:51,014
Thank you, Dinikar.

91
00:07:51,494 --> 00:07:55,174
Okay, real quick, about more of the technical aspect of the project.

92
00:07:55,174 --> 00:07:56,694
So we have a problem.

93
00:07:56,854 --> 00:08:00,294
It takes too long for our sales reps to understand a lead.

94
00:08:00,534 --> 00:08:01,494
There's too many notes.

95
00:08:01,494 --> 00:08:02,694
It's difficult to pull out.

96
00:08:03,254 --> 00:08:04,374
important information.

97
00:08:04,694 --> 00:08:11,694
We know that AI is very good at taking large amounts of unstructured data and summarizing it, right?

98
00:08:12,374 --> 00:08:14,054
And we have our requirements.

99
00:08:14,054 --> 00:08:19,174
We met with our sales experts at the beginning to get exactly what they're looking for in a summary.

100
00:08:19,174 --> 00:08:24,454
So around the problem, the AI solution, and requirements, we built this pipeline.

101
00:08:25,414 --> 00:08:27,414
Some of the more technically inclined in here will

102
00:08:27,974 --> 00:08:34,534
recognize this as an ETL process, basically, and that's precisely what it is, with a little AI flavor to it.

103
00:08:35,174 --> 00:08:37,814
Quickly, we have source data.

104
00:08:37,894 --> 00:08:40,054
This is where our raw data lives in our data lake.

105
00:08:40,054 --> 00:08:47,734
We pull it in, we structure it so that the AI can better understand it and better make use of it to create the summaries.

106
00:08:48,614 --> 00:08:57,374
Then we inject it into a prompt, and then we, and then we, can you go back?

107
00:08:57,374 --> 00:08:57,494
Yeah.

108
00:08:58,294 --> 00:09:09,814
Inject it into a prompt, send it out to the LLM endpoint, it generates a summary, comes back, we put it in Salesforce and it's ready for our sales team, and then we have that continuous improvement loop at the bottom.

109
00:09:10,294 --> 00:09:10,734
Next slide.

110
00:09:11,614 --> 00:09:13,254
And just a little more detail on each step.

111
00:09:15,334 --> 00:09:17,574
This is, we've got to go grab the right information, right?

112
00:09:17,574 --> 00:09:20,374
So we know what we're creating, so we know which information to grab.

113
00:09:21,174 --> 00:09:22,454
I would say about this,

114
00:09:22,854 --> 00:09:29,894
This isn't only what it's summarizing, but it's also information that'll give context to the LLM to create the summaries.

115
00:09:31,574 --> 00:09:36,054
After that, you get the raw data and you have to structure it so the LLM will understand it.

116
00:09:36,534 --> 00:09:40,774
Activities, for example, we're not just throwing in all the sales activities.

117
00:09:40,774 --> 00:09:44,054
There's tons of sales activities and a lot of it is just noise.

118
00:09:44,774 --> 00:09:50,454
We have like templates that exist in the notes, e-mail templates, event invites, things like that.

119
00:09:50,934 --> 00:09:51,974
would just be noise.

120
00:09:52,534 --> 00:09:55,254
would add no value, and it would be more expensive.

121
00:09:55,254 --> 00:09:57,014
That's just more tokens you have to send out.

122
00:09:57,414 --> 00:10:00,614
So got to filter all that out, and then you structure it.

123
00:10:00,934 --> 00:10:08,334
We have the date, name of the activity, associated note, and that's one field, and it's ready to inject into that prompt.

124
00:10:08,334 --> 00:10:11,134
And we do that for every...

125
00:10:11,134 --> 00:10:12,814
Can you go back, please?

126
00:10:12,814 --> 00:10:17,414
We do that for every field, every piece of information we get.

127
00:10:20,214 --> 00:10:23,654
And then once it's structured, then it's ready to put into the prompt.

128
00:10:26,774 --> 00:10:30,534
So a prompt, this is the centerpiece of the project.

129
00:10:31,814 --> 00:10:34,214
This is what I like to think of it as a contract.

130
00:10:34,294 --> 00:10:39,454
This is how the shape and the quality of the summary, this is what dictates it at all.

131
00:10:39,454 --> 00:10:48,214
I like to think it's a contract, so it's telling explicitly exactly what the output will be.

132
00:10:48,774 --> 00:10:49,654
The rules, it's got a

133
00:10:50,214 --> 00:10:53,254
stay within and the expectation of the output.

134
00:10:55,094 --> 00:10:57,814
It's 3 layers, basically, rules and instructions.

135
00:10:58,054 --> 00:11:10,054
So the instructions tell it exactly what it's creating, the data it's going to pull for each piece of information that we need at the end in that final summary.

136
00:11:10,214 --> 00:11:14,614
You recall there were specific pieces of information they were wanting to pull out.

137
00:11:15,054 --> 00:11:17,734
And so we explicitly define each of those.

138
00:11:18,694 --> 00:11:28,174
Output schema, we have it exported in a, we have it come back from the LLM as a JSON, structured JSON.

139
00:11:28,374 --> 00:11:34,054
And the reason for that is so we can send that, save that to our data lake in the way we want to.

140
00:11:34,454 --> 00:11:37,414
Otherwise, it would just return as just a wall of text.

141
00:11:37,894 --> 00:11:46,534
And then live data, that's the data we just created in the last step, and we insert that to the prompt and it's ready to go to the endpoint.

142
00:11:48,774 --> 00:11:55,494
This is the most technically simple of the process, so I'll go into a little bit about how we chose the model.

143
00:11:56,454 --> 00:12:03,494
So there's dozens of models, each with varying capacity and wildly varying costs.

144
00:12:05,094 --> 00:12:09,214
So we thought, well, how are we going to figure out which one to use?

145
00:12:09,214 --> 00:12:10,614
This is our first LLM.

146
00:12:11,574 --> 00:12:13,334
sort of project at scale.

147
00:12:13,654 --> 00:12:17,494
So we said, it's going to be something to do with quality and cost.

148
00:12:17,734 --> 00:12:21,254
It needs to be good enough to do the job we're wanting it to do.

149
00:12:21,574 --> 00:12:24,134
At the same time, we have to keep in mind the cost.

150
00:12:24,134 --> 00:12:27,654
So we estimated the number of summaries we'd need over a year.

151
00:12:27,894 --> 00:12:30,494
We knew the output token, input token costs.

152
00:12:30,494 --> 00:12:34,574
So we were able to estimate total cost over a year.

153
00:12:34,574 --> 00:12:37,134
And then quality was a little more difficult.

154
00:12:37,894 --> 00:12:39,414
Ideally, what you would do

155
00:12:39,974 --> 00:12:44,214
is create a bunch of summaries and get that out in front of your experts.

156
00:12:44,214 --> 00:12:49,414
They would label it for you, and then you would know the quality of the various models you're looking at.

157
00:12:50,374 --> 00:12:55,254
Our salespeople did not have that time, so we came across this concept of LLM as a judge.

158
00:12:55,734 --> 00:13:00,934
And essentially what it is, you use an LLM to label the output of another LLM.

159
00:13:01,614 --> 00:13:03,334
And we did that on several criteria.

160
00:13:04,134 --> 00:13:05,414
Was it missing any data?

161
00:13:05,654 --> 00:13:06,614
Is it accurate?

162
00:13:06,694 --> 00:13:07,814
Was it helpful?

163
00:13:08,294 --> 00:13:09,974
And was it easy to read?

164
00:13:10,454 --> 00:13:14,454
We found that on the two aspects of, is it missing anything?

165
00:13:15,374 --> 00:13:16,774
And was it accurate?

166
00:13:17,094 --> 00:13:18,134
It did spectacular.

167
00:13:18,454 --> 00:13:21,014
Probably, I'd say, even better than a human could do.

168
00:13:21,934 --> 00:13:25,894
On the other two more subjective measures, it was not so great.

169
00:13:26,694 --> 00:13:27,974
Is it easy to understand?

170
00:13:28,054 --> 00:13:29,174
And was it helpful?

171
00:13:29,734 --> 00:13:32,374
But we thought, we found it to be directionally correct.

172
00:13:32,374 --> 00:13:35,654
So if it scored higher,

173
00:13:37,014 --> 00:13:40,934
say, if it was an 8, four, is it easy to understand?

174
00:13:41,334 --> 00:13:46,054
And another model scored a four, we could say, well, the eight was better.

175
00:13:47,654 --> 00:14:01,414
To our surprise, we found that an open source LLM model, I think it was Llama Maverick, was just as good as even the most, you know, recent Opus model.

176
00:14:01,934 --> 00:14:03,654
And it was much more cost effective.

177
00:14:04,694 --> 00:14:06,134
So that was a surprise.

178
00:14:06,934 --> 00:14:14,374
So once we get the summaries, we send them into Salesforce, and this is the final product, what they see in Salesforce.

179
00:14:14,614 --> 00:14:23,374
If you recall the wall of text that Denikar showed, it's turned into this, so it went from minutes to understand to seconds.

180
00:14:23,374 --> 00:14:23,494
Yeah.

181
00:14:28,534 --> 00:14:33,334
And the all-important feedback loop, of course, the feedback loop is very important for our project.

182
00:14:34,854 --> 00:14:41,254
You see at the bottom, those two fields down there, that's on the summaries, every summary in Salesforce.

183
00:14:41,934 --> 00:14:43,694
And is it accurate?

184
00:14:43,694 --> 00:14:45,014
That's basically a label.

185
00:14:45,014 --> 00:14:47,014
This is a good summary, it's a bad summary.

186
00:14:47,014 --> 00:14:48,534
And then we provide a text box.

187
00:14:48,614 --> 00:14:54,414
If it's not good, precisely what the issue might be, and then we can correct the prompt.

188
00:14:54,414 --> 00:14:57,574
And then it just is a circle, improvement circle.

189
00:14:59,974 --> 00:15:00,454
Next slide.

190
00:15:01,374 --> 00:15:03,574
And then once we have the project running,

191
00:15:04,134 --> 00:15:07,414
It's a matter of measuring the success of the project.

192
00:15:07,414 --> 00:15:09,894
We broadly think of this in three buckets.

193
00:15:09,894 --> 00:15:10,614
Adoption.

194
00:15:10,934 --> 00:15:17,574
So of the leads getting summaries, what percentage are getting used?

195
00:15:18,294 --> 00:15:20,374
Return, repeat usage.

196
00:15:21,494 --> 00:15:24,854
If they've used the summary, how many are returning?

197
00:15:25,654 --> 00:15:27,934
And we hope that increases over time.

198
00:15:27,934 --> 00:15:32,134
It takes some time for a solution like this to get into the workflow.

199
00:15:32,934 --> 00:15:35,574
But hopefully that continues to grow over time.

200
00:15:36,374 --> 00:15:39,014
The middle one, workflow impact, that's the biggest measure.

201
00:15:39,014 --> 00:15:42,294
The problem was it took too much time to understand a lead.

202
00:15:42,294 --> 00:15:45,494
So is the time decreasing to understand a lead?

203
00:15:46,214 --> 00:15:47,574
So that's very important to measure.

204
00:15:48,134 --> 00:15:50,934
Workflow fit, does it fit into the workflow easy?

205
00:15:52,694 --> 00:16:00,214
Handoff effectiveness, if it's easier to understand a lead, you would expect it to be easier to hand off that lead across the sales team.

206
00:16:00,374 --> 00:16:08,374
So are we seeing more team sales, for instance, or more sales reps touching the same lead, essentially?

207
00:16:09,174 --> 00:16:14,054
Confidence in next action, if they use the summary,

208
00:16:14,534 --> 00:16:17,734
Are they more confident in their next sales activity?

209
00:16:18,294 --> 00:16:19,254
So we'd like to know that.

210
00:16:19,974 --> 00:16:21,654
And then finally, quality and outcomes.

211
00:16:21,654 --> 00:16:23,334
I've talked about quality before.

212
00:16:23,334 --> 00:16:29,654
We'll probably continue to use the LLM as judge system to ensure those prompts remain as good as they can be.

213
00:16:30,454 --> 00:16:35,094
Better notes, we're hoping they make the connection that better notes equal better summary.

214
00:16:35,334 --> 00:16:37,254
More detailed notes equal better summary.

215
00:16:37,254 --> 00:16:40,854
So hopefully see better notes and more volume.

216
00:16:41,734 --> 00:16:53,014
And business outcomes, obviously we'd like to see if there's any impact on sales, move-ins, conversion rates, if it's faster to understand a lead, it might be faster to get to those conversions.

217
00:16:53,654 --> 00:17:01,494
And time between activities, you know, if it's faster to understand the lead, maybe it's faster to get, you know, to the next activity.

218
00:17:02,454 --> 00:17:09,334
And to conclude, so we had a clear problem.

219
00:17:09,654 --> 00:17:11,254
It was difficult to understand a lead.

220
00:17:11,254 --> 00:17:21,774
We had a pretty solid AI solution and we implemented it and it went from minutes to understand the lead to seconds.

221
00:17:21,774 --> 00:17:31,254
And we think that's where AI will fit in with our organization in the future, not necessarily to replace judgment or replace workflow.

222
00:17:31,894 --> 00:17:38,814
but to reduce that friction between a good salesperson and them doing an even better job.

223
00:17:39,014 --> 00:17:42,134
So with that, we are done.

224
00:18:03,464 --> 00:18:03,744
Thank you.

225
00:18:05,024 --> 00:18:05,824
Thank you, guys.

226
00:18:06,064 --> 00:18:06,944
Really interesting.

227
00:18:07,344 --> 00:18:08,624
My question is,

228
00:18:09,014 --> 00:18:12,054
Salesforce is a large company that's doing a lot of AI investment.

229
00:18:12,454 --> 00:18:16,774
How quickly was what you guys are doing be basically part of Salesforce?

230
00:18:16,814 --> 00:18:17,734
I mean, right?

231
00:18:33,524 --> 00:18:40,404
Yes, so Salesforce, as you may know, they have a AI platform called Agentforce, right?

232
00:18:40,404 --> 00:18:42,804
It's their platform.

233
00:18:43,574 --> 00:18:45,374
They are learning about this tool.

234
00:18:45,374 --> 00:18:47,094
They do have a summarization tool.

235
00:18:47,414 --> 00:18:55,734
Now, like Levi explained, yeah, we had to choose between whether we use native Salesforce functionality, which actually also has a cost.

236
00:18:55,734 --> 00:18:57,174
It's not free.

237
00:18:57,574 --> 00:19:00,134
It's an additional license, right?

238
00:19:00,134 --> 00:19:01,734
And we have to use tokens and all that.

239
00:19:02,134 --> 00:19:05,894
So that was the kind of due diligence we did.

240
00:19:06,054 --> 00:19:10,374
So the functionality is there, but is it the right way for this specific capability or the use case?

241
00:19:10,934 --> 00:19:11,894
is what we tried out.

242
00:19:11,894 --> 00:19:20,934
Now, there are other use cases or capabilities that we know we are going to use Salesforce, the embedded AI functionality within Salesforce, right?

243
00:19:20,934 --> 00:19:28,574
If it's sales coaching is a great example, or the SDR, so there's an SDR agent within Salesforce.

244
00:19:28,574 --> 00:19:32,294
So those are things that, yeah, we're not probably going to build it out, right?

245
00:19:32,294 --> 00:19:34,654
We're going to just use the functionality that Salesforce gives.

246
00:19:35,334 --> 00:19:37,014
The other thing with this is

247
00:19:40,054 --> 00:19:51,014
This is not, so we had some leeway in terms of this is not real time in the sense that what the model output gives, we do that on a nightly basis.

248
00:19:51,414 --> 00:19:59,974
So we didn't have to, it's not like someone opens up the record and then at the same time we have to figure out and give that summary back, right?

249
00:19:59,974 --> 00:20:03,094
That calculation has already happened, right?

250
00:20:03,094 --> 00:20:04,934
We're just showing them at that moment in time.

251
00:20:05,334 --> 00:20:09,014
So because of that, yeah, we could actually use less expensive

252
00:20:09,334 --> 00:20:10,134
ways of doing it.

253
00:20:10,134 --> 00:20:14,934
So it's always that balance of doing that due diligence, right?

254
00:20:14,934 --> 00:20:21,694
Hey, is it better to kind of build this, which is what we did in our case, or buy it off the shelf?

255
00:20:21,694 --> 00:20:22,174
I have a question.

256
00:20:24,294 --> 00:20:24,414
Yep.

257
00:20:25,254 --> 00:20:25,454
Thanks.

258
00:20:31,334 --> 00:20:33,894
Levi and Danikar, thanks for talking about your case study.

259
00:20:33,894 --> 00:20:45,414
Can you give us an idea of like your team size, how long this project took, maybe once you decided which model you were going to use or off the shelf, and just give those types of details?

260
00:20:52,494 --> 00:20:57,534
So our data science team that Levi is part of, there are four?

261
00:20:58,494 --> 00:21:00,254
Yeah, we have 4 data scientists.

262
00:21:01,134 --> 00:21:03,774
My team, which is the Salesforce team, we have

263
00:21:04,614 --> 00:21:09,574
Six people, 5 business analysts, system analysts, and one developer.

264
00:21:10,214 --> 00:21:16,214
But for the most part, it was Levi who worked on it from the model perspective.

265
00:21:16,534 --> 00:21:22,654
And then we spent, I would say, a couple of days bringing it into Salesforce.

266
00:21:22,654 --> 00:21:25,654
We only had some of the pipelines built with our warehouse.

267
00:21:27,014 --> 00:21:33,414
But it took us, I would say, two to three months from the inception.

268
00:21:34,374 --> 00:21:35,974
to rolling this out.

269
00:21:36,054 --> 00:21:40,214
And the reason for that is, like I mentioned, this was really a cross-functional project.

270
00:21:40,294 --> 00:21:44,214
So we actually involved our sales folks from day one.

271
00:21:44,694 --> 00:21:53,494
So it was a point of, hey, showing them, getting their feedback, going back, iterating, correcting the model, showing them again.

272
00:21:53,654 --> 00:21:59,734
So we actually had, I don't know, weekly, many, many rounds of those feedback loops.

273
00:22:00,974 --> 00:22:02,814
And then that's how we came up with the solution.