1 00:00:00,063 --> 00:00:00,703 good afternoon. 2 00:00:00,703 --> 00:00:08,783 It's my pleasure to introduce Nick Nystrom, an experienced lead specializing in design and delivery of enterprise-scale digital experiences. 3 00:00:08,783 --> 00:00:18,303 His work brings together human-centered design, product strategy, and emerging technologies with a focus on using AI to simplify complex customer journeys, particularly in healthcare. 4 00:00:18,383 --> 00:00:20,543 So Nick has led cross-functional teams. 5 00:00:20,943 --> 00:00:29,103 across design, research, analytics, and engineering to translate high-complexity systems into intuitive and accessible user experiences. 6 00:00:29,103 --> 00:00:42,823 So in today's session, he will explore how natural language search can improve the way members find and understand their benefits, demonstrating how AI can reduce friction, increase clarity, and deliver meaningful value in real-world applications. 7 00:00:42,823 --> 00:00:45,423 So please join me in welcoming Nick Nystrom. 8 00:00:48,783 --> 00:00:49,103 Thank you. 9 00:00:49,103 --> 00:00:50,863 Appreciate it, everybody, for being here. 10 00:00:51,343 --> 00:00:52,063 Good afternoon. 11 00:00:52,503 --> 00:00:55,983 I'm Nick Nystrom from Walmart Blue Cross Blue Shield of Iowa and South Dakota. 12 00:00:55,983 --> 00:01:06,223 I'm going to walk you guys through today a story about how to apply AI in a place maybe where information is very complex, regulated, and really incredibly human. 13 00:01:06,703 --> 00:01:10,223 Member benefits and customer service is really where we're going to focus today. 14 00:01:10,783 --> 00:01:12,703 This isn't a product launch or a live demo. 15 00:01:12,703 --> 00:01:14,063 It's a real exploration. 16 00:01:14,583 --> 00:01:20,143 What worked, what didn't work, and really what we learned as a team through that lens of AI-driven experiment. 17 00:01:20,143 --> 00:01:24,943 So, what I want to start with first today is just a quick overview about myself and a little bit about Wellmark. 18 00:01:24,943 --> 00:01:26,703 Wellmark is in Iowa and South Dakota. 19 00:01:27,503 --> 00:01:31,743 And we have about 2,000 employees, and the headquarters is in Des Moines, where I work. 20 00:01:31,743 --> 00:01:38,063 I'm on a member experience team that's embedded within technology at our company, and there's about 60 of us on our team. 21 00:01:38,303 --> 00:01:46,863 We have everybody from disciplines ranging from design to analytics to user testing to research to experience delivery. 22 00:01:48,103 --> 00:01:51,503 a lot of different, we have content writers, we have designers, obviously. 23 00:01:51,743 --> 00:01:57,863 So a lot of disciplines that really help me as an experience lead deliver epic experiences to our members. 24 00:01:57,863 --> 00:02:00,783 So just by a raise of hands, how many have Walmart Insurance? 25 00:02:00,903 --> 00:02:01,503 Anybody here? 26 00:02:01,703 --> 00:02:02,583 Okay, a lot of people. 27 00:02:02,583 --> 00:02:05,263 We have a lot of members, about 2 million members in Iowa and South Dakota. 28 00:02:05,583 --> 00:02:08,863 My team's focused on really the first part of the member's journey. 29 00:02:08,863 --> 00:02:11,823 So that's the invite, the shop, the enroll, and the welcome moments. 30 00:02:12,063 --> 00:02:15,663 So how do we show up for those members in those service moments? 31 00:02:15,823 --> 00:02:17,183 That's what my team's focused on, right? 32 00:02:17,183 --> 00:02:17,983 So a lot of it's 33 00:02:18,103 --> 00:02:21,503 driving adoption to our self-serve platform, which is our mobile app. 34 00:02:21,983 --> 00:02:26,143 But there's also initiatives I'm leading across our enterprise as well, and I'll talk a little bit about that as well. 35 00:02:26,783 --> 00:02:29,023 We are all trained in human-centered design. 36 00:02:29,023 --> 00:02:33,743 We use Luma, which is a methodology for practitioners of human-centered design. 37 00:02:34,183 --> 00:02:42,503 I'm A Luma certified instructor as well, so we're also rolling out Luma practitioner trainings to the rest of the organization on different product teams. 38 00:02:42,503 --> 00:02:42,783 So 39 00:02:43,263 --> 00:02:46,943 Human-centered design, if you're not familiar with it, there's a lot of different methodologies. 40 00:02:46,943 --> 00:02:48,463 Luma is the one that we use at Walmart. 41 00:02:48,863 --> 00:02:55,423 It's really just about putting the user at the center of how to solve your problem, which yields really good results for us. 42 00:02:55,423 --> 00:02:59,903 We do a lot of data-driven decisions, so we measure everything we build. 43 00:02:59,903 --> 00:03:03,263 There's been a lot of themes throughout today's talks around measurement. 44 00:03:03,743 --> 00:03:09,263 We measure in the form of interaction metrics using Google Analytics and our digital properties. 45 00:03:09,503 --> 00:03:10,783 We also have Voice of Customer. 46 00:03:10,783 --> 00:03:12,783 We have a really extensive Voice of Customer program. 47 00:03:12,783 --> 00:03:19,503 So if you interact with something, whether it's physical customer service phone call on our mobile property, you're going to get an e-mail, right? 48 00:03:19,503 --> 00:03:21,223 And we're going to ask questions about that. 49 00:03:21,223 --> 00:03:26,063 And we track that on a scorecard so we can start to see sentiment on an experience that we deliver. 50 00:03:26,063 --> 00:03:30,183 And that really helps us iterate and refine that once we launch. 51 00:03:30,183 --> 00:03:32,143 And then the final thing is outcomes, right? 52 00:03:32,143 --> 00:03:33,463 So when I drive an 53 00:03:33,623 --> 00:03:34,943 Epic experience. 54 00:03:35,503 --> 00:03:36,783 I'm looking for outcomes. 55 00:03:36,783 --> 00:03:38,943 Some of them are business, some of them are member. 56 00:03:39,423 --> 00:03:45,103 But ultimately, outcomes, interactions, and VOC is how we really make decisions at Wellmark on the experience team. 57 00:03:45,743 --> 00:03:47,903 I've been at Wellmark for four years. 58 00:03:48,223 --> 00:03:52,183 Before that, I was at a company called Kingland, right down the road here in Ames, Iowa. 59 00:03:52,183 --> 00:03:55,743 I spent 5 1/2 years there as a product manager and product analyst. 60 00:03:56,463 --> 00:03:57,743 Got a lot of good experience there. 61 00:03:57,943 --> 00:04:03,263 I was also part of the Technology Association of Iowa's ITLI, which is their leadership program last year. 62 00:04:03,263 --> 00:04:04,223 I was a graduate of that. 63 00:04:04,623 --> 00:04:08,463 And I'm a DJ, 20 years in the wedding and corporate event space. 64 00:04:08,543 --> 00:04:11,423 And we've had a lot of talks about AI today. 65 00:04:11,663 --> 00:04:18,063 I dove in last year, actually, into creating my own music, and I leveraged Suno AI to do the vocals. 66 00:04:18,303 --> 00:04:20,303 I produce a lot of my own beats and all that. 67 00:04:20,823 --> 00:04:23,023 It's EDM house music, but I didn't have a singer. 68 00:04:23,103 --> 00:04:29,663 And during JC's keynote, I was loving that he was bringing a little bit of AI music flavor into that because that hit home with me. 69 00:04:29,663 --> 00:04:36,703 So I have 16 songs out on all streaming platforms, so if you feel like a little workout music later on, you can search me up and find my music out there. 70 00:04:37,823 --> 00:04:39,743 That's potentially, potentially. 71 00:04:40,943 --> 00:04:42,623 So let's talk about the problem statement, right? 72 00:04:42,623 --> 00:04:44,383 This is not a product demo, as I mentioned. 73 00:04:44,863 --> 00:04:47,023 This is really a story about how we got here. 74 00:04:47,583 --> 00:04:52,463 As AI adoption accelerates, a lot of the conversation focuses on complexity and capabilities. 75 00:04:53,063 --> 00:04:59,103 What technology can do, what we're going to focus on, where it fits in within our organization, how it can help our people. 76 00:04:59,783 --> 00:05:04,223 All of those things, if applied carelessly, can be not good for your organization and culture. 77 00:05:04,743 --> 00:05:07,423 So we're going to talk about that experience, not just the technology. 78 00:05:07,663 --> 00:05:09,903 We base everything in personas, as I mentioned. 79 00:05:09,903 --> 00:05:22,143 We have our great research team that will do foundational research that yields journey maps, personas, all of the things that I need as an experience lead to make a decision on strategy of how we approach something, which is so great to have that ability on our team. 80 00:05:22,783 --> 00:05:26,943 We did some testing around just searching benefits, which is a large problem. 81 00:05:26,943 --> 00:05:28,703 You guys may have ran into this before. 82 00:05:29,263 --> 00:05:30,463 Finding out what's covered, 83 00:05:30,943 --> 00:05:31,823 How to get service? 84 00:05:31,823 --> 00:05:33,743 Can I get this surgery? 85 00:05:33,983 --> 00:05:35,383 Is this preventative covered? 86 00:05:35,383 --> 00:05:36,183 All those things, right? 87 00:05:36,183 --> 00:05:40,143 It's complex, depends on your plan and your familiarity with healthcare. 88 00:05:40,143 --> 00:05:48,343 So what we looked at here was looking at trying to figure out how participants in that space want to search for benefits. 89 00:05:48,343 --> 00:05:51,263 And surprise, they want to use regular language. 90 00:05:51,343 --> 00:05:53,223 Like they just want to ask it questions. 91 00:05:53,223 --> 00:06:00,623 And I think we have probably ChatGPT on the commercial side to blame for that because everybody's using it off their mobile phone on the side of their desk. 92 00:06:00,623 --> 00:06:00,823 So 93 00:06:01,103 --> 00:06:06,623 Querying and asking standard questions in human language is kind of the norm now. 94 00:06:06,863 --> 00:06:14,303 And so we thought, well, how are we going to solve that for our complex benefits when it comes to medical jargon and all of that stuff? 95 00:06:14,543 --> 00:06:15,503 How are we going to do that? 96 00:06:15,823 --> 00:06:17,183 So that's what I want to talk about today. 97 00:06:17,903 --> 00:06:19,263 And I want you guys to meet Sarah, right? 98 00:06:19,263 --> 00:06:21,183 So I want to start with a little bit of a story. 99 00:06:21,863 --> 00:06:24,703 Imagine you guys are a member, which I think a lot of you in this room are. 100 00:06:24,943 --> 00:06:29,103 You just got a bill in the mail, and it's higher than expected, but you don't get why. 101 00:06:29,623 --> 00:06:30,703 So what do you do, right? 102 00:06:30,863 --> 00:06:31,823 You do what all of us do. 103 00:06:31,823 --> 00:06:35,663 You go online, you search, you type something like, why wasn't my visit covered? 104 00:06:35,663 --> 00:06:36,943 Why was this denied? 105 00:06:37,743 --> 00:06:39,503 And what you get back isn't an answer. 106 00:06:39,503 --> 00:06:40,863 You usually get PDFs. 107 00:06:41,023 --> 00:06:42,623 Sometimes you'll get legal language. 108 00:06:42,943 --> 00:06:48,543 Sometimes you get benefit summaries written for compliance, not comprehension from a member point of view. 109 00:06:49,263 --> 00:06:54,943 So now you're frustrated, not only because the information doesn't exist, but because it feels impossible to find your answer. 110 00:06:55,543 --> 00:07:00,703 So eventually what you do, call customer service, right, which is our highest cost channel to serve our members. 111 00:07:01,183 --> 00:07:03,743 So you get the information from the customer service rep. 112 00:07:03,783 --> 00:07:09,863 A human has to translate that complex information in real time under pressure and deliver that for you, right? 113 00:07:09,863 --> 00:07:15,503 That moment where that search failed and support takes over is really where our story begins. 114 00:07:15,823 --> 00:07:19,503 So I want to show a quick video that I think will really hit home with this audience. 115 00:07:19,983 --> 00:07:21,423 And I'm going to switch over. 116 00:07:23,103 --> 00:07:24,463 Sorry, I'm not sure what that is. 117 00:07:28,383 --> 00:07:29,103 Okay. 118 00:07:34,513 --> 00:07:40,393 Jamie is about to have her first baby, so she goes to mywellmark.com to understand her medical benefits. 119 00:07:40,953 --> 00:07:44,233 She scrolls and scrolls and scrolls. 120 00:07:44,633 --> 00:07:46,153 Hundreds of options. 121 00:07:46,633 --> 00:07:48,953 The answers are there, but she can't find them. 122 00:07:49,753 --> 00:07:52,313 Frustrated, Jamie gives up and calls for help. 123 00:07:53,353 --> 00:07:56,473 That happened far too often, so Wellmark fixed it. 124 00:07:57,303 --> 00:08:02,783 Using AI, we created a new way to search, one that understands the way people actually talk. 125 00:08:03,423 --> 00:08:08,703 Now Jamie types one word, and natural language search instantly finds the right coverage. 126 00:08:09,663 --> 00:08:11,023 Suddenly, it's all there. 127 00:08:11,343 --> 00:08:15,983 Prenatal care, postnatal support, even breast pumps she didn't know were covered. 128 00:08:16,543 --> 00:08:21,263 One search, clear answers, and Jamie can get back to what matters most. 129 00:08:22,223 --> 00:08:26,623 Fewer calls, faster help, a more efficient system for everyone. 130 00:08:28,623 --> 00:08:32,063 So as you guys can imagine, pretty awesome, right? 131 00:08:32,143 --> 00:08:35,263 For our members that are calling and trying to find this stuff, they could self-serve. 132 00:08:35,903 --> 00:08:39,183 Our customer service agents can leverage this as well to help serve members. 133 00:08:39,823 --> 00:08:40,903 And that's super important, right? 134 00:08:40,903 --> 00:08:46,063 Because that's going to give them the value as being a Wellmark member, maybe that another health insurance carrier might not do. 135 00:08:46,463 --> 00:08:47,823 So we talk about the core problem. 136 00:08:48,223 --> 00:08:50,223 Members don't search for benefits, right? 137 00:08:50,223 --> 00:08:51,143 They ask questions. 138 00:08:51,143 --> 00:08:51,983 That's what they do. 139 00:08:52,303 --> 00:08:53,023 Is it covered? 140 00:08:53,143 --> 00:08:54,143 What do I do next? 141 00:08:54,383 --> 00:08:55,343 Why do I owe this? 142 00:08:55,783 --> 00:08:57,663 That's the fundamental issue here. 143 00:08:57,663 --> 00:08:58,943 Members don't search, right? 144 00:08:58,943 --> 00:08:59,943 They ask those questions. 145 00:08:59,943 --> 00:09:02,703 So search is assuming people know what to ask for. 146 00:09:03,103 --> 00:09:05,743 Benefits assume people know how the coverage works. 147 00:09:05,903 --> 00:09:07,823 Neither of those assumptions are true, however. 148 00:09:08,303 --> 00:09:14,503 So there's a constant mismatch between how the systems are built and how those systems behave. 149 00:09:14,503 --> 00:09:17,423 And that's really what we're going to focus on today with that core problem I mentioned. 150 00:09:17,983 --> 00:09:19,183 Why does the search fail? 151 00:09:19,703 --> 00:09:22,223 Traditional search assumes 2 things, right? 152 00:09:22,303 --> 00:09:23,743 Actually, three things when I think about it. 153 00:09:24,063 --> 00:09:28,223 The first is that users know the right words, which they don't always know the right words to search. 154 00:09:28,463 --> 00:09:30,623 Second, that the content's readable. 155 00:09:31,023 --> 00:09:33,823 And 3rd, that answers all live in one place. 156 00:09:34,223 --> 00:09:37,983 Sadly, in healthcare, they don't live all in one place. 157 00:09:38,303 --> 00:09:39,423 This really breaks down. 158 00:09:39,743 --> 00:09:46,143 Terminality varies between plans, contents fragmented, and answers depend on context. 159 00:09:46,743 --> 00:09:53,903 That plan, the claim, the timing, and all the expectations that the member has, all of those vary depending on the situation for the member. 160 00:09:54,063 --> 00:09:59,703 So that can be a big struggle on why they can't get answers and why they can't search on what they're looking to find. 161 00:10:00,303 --> 00:10:04,543 So when search fails, customer service absorbs the cost for us. 162 00:10:04,543 --> 00:10:07,103 And I mentioned that that's the highest channel cost that we have. 163 00:10:07,503 --> 00:10:21,743 So calls increased, handle times go up, our customer service agents are forced to act as like translators between what the member's asking in that complex benefit question and really giving them that answer that'll help them in the real human situation. 164 00:10:22,063 --> 00:10:24,783 So this is not really a digital experience problem. 165 00:10:24,783 --> 00:10:26,863 It's kind of an operational one if you look at it. 166 00:10:27,183 --> 00:10:36,863 So this just shows kind of generically how, you know, someone would call, the confusion, a member calls, a customer service representative has to translate that, which leads to longer handle time and obviously 167 00:10:37,343 --> 00:10:38,783 less customer satisfaction. 168 00:10:38,783 --> 00:10:45,343 We all want to make sure that we can handle those member requests as soon as they come into customer service and get the member what they need when they need it. 169 00:10:46,583 --> 00:10:49,743 I'm also going to talk here about why this is a hard AI problem. 170 00:10:50,863 --> 00:10:55,183 If it were easy to apply here, I mean, it would be everywhere already. 171 00:10:55,503 --> 00:10:58,223 It's slowly starting to get to a place where AI is embedding everywhere. 172 00:10:58,463 --> 00:11:02,223 But in terms of health care and searching, it's not there yet, right? 173 00:11:02,863 --> 00:11:05,423 These health care benefits involve regulated content. 174 00:11:05,583 --> 00:11:09,543 I mentioned fragmented systems and really 0 tolerance for hallucinations. 175 00:11:09,543 --> 00:11:16,303 You couldn't imagine someone wanting to do a preventative service, a heart surgery, transplant, or something very serious and 176 00:11:16,743 --> 00:11:21,463 getting information that it's covered, and then they go and have the surgery, and then they're stuck with a $100,000 bill. 177 00:11:21,463 --> 00:11:22,703 I mean, that happens. 178 00:11:22,703 --> 00:11:24,223 We hear stories of that happening. 179 00:11:24,743 --> 00:11:29,023 I'm sure you guys maybe know people that have had issues with getting the wrong information. 180 00:11:29,023 --> 00:11:34,743 So hallucinations, obviously, as you know, in AI can happen, and that's, there's zero tolerance in the healthcare space. 181 00:11:34,743 --> 00:11:41,343 So getting something almost right can actually be worse than getting it wrong in our profession in healthcare. 182 00:11:41,343 --> 00:11:45,583 So the reality is heavily shaped on how we approach this work. 183 00:11:46,143 --> 00:11:49,903 But you can see here there's a lot to be considered in this domain specifically. 184 00:11:50,863 --> 00:11:52,543 So let's begin with internal testing. 185 00:11:53,583 --> 00:11:57,023 We used a hackathon actually to do this work. 186 00:11:57,503 --> 00:12:00,703 Wellmark decided last year to do our first official hackathon ever. 187 00:12:00,703 --> 00:12:01,823 It was three days. 188 00:12:02,143 --> 00:12:03,583 You got to partner with anyone you wanted. 189 00:12:03,583 --> 00:12:10,463 You can submit ideas for a period of two weeks, and then you can actually request to be on a team, and then the teams were assembled for those three days on site. 190 00:12:11,423 --> 00:12:12,863 It was actually an incredible experience. 191 00:12:13,023 --> 00:12:18,223 This was the idea that my team had to solve search for members using AI. 192 00:12:18,703 --> 00:12:21,903 And we actually, out of 19 teams, we actually won. 193 00:12:21,903 --> 00:12:23,503 We placed first place last year. 194 00:12:23,503 --> 00:12:31,903 And because of that, Wellmark funded the work, which I just thought was super cool for a company to not only sponsor Hackathon for three days, but then fund the winning project. 195 00:12:32,543 --> 00:12:38,383 So we funded that last year, and we're getting ready to implement it next month for our members, which is just incredible. 196 00:12:38,903 --> 00:12:47,663 A year, yeah, it took a long time, but you can imagine the legal conversations and compliance conversations we've had to have and go back and forth with what we're actually saying on the screen. 197 00:12:48,103 --> 00:12:53,983 I think we've landed on AI assist with a bunch of legal language, really small. 198 00:12:54,303 --> 00:12:55,423 So there's that too. 199 00:12:55,423 --> 00:12:59,583 But we leveraged, like I said, time-boxed, low-risk, cross-functional. 200 00:12:59,583 --> 00:13:03,503 So we had developers, we had analysts from different parts, we had operations folks. 201 00:13:03,503 --> 00:13:05,703 We had about 12 people on my team for those three days. 202 00:13:05,703 --> 00:13:09,783 And then we presented to leadership and everybody else, which was really fun, right? 203 00:13:09,783 --> 00:13:16,063 So that rapid failure and that controlled structure, took that from JC this morning from his keynote, 204 00:13:16,703 --> 00:13:19,983 That was key for the hackathon, so they're getting ready to do that again this year. 205 00:13:20,303 --> 00:13:31,823 I'm looking at some potential teams to join, but this is a very cool way to not only get everybody together from a culture perspective, but actually deliver working stuff now that we've implemented, which I think is super awesome. 206 00:13:32,783 --> 00:13:33,583 So let's keep talking. 207 00:13:33,583 --> 00:13:34,943 The hypothesis, right? 208 00:13:35,183 --> 00:13:36,543 Our hypothesis was simple. 209 00:13:36,783 --> 00:13:41,343 What if people could ask questions in their own words and the system met them halfway? 210 00:13:41,503 --> 00:13:49,183 Not a chatbot replacing humans, not a magic answer engine, but a bridge between human language and that complex benefit logic, right? 211 00:13:49,823 --> 00:13:52,303 We thought about Amazon Alexa as like, 212 00:13:52,783 --> 00:13:55,343 vibes and we were thinking like, how do we want this to feel? 213 00:13:55,903 --> 00:13:58,863 We said, why couldn't you just ask Alexa like if it's covered, right? 214 00:13:58,863 --> 00:14:03,103 So that jokingly became how we thought about this. 215 00:14:03,103 --> 00:14:07,463 How easy would it be just to be like, are the things I walk around on covered, right? 216 00:14:07,463 --> 00:14:09,023 Which would be orthotics, right? 217 00:14:09,023 --> 00:14:09,743 But how does 218 00:14:10,303 --> 00:14:12,823 search know that you're talking about feet, right? 219 00:14:12,823 --> 00:14:14,783 And that's where AI comes in, right? 220 00:14:14,783 --> 00:14:16,383 That's where that language model comes in. 221 00:14:16,383 --> 00:14:18,223 So it was pretty cool to see. 222 00:14:18,223 --> 00:14:26,783 We had a working demo for our hackathon debut, and we had a bunch of executives coming up trying to like stump it, trying to like get it to not bring back benefits. 223 00:14:26,783 --> 00:14:28,863 But surprisingly, it worked very well. 224 00:14:28,863 --> 00:14:30,863 And they were like, okay, we can see the benefit in this. 225 00:14:31,503 --> 00:14:36,623 So conceptually, the architecture, legal sadly wouldn't let me put anything in here that we used. 226 00:14:37,263 --> 00:14:39,343 But you guys can use your imagination. 227 00:14:39,343 --> 00:14:46,943 We have a repository here, so that's all of our documents that we have, benefit documents, think all of that historic document. 228 00:14:47,423 --> 00:14:50,823 We're using AI retrieval, so semantics, vector matching. 229 00:14:50,823 --> 00:14:56,863 We use chunking methods, which some of the big service providers offer that in their language models. 230 00:14:57,743 --> 00:15:00,543 The chunking is how it takes that segment of 231 00:15:00,863 --> 00:15:03,023 information and displays it to the member. 232 00:15:03,623 --> 00:15:05,263 And there's various different methods of chunking. 233 00:15:05,263 --> 00:15:11,823 So we tried and tested a bunch of ones until we kind of got the result that we felt was going to give the member the best result, which was cool. 234 00:15:12,103 --> 00:15:20,943 And then ultimately, we had some lambdas and some service layers we built to connect it to our member portal and our customer service CRM, things like that, which is great. 235 00:15:20,943 --> 00:15:23,663 So this is a little bit overview of the conceptual architect. 236 00:15:23,663 --> 00:15:26,783 If you want to get involved with that after, I'm more than happy to dive into that. 237 00:15:27,423 --> 00:15:28,783 Let's talk experience, right? 238 00:15:29,263 --> 00:15:36,703 So what we did, what we deliberately did not do was surface raw policy text. 239 00:15:37,183 --> 00:15:39,103 We didn't pretend that AI was certain. 240 00:15:39,183 --> 00:15:40,943 We didn't optimize for cleverness. 241 00:15:40,943 --> 00:15:44,383 We optimized for clarity, restraint, and trust, right? 242 00:15:44,383 --> 00:15:45,983 We just heard all about trust. 243 00:15:45,983 --> 00:15:47,743 How is our members going to trust? 244 00:15:47,743 --> 00:15:48,863 If we get a wrong answer, 245 00:15:49,423 --> 00:15:52,143 And they go to the doctor and it's not covered and they used AI. 246 00:15:52,143 --> 00:15:53,263 They're not going to trust Walmart. 247 00:15:53,263 --> 00:15:55,263 They're not going to trust anything that we tell them. 248 00:15:55,503 --> 00:16:04,063 So it's super important for us to be clear and really restrain ourselves because in healthcare, that confidence without that accuracy is super dangerous for our members. 249 00:16:04,863 --> 00:16:06,223 We talked about the current climate. 250 00:16:06,943 --> 00:16:09,823 We all know kind of the story around United Healthcare and 251 00:16:10,543 --> 00:16:18,383 That whole sad thing that happened, we are not using AI at Wellmark for any healthcare outcomes, any determinations of claims. 252 00:16:18,383 --> 00:16:19,743 We are not doing any of that. 253 00:16:19,983 --> 00:16:22,943 This is the first, we're using it internally as a workforce. 254 00:16:22,943 --> 00:16:27,423 We have copilot and all of that stuff, but we're not leveraging it for member-facing things by any means. 255 00:16:27,423 --> 00:16:30,223 This would be the first thing that we're leveraging AI for. 256 00:16:30,623 --> 00:16:34,063 But I feel like it's a very controlled application of AI. 257 00:16:34,063 --> 00:16:35,183 It's not generative. 258 00:16:35,503 --> 00:16:36,783 It's very specific. 259 00:16:36,783 --> 00:16:39,823 So that's kind of where we're at with the experience lens on this. 260 00:16:40,703 --> 00:16:42,463 Two audiences, two jobs, right? 261 00:16:42,463 --> 00:16:45,663 So I mentioned we have our members and we have our customer service agents. 262 00:16:46,703 --> 00:16:48,623 They don't need the same answers, right? 263 00:16:48,623 --> 00:16:53,423 Members need the clarity, they need the confidence, and they need that empathy. 264 00:16:53,543 --> 00:16:55,423 And what are their next steps, right? 265 00:16:55,423 --> 00:16:56,703 That's what they're looking for. 266 00:16:57,263 --> 00:17:04,063 The CXAs, however, they need the speed and the traceability to get that information quickly to deliver to the members. 267 00:17:04,063 --> 00:17:10,543 So it's two audiences with two completely separate jobs, but we designed for both of them at the same time. 268 00:17:10,783 --> 00:17:15,023 And that forced us to think more carefully about the experience outcomes that we're trying to drive, right? 269 00:17:15,023 --> 00:17:18,943 So that's the self-service side of things versus the customer service side of things. 270 00:17:19,183 --> 00:17:27,023 But either way, they both can use the same solution, which I thought was really awesome and it was really impactful, I think, to the leadership group to hear 271 00:17:27,343 --> 00:17:33,263 we can actually not only solve members' issues, but we can solve speed and clarity around what our CXAs are doing. 272 00:17:34,143 --> 00:17:35,343 So what broke first, right? 273 00:17:36,623 --> 00:17:40,223 AI struggled where humans also struggle, which is not a surprise. 274 00:17:40,223 --> 00:17:41,263 That's ambiguity. 275 00:17:41,703 --> 00:17:46,303 So conflicting sources, vague benefit terms, those edge cases. 276 00:17:46,543 --> 00:17:51,023 We saw early tendencies toward overconfidence, which reinforced the need for those guardrails. 277 00:17:51,663 --> 00:17:56,463 We talk about enterprise-wide initiatives a lot in this space as well. 278 00:17:56,943 --> 00:17:58,223 This is an enterprise-wide 279 00:17:58,943 --> 00:17:59,503 initiative. 280 00:17:59,503 --> 00:18:01,503 We had to get the buy-in from the stakeholders. 281 00:18:01,783 --> 00:18:13,503 A lot of different departments are siloed, but we're working to come together and say, like, how can we all leverage the same tool so we can get confidence in deploying this to our members and have a better experience when it comes to finding their benefits? 282 00:18:14,143 --> 00:18:15,183 So what surprised us? 283 00:18:16,143 --> 00:18:19,903 It was how much less AI actually needed to do. 284 00:18:20,303 --> 00:18:24,863 Smaller, well-scoped answers actually built more trust with our testers. 285 00:18:25,503 --> 00:18:32,823 Retrieval, beat generation, and transparency built confidence, even when the answer was it depends, right? 286 00:18:32,823 --> 00:18:44,143 So it was really about making sure that we keep that level of trust when we're having interactions with our members and really restraining the scope around it versus just letting it go. 287 00:18:44,543 --> 00:18:46,063 It was really important as well. 288 00:18:46,383 --> 00:18:50,703 So I have a couple of minutes left here, and I want to make sure I allow time for questions. 289 00:18:51,023 --> 00:18:52,463 From an experience lesson, 290 00:18:53,503 --> 00:18:55,983 This isn't an experience on its own it's part of 1. 291 00:18:55,983 --> 00:18:59,183 It can reduce cognitive load, but not remove complexity. 292 00:18:59,183 --> 00:19:02,383 It can assist, but it still needs humans in the loop, right? 293 00:19:02,383 --> 00:19:03,743 So that's what we're looking at here. 294 00:19:04,223 --> 00:19:09,343 We just were awarded some Corporate Insights Awards, which is an industry kind of... 295 00:19:10,263 --> 00:19:13,983 recognition for best mobile app and desktop app. 296 00:19:13,983 --> 00:19:15,263 We finished second on that. 297 00:19:15,263 --> 00:19:25,783 That's 24 health insurance carriers were rated, and we were #2 on that, right behind Anthem, which Anthem has a massive budget compared to what Walmart has in terms of experience. 298 00:19:25,783 --> 00:19:31,183 So I think we're doing some things right in this space, and this just kind of shows that some of the recognition is coming our way. 299 00:19:31,543 --> 00:19:36,143 From A technical perspective, bounded domains, clear governance, strong content, 300 00:19:36,783 --> 00:19:41,023 Responsible AI is an experience decision, not just a platform decision. 301 00:19:41,583 --> 00:19:43,503 A few organizational lessons we learned. 302 00:19:44,223 --> 00:19:46,463 This changed kind of how we collaborate as teams. 303 00:19:46,983 --> 00:19:51,743 Experience, content, operations, technology can't operate independently anymore, right? 304 00:19:51,903 --> 00:19:55,503 AI forces alignment or exposes the lack thereof. 305 00:19:56,223 --> 00:19:59,583 So our hackathon part 2, like I said, is coming up next. 306 00:19:59,903 --> 00:20:01,103 But this could scale 307 00:20:01,943 --> 00:20:02,863 beyond healthcare, right? 308 00:20:02,943 --> 00:20:05,503 We're talking other domains, insurance, government, higher ed. 309 00:20:05,663 --> 00:20:07,343 These challenges show up everywhere. 310 00:20:07,823 --> 00:20:14,303 It's just about allowing people to ask the right questions in the way that they want to and providing the answer to them. 311 00:20:15,263 --> 00:20:17,503 What this is not, though, is an AI reality check. 312 00:20:17,503 --> 00:20:23,023 It's not a replacement for humans, not a knowledge oracle, and of course, not without guardrails. 313 00:20:23,583 --> 00:20:25,663 I'll leave this last quote up here before I end. 314 00:20:26,263 --> 00:20:39,983 This was from one of our team leaders that was part of our testing group, and she, I'm not going to read the quote, but basically she handles our customer service team, and she just really saw the value of having something like this for her CXAs to use to help members. 315 00:20:40,543 --> 00:20:45,823 She thought this could be magnified by 200x agents if we implemented this at Wellmark. 316 00:20:46,063 --> 00:20:54,223 So it's a very promising piece of technology, not only for internal operations, but for our customers that, you know, use Wellmark Insurance. 317 00:20:54,743 --> 00:21:00,783 And with that, I know I ran through that a little fast, but we are a little short on time, but I wanted to open up for a question or two, if you guys have any. 318 00:21:01,463 --> 00:21:04,783 Yes, do you have metrics that measure sales queries? 319 00:21:05,183 --> 00:21:07,343 Yes, we do, so we're compiling that right now. 320 00:21:07,423 --> 00:21:11,983 I mentioned we're in UAT with all of our stuff right now, and we're finalizing all the testing. 321 00:21:12,423 --> 00:21:17,343 And we still have to get through the rest of the legal compliance stuff, but they are documenting that. 322 00:21:17,343 --> 00:21:19,823 I haven't been as close to the implementation team, sadly. 323 00:21:19,823 --> 00:21:26,463 I've been, I was part of the hackathon team, but we have another AI-focused team that's doing the implementation at Walmart, but more than happy to find out. 324 00:21:26,463 --> 00:21:31,343 Is there a bar, like a percentage of success or failure? 325 00:21:31,503 --> 00:21:35,063 No, I'm not even sure, to be honest with you, but I'd be more than happy to find out for you. 326 00:21:35,503 --> 00:21:35,743 Yeah. 327 00:21:36,463 --> 00:21:40,623 One of the things I mentioned is like you've got to go in and treat your data. 328 00:21:41,103 --> 00:21:44,983 So you find out that on your documentation there were a lot of contradictions. 329 00:21:46,863 --> 00:21:53,743 Yep, we had to, we spent a lot of time and effort over the last year cleaning up our benefits document catalog and really optimizing that. 330 00:21:53,743 --> 00:21:56,943 So it, like everyone says, you got to have good data to get good output. 331 00:21:56,943 --> 00:21:58,543 So we did some of that legwork. 332 00:21:58,543 --> 00:21:59,823 It was in a pretty good spot before. 333 00:21:59,823 --> 00:22:04,303 It just was a lot, very complex and hard to understand for most people. 334 00:22:05,303 --> 00:22:09,743 So if it has a problem that it can't answer, does it have kind of an escape mechanism to talk to a human? 335 00:22:09,903 --> 00:22:15,423 I'm using an example for last year, it took me 6 hours with AT&T to fix the problem once. 336 00:22:15,583 --> 00:22:15,703 Wow. 337 00:22:15,823 --> 00:22:17,983 Because every time I went on, I had to... 338 00:22:18,503 --> 00:22:21,823 menu, AI, get to a person, get to the next person. 339 00:22:22,343 --> 00:22:23,503 It was highly frustrating. 340 00:22:23,663 --> 00:22:25,583 Yeah, so that's a great call out. 341 00:22:25,823 --> 00:22:28,063 Our design team has put in some things. 342 00:22:28,063 --> 00:22:31,103 So if you're searching and you're just not getting, I think it's like 2 searches. 343 00:22:31,103 --> 00:22:35,903 If it doesn't come back, there's of course some chat bot help or send a secure message or call customer service. 344 00:22:35,903 --> 00:22:38,383 So we're not going to eliminate that, but we hope that self-serve. 345 00:22:39,023 --> 00:22:42,703 You know, that's always the way, I think, is members want to self-serve if they can. 346 00:22:42,863 --> 00:22:44,463 It's just making it easy for them to do that. 347 00:22:44,463 --> 00:22:47,183 Just learning yours get you to a person, but that works in other ones. 348 00:22:47,543 --> 00:22:47,983 What was that? 349 00:22:47,983 --> 00:22:48,743 Yeah, that's true. 350 00:22:50,783 --> 00:22:52,983 That works for CVS. 351 00:22:53,343 --> 00:22:54,303 Oh, good to know. 352 00:22:54,303 --> 00:22:55,743 They're our pharmacy partners. 353 00:22:55,743 --> 00:22:55,863 Cool. 354 00:22:58,143 --> 00:23:00,303 All right, folks, I think that puts us pretty much at time. 355 00:23:00,303 --> 00:23:04,143 So everybody, please join me in thanking Nick for his wonderful presentation. 356 00:23:08,343 --> 00:23:10,223 That brings us pretty close to the end of the day, folks. 357 00:23:10,303 --> 00:23:18,623 There should be some closing comments and some stuff from Robert Ivester of NIST back in the main room, and then followed by our final closing comments from Paul. 358 00:23:18,703 --> 00:23:19,423 Thank you for being here.