All right. We are live. I would like to say a very warm welcome to everyone that has joined this webinar, where we're going to dive into how did Forza, together with Implement, achieve real efficiency gains using AI. Today we're going to talk a little bit about what was the motivation for this, what are the actual results. Now, I know you have been teased a little bit, but we do have more coming. Then, how do you actually get started with AI, and how did we do in this project? Then we're going to talk a little bit about the solution design that we did, and we're going to talk about evaluation and compliance. So, there's a lot of goodies coming. Today, I know the invitation said 30 minutes, but we will be rolling for 45 minutes, so I'm sorry if I'm going to take a little bit of your lunch break. I hope that is all right. If there are any questions, please post them in the chat. We have good colleagues here to help you answer those. And if there is time at the end, I will try to answer some questions. Okay. So, today, it's me presenting. My name is Victor, and I'm a consultant here at Implement in the digital transformation practice, and I mainly sit with AI, and especially generative AI. I was supposed to be joined by Nikolas from Forza, but unfortunately, he had to make a last-minute cancellation. Now, I will try my best not to butcher his slides completely, but it will be hard to replace 17 years of experience from Forza, but I will do my best. So, obviously, Implement, we are a consulting firm, but who's Forza? So, Forza is pension fund services, so they're owned by pension funds, and that includes as this case is a call center with highly skilled advisors that help the members do the right things with their pension. Right. The purpose of today is to go into this project that we did, so it's kind of a case study of Forza, and I think many of you have good ideas, you know, how can we use AI, but actually, this is also about how do we actually, you know, get these efficiency gains, get the potential that AI promises. So, how do you move from a proof of concept or an idea to actually having something in production that will create the effects that you desire? So, back to the case. It all started with a quote, with a challenge for Forza, which was that the advisor says, we spend far too much time documenting calls, it's repetitive and tedious, the quality suffers, and it takes time away from what matters most, which is advising and supporting our members. And all the while call volumes continue to rise, and we struggle to keep up without compromising. So, that was kind of the stepping stone for this project. And I think it really highlights that you should not be doing AI for the sake of AI, but we should do it because if it's a challenge that is fit for AI. So, it's all about, you know, capturing the business value and not about implementing some cool AI stuff. Right. So, the challenge here is that member service is the heart of Forza, and, you know, keeping the members of the pension funds happy, giving them good advice. And that puts pressure on the advisors. So, we can infer from the quote a couple of things. The first thing is that there is a growing demand. There's an external pressure, you know, on Forza to deliver highly, you know, skilled advisors that can do good advisory. And actually, and actually, the projection in calls from now and a year onwards is about a 30% increase. So, how can we help that? Then, it's time consuming to do documentation. Every time that you pick up the phone, you have to write something about what did we talk about, what was agreed. And with a growing demand, that will be even more time consuming. Right. Then, another point is that the advisors don't really want to be doing this documentation. I mean, their job is to speak with people and help the members. So, it's all about, you know, the service and, you know, taking care of the members. Additionally, it can take focus away from the actual conversation. So, if you're thinking about writing notes. So, if you're thinking about writing notes, thinking about how can I frame this in the documentation, you might be a bit distracted. Lastly, all of these points or the first three points kind of points to this one, which is data quality. So, if there's a lot of calls incoming, you don't have the time to do it and you maybe don't want to do it actually, then the data quality will suffer. So, that is kind of the four points that we seek to address. Then, how did we do that? Now, it says here, enterprise integrated AI solution that enhances documentation workflow. So, we looked into the challenge and we tried to design an AI solution that can enhance the workflow of the advisors and not replace them. As it is very important that having, you know, as it is very important that having, you know, the human, having the warmth of a human in this session with the members. So, we can handle the growing demand without adding headcount. I mean, that's a huge plus. It's a quick process that happens in the background. So, we do not have to alter how the advisors work, only slightly. Then, we're also kind of lightening the burden of thinking about this documentation and, you know, thinking of doing it afterwards and also in the call. And then lastly, of course, self-explanatory to some extent is that the data quality will improve significantly because there will always be a documentation and it will always follow the same structure. So, that was kind of how we dealt with those challenges. Now, what is most surprising and most fantastic is the results of this solution. So, based on how much time advisors used on writing documentation before and how much they do now, we see that already there is a 10% increase in efficiency. So, they actually save time and they get more time doing what they actually want to do. And also, it's not just saving time. And also, it's not just saving time but it's also increasing the quality. So, from the get-go, the advisors actually preferred the AI documentation over the human documentation. And how we measured that, I will get back to. But this is quite significant numbers and something that is really, really cool. So, how did we see that AI writes better than humans? Why did they pick it 80% of the time? Well, this graph, to some extent, tries to explain that. So, we had the counselors score each of these documents in a head-to-head battle on these different quality dimensions. And it kind of speaks for itself. So, we see that accuracy, conciseness and stuff like that. It is better. Right. So, that was kind of the teaser. Now, we'll take a step back and look a little bit about or look a little bit on the strategy from Forza but also on how the project actually developed. So, with all my respect to Nikolas, I will try my best here. So, Forza has in the past always been very keen on technology and has tried and tested many different use cases from RPA and natural language processing. And some of these emerging technologies has always been on the radar and is actually still in production at Forza. But, as we all know, our good friend, ChadGBT, came in and transformed how we look at processes and how we do our job. So, it was quite natural for Forza that a couple of years ago were looking at their IT strategy. So, they thought, okay, let's make AI an integral part of this strategy. And so, they did. And that is, of course, taking a lot of strategic, you know, answering a lot of strategic questions. But from the AI angle, it's also about finding out where are the challenges, where can we actually do some, where can we realize some business value inside of Forza. And they defined and described and most importantly prioritized among over 70 different AI use cases. And one of them were these AI-generated call documentations. And that's what we helped them build. Right. Okay. And I have touched a little bit upon this. And I know that Nikolas, he could talk a bit more about this than I can. But I think the point of this slide is just to say that the member is at the center. So, that's the premise for success. We cannot just go out and create solutions, deploy this, deploy that. If we don't have the core business, as our goal. What is it that we want to achieve. And in this case, it was to provide even better service for the members. Right. So, I think that's quite important that we don't do it just because the technology is there, but we do it because we have a challenge. Right. Okay. So, how do you actually move from idea to impact in 10 to 15 weeks? I will talk a little bit about that. So, the first thing we did is that we had a lot of workshops. So, with all the people that actually met us, with all the subject matter experts, we sat down and we sketched out the most important points. So, for example, we had a lot of workshops with the advisors. So, we sketched out what is good documentation. What does it look like? What are the criteria? What are the keywords? All that kind of stuff. Because in the end, it's them who are going to use it. Then we also did workshops with other business areas, such as the infrastructure and network team. So, how do we successfully deploy this solution without too much hassle? And we'll also return a bit to that. So, gathering all of these insights, what was the first step? Then, of course, we needed to document and we needed to, you know, all the requirements for the solution need to be sketched out. And then we could start developing. And I think this is, I hope that some of you maybe can recognize this because this is quite, this is where I feel that a lot of people say, you know, a lot of organizations have done these steps. But now it becomes a little bit more tricky. So, I think that it's pretty, I would say, it's fairly easy to build proof of concepts with generative AI because it's so powerful out of the gate. But how do you actually move on from that step? And the first thing we did was to keep evaluating the system. So, is it good enough? What are the critical questions we need to ask such that we can, here at this webinar, show you now, show you, show you the graphs, show you the numbers that we did? So, it's, again, thinking about the business value and not so much about how, not so much thinking about AI this, AI that. Then, based on that feedback, based on that evaluation sessions we had with the advisors, we can adjust the solution. And that led us to the next step. When we were happy with that, we started the integration process. So, in this project, we were tapping into a hybrid setup. And we'll look a little bit on that. But just to say that, that is not easy and that can be challenging. The last thing is, of course, to set up the governance setup. How do we keep the performance high and who should do what? That is, of course, an integral part of having things in production. Right. Now, we're going to dive into the actual solution. So, what is it that we built? So, it's an AI-powered documentation assistant and it works in the background and it doesn't necessarily change the workflow of the advisors, but just enhances them to be even better, to be even faster without compromising on quality. So, the first thing was, of course, to sketch out some design principles for this solution. So, it had to be built on Forza's infrastructure. It needed to have human in the loop as a core feature. I think that is very important because, in the end, it is the advisor who has responsibility for what happened in that session. Then, we would like to have compliant data handling goes without saying. Fast processing speed. I mean, the advisor shouldn't be sitting and waiting for the documentation. It should be there instantaneously. Then, we would also like to build it such that it's model agnostic, meaning that the analogy here could be, we build together with Forza, we build a car, but we build it such that we can get gasoline wherever we want. The same thing, or no, the last thing, sorry, is that it had to use the same interface as the one the advisors already were using. So, we had to, yeah, sorry, okay, let's look at the workflow. So, the member will call and want some counseling, some advisory, and there will be a conversation. When that conversation ends, the advisors will, on their screen after a couple of seconds, have the AI documentation presented. So the only thing that we have changed is now that instead of them, you know, hanging up and then starting to write, there will be a draft already. Then they can skim through that text and then either accept it as it is or, of course, they're allowed to edit as well. So that's kind of the sounds pretty, pretty easy and pretty straightforward. But what happens in the background is that we build an integration to the phone system such that we can listen to the conversation. Then take that sound file and transcribe it using, of course, generative AI solutions. So it's a, in this case, a speech to text model. Then we take the transcription and we turn that into documentation. And we do that using a very customized, highly customized prompt that we built together with the advisors, of course, but also making sure that we are doing output parsing, just meaning that we always have the same structure of the documentation. Then that happens in the background. And that is what pops up at the screen after the advisors hang up the phone. Right. Then the original AI documentation is kept. And along with that, the one that was approved by the advisors that is stored on the core platform. And now that allows this continuous evaluation of the system. So we can see what was edited. Is there something that is always deleted? What is always added? And stuff like that. Very cool. So that's kind of, that's kind of it, I would say. But I mean, building AI solutions, building these functionalities is fairly easy. Then the next point would be, building, building those tools, building these functionalities and, to scale them. That is challenging. So tapping into a production environment, in this case, a hybrid set up, that can be transformative, but it also takes some effort. So now I won't be going through this in detail, but I will make some highlights. So the first thing is, what was important to us and to Forza is that we tapped into the current system. protocols. So what we built, of course, needed to be secure by design such that there is no cybersecurity flaws. Then what happens in the cloud is, of course, happening in the background. But as you see in the top right corner, it says database and it says templates. That means what we did is that we built a little feature on the site where the advisors can go in actually on a daily basis and they can test out new prompts. And if they like it, they can add it as a template and that will then go into the solution. So there is no need for the advisors to be or they're not dependent on developers. So that's a pretty cool feature. Now, how do you solve the prem cloud talking to each other? Well, in this case, we used a message bus, just meaning that there is this rule that only the on-prem can push and pull. So it's in charge. And we solved that by setting up a clever queue system that makes sure that all the advisors get the right documentation at the right time. And I think that's it for this drawing. If you would like to hear more about this in detail, reach out and we can discuss. But I think the main point is just that you can build a POC, a proof of concept, and that works pretty well. But how do you take it to 100%? That can be challenging and requiring some technical capabilities. Good. Now we're going to talk a little bit about the evaluation and the results. So now you have seen how we built this solution. But more importantly, how did we measure it? How did we evaluate it and make sure that we capture these efficiency gains? So first thing was, of course, to think about how can we measure the preference of the advisors? So that we did in kind of a head-to -head battle between documentations. So we took earlier calls and we looked at the human documentation and then we generated some AI documentation and that went head-to-head. Now, these two different pieces of documentation, we actually had to label them. So one might say that, okay, you need to have a blind test of this. So you cannot know which was made by which. But in this case, the quality of the AI documents were actually insanely high. So it was so obvious that it was AI generated. So there was no point in doing that really. So this is actually more or less how it looked like. And then along with a lot of other questions meant for collecting feedback. So for example, was there anything missing? What concepts were misunderstood? And stuff like that. So we can take that back and improve the system. So one thing is measuring the preferences, but also trying to understand them. So why is it that the AI is better? So again, a head-to-head battle where you choose between these seven quality documents that we find fitting. And this just goes to show that the AI is just really good at these quality documentations. So if we look at the numbers again, I did show this earlier, but it's pretty obvious that it's pretty obvious that, and this is just the baseline test that AI outperforms the human in this manner. Okay, so a little bit about the process. Some of this may be a bit repetitive, but I think it's really important to think about how to structure this process. So the first thing we did was to obviously create the baseline test. And in that case, that's where we got the 80%. And that's also very, that's a very good kind of gate or sanity check. So should we keep going with this solution? Is it even possible? So that's one thing. And the other thing is, of course, capturing a business value. Then use all that input from that survey to update the functionality, update the solution. And then again, when we have updated the solution, we took the AI documentation and we put that up against, now here it says better AI just for the fun of it, but we tested on the new data. And in this case, the advisors found that the new updated functionality that has produced this new output were actually preferred in 73% of the documentation or in the cases. So now I don't hope that this is too confusing. So now I don't hope that this is too confusing, but it's just to say in the baseline test, we beat the humans with AI. And in the second test, the new improved AI beat the original AI. So it's just to say that these updates and these feedback loops are super important when you want to maximize the AI usefulness. And of course, this is an iterative process. So this is something that you should keep doing. Right. The next thing here is, so now we have built a system, we built a solution that works. We have integrated it into a hybrid setup. It's fully working inside of Forza. And we know it works. We know the quality is good. But of course, we need to be aware of compliance. And I put this last. Maybe I should actually have put it first because compliance is the ally on this AI journey that you can never get rid of. And I think for me, it's important to highlight that this can never be a trade-off. And there will be risk-based decisions when you do these kinds of systems. And they need to be fully informed. Right. So the first thing we did was, you know, like a kickoff. So this goes back to the workshops. So we sat down with the legal department and find out, okay, what can we do and how should we do it? Then we started the serious work, I would say. So we had to do a data ethical assessment. I think most of you working with AI solutions, you have tried this already. This is something that you just need to do. We worked on a GDPR assessment. And I think that's fairly standard. But what we also did at Forza is that we created a consequence analysis. So we looked at what happened if the system fails. What are the suppliers? What are the models actually doing? So we mapped out a lot of different things such that we could say what is the risk of implementing this solution into our production environment. So a really cumbersome task, but it gave a lot of value and a lot of, you know, clarity and calmness for those that had to actually say, yes, we go forward with this solution. Then in respect of the EU AI Act and also just in general, when you do things inside of an organization, we created, now I've called it a compliance risk map, but it was, you know, kind of a map of decisions. So how risky are they and how should we go about them and what are the pros and cons? And we use this document or this map to get sign-off different places in the organization to ensure that we know there's a risk doing this. Maybe it's high, maybe it's low, but we know it's there and we say yes to that risk. Right. The last thing, which is pretty cool, I think Forza, they're really good at doing is that every time that they want to start a business, project, they have a project. They have a project project project, they have a project project, they have a very predefined process for how to pitch and how to actually create the proposal. So this proposal is an internal thing that we help with, but it kind of collects all the things that were done in one, two, three, and five into a single document and it gets a nice overview. And yeah, very good. So that was on the compliance side. Okay, now we've been through most of it, so I hope you have a clear impression of how we built this, how we kind of structured the process and how we evaluated the system. Lastly, I will talk a little bit about the future opportunities for Forza and for this solution particularly. So the first thing, which is quite obvious and I think we're already working on this, is to expand the solution. So now that we have built something that really works, we should also use it in other contexts. So right now, it's only used for the inbound calls, but why not use it on outbound calls as well? And we also in Forza do some web-based advising. Maybe we can tap into that as well. And lastly, we also have other places in Forza where the phone is picked up and it's quite, as we say in Danish, to the right leg to also use that technology there. Then, it says here future integration, but could also say future possibilities of this solution. So how can we, now that we have a lot of quality documentation, what does that bring? I mean, I mean, I mean, I mean, I mean, I mean, this data can be used for a lot of things downstream, creating a lot of insights into our members and into how we could probably, one could think that how to optimize and how to better the service. Additionally, so that's kind of on the data part. Additionally, there is, we can also leverage the technology. So now that we have built this system, well, or this very, this, how to say, well, we build this functionality, we can also use it in other areas, not only for, not only for, for, for voice to text, but, but how to structure documentation, how to generate documentation based off something is, is something that we can use in, in a lot of places inside of Forza. Right. Right. The last point here is that, you know, doing it right the first time is something that creates momentum. So, so this has really been a stepping stone for Forza. And, you know, really, they have really shown that we can succeed building with AI, we can succeed in creating business value. So, so, so, so this is something that, that, that really, really have inspired a lot of people in Forza. And, and, and it also gives them the, the confidence and, and, and, and, you know, they now, they now have a great trust in AI and a confidence to, to go and deliver on a lot of other use cases, a lot of other AI use cases. Right. Yes. So, so that was kind of on, on, on, on, on the outlook. So, so that was, you know, we can, you know, we can, you know, we can kind of go from, as it says here, cost to inside center, but there are some levers we can pull. And if you do it the right way, it, it can just be really powerful. And you can really get some cool numbers out of it. And, and it doesn't have to be, you know, doesn't, yeah. What I'm trying to say is just, it was a really cool project. And thank you so much to, to Forza also for, for leaning in and being part of this, of this webinar. That was all from me now. So I think I will try to take some of the questions. We do have a little bit of time. So Nils Christian, you are asking here, what was the full project timeline for the use case from idea scoping to project launch? Yeah. Yeah. So I'll try to, to answer to the best of my ability here. So from idea scoping, I think that was part of, of, of the AI strategy and, and saying, this is something we want to do. And they have taken, um, a lot of, a lot of strategic choices. For example, we want to build this, uh, and have it inside of Forza. It's not something that we want to buy as a service. Yeah. So it kind of starts there. Um, but, but getting back to, to how much time we used on it, it was that, um, from idea to scoping, I think it also said in the slide, but around 10 to 10 to 15 weeks. And that can be done in, in many different ways. So, so that is working time. I'm talking here. So, so you can either be very focused, very, uh, you know, and spending all of the resources at once. Or as I, in, in, in this case with Forza, we did stretch it out a bit. So we had time to, to conduct, uh, a lot of workshops. We had time to do the evaluation processes and so on. So it was a bit more than 15 weeks, but, but not in, in, uh, in working, working hours. Right. I hope that was, uh, um, answer enough. I don't see any other questions in the chat. So again, thank you. Thank you for listening and, uh, we will reach out to you. And you're also more than welcome to, to reach out to us if you have any questions. Uh, once again, I would just like to say thank you for joining and thank you for, uh, thank you to Forza for, for, for letting us do this case together with them. Thank you.