Guardrails for Greatness: Governing Content Quality at Scale

Welcome and Introduction

Steve: This next session is, I think, goes hand in hand with the conversation about content pipelines and content systems, which is about content governance.

Steve: So as we put the power into the hands of more and more creatives, more and more marketers, more and more salespeople, whatever your use case for your organization is, one of the first questions that comes up is how do you actually go ahead and keep that content on brand?

Steve: How do you keep it consistent?

Steve: How do you do that in heavily regulated industries?

Steve: I know a lot of folks are in industries where compliance is really important.

Steve: So to introduce that conversation, we have our Head of Content and comms, Esther Chung, to talk about guardrails for content.

Steve: So please welcome Esther and our esteemed panelists.

Steve: Here you go.

Esther Chung: Well, Steve set it up so well that I'm going to.

Esther Chung: Oh, you can.

Steve: Sorry, I signal.

Esther Chung: I didn't even realize you weren't this whole time.

Esther Chung: But yeah, as Steve said, I'm Esther Chung.

Esther Chung: I'm the Head of Content and Comms here at Jasper.

Esther Chung: I'm really excited about this conversation.

Esther Chung: Just like Steve said, this goes hand in hand.

Esther Chung: Everything we've been talking about, it's like we planned the theme very well, but this conversation is really going to be centered around one of the most fascinating and relatable challenges that a lot of marketers are facing today.

Esther Chung: We have a great diverse panel of different industries, from banking to pharma to enterprise tech.

Esther Chung: So it's really, we're really trying to answer that big question of how do you scale content with AI without sacrificing quality, without sacrificing consistency, without sacrificing trust with your customers?

Esther Chung: And then how do you do that with the added nuance of being in a very highly regulated industry, especially with a lot of these organizations that are managing massive content ecosystems?

Esther Chung: I'm sure Grid is going to be very helpful there too.

Esther Chung: But how are we able, how are these companies and how are these organizations dealing with the changing landscape of AI as well as the high risk of being in your industries?

Esther Chung: So please join me in welcoming our panelists from NetApp, Sanofi and US Bank, John, Madeline and Ken.

Esther Chung: And just a reminder, we'll have a Q and A at the end of this, so if you have any burning questions, just save them and we'll have the MIC be passed around.

Panelist Introductions

Esther Chung: First off, I want to start Madeline with you and we'll go down the list. What do you do at Sanofi? Tell us a little bit about what your focus is and just share a little bit about your role.

Madeleine Glomsten: So good morning. I know I had the pleasure to meeting some of you guys either earlier today or last night, but really happy to be here and share a little bit around the work we're doing at Sanofi.

Madeleine Glomsten: So my name is Madeleine Glomsten. I'm the head of digital marketing for Sanofi.

Madeleine Glomsten: So for those not familiar with the pharma industry, we are one of the big pharma. So in line with JSK or Johnson and Johnson or whatever else, we are headquartered in Paris.

Madeleine Glomsten: In my role, I'm really responsible for kind of like the incubator of AI within Sanofi and scaling the AI capabilities globally.

Madeleine Glomsten: And maybe before you all ask, yes, we did all coordinate our gray theme today, thankfully.

Esther Chung: But we're like a girl group.

Madeleine Glomsten: Yeah, it's like a girl group. Four shades of gray.

Esther Chung: All the conversations we had leading up to this.

Audience: On brand.

Madeleine Glomsten: Did someone say that?

Esther Chung: Yeah, on brand.

Madeleine Glomsten: Yeah, very much on brand. So I guess I hand it over to John.

John Dotto: Hi, John Dotto. I am at US bank and I sit in a group that's called Marketing Activation. We are basically an enablement function for the rest of the, you know, marketing activities that happen across U.S. bank.

John Dotto: So my team leads, I have a bunch of content strategists, authors, SEO geo activities, our local pages. So you know, I onboarded Jasper for US bank and that's sort of like centralized space in order to be able to roll it out and bring on board all the other content strategists across the marketing organization.

John Dotto: And we've actually expanded the program out. So Elevon is our merchant processing vendor. So we're onboarding them into Jasper as well as our core digital group as well. So again, enablement function and yes, I did get the memo on gray.

Ken Boney: Good morning everyone. My name is Ken Boney and I am the director for the marketing technology office at NetApp and NetApp is the intelligent data infrastructure company.

Ken Boney: We in marketing are responsible for enabling technology. We have 80 applications that we use in our tech stack. So we are the convergence of technology as well as business process and strategy and we try to make sure that everyone in marketing is enabled to do what they need to do in their day to day jobs and we've been a Jasper customer for almost two years now and it's been a great relationship and a great partnership and we're looking forward to some exciting things in the coming year.

Esther Chung: Great. I love how different each of your roles is why I think this panel is going to be the best panel. Just joking.

Defining Regulation in Different Industries

Esther Chung: But to set the stage for each of you. Obviously, regulated means something different in each of your industries. Can you talk a little bit about what regulated means for you, Madeline? And then we can go down.

Madeleine Glomsten: Yeah. So working within pharma, we are probably one of the planet's most regulated industries for all the good reasons. Right. We want to make sure that both healthcare professionals and patients have the correct information that they need to make really complicated and complex healthcare decisions. Right.

Madeleine Glomsten: These regulations are brand specific. So on very much the product level, as well as region and country level, as well as audience level. So a lot of complexities. And as a result of this pharma, Natya Sanofi, the industry overall has been very dependent on agencies to support some of these complexities. Right.

Madeleine Glomsten: And to be completely honest, there's really no way for one individual, or us as individual humans, Right. To keep track of all these rules across hundreds and thousands of brands, hundreds of markets and many, many regions and nuances that appear so by the fact of outsourcing that complexity. So agencies, for the last decade or so, we also outsource the capability of creating content and also understanding how to create compliance content.

Madeleine Glomsten: So I think with the emergence of AI and what we have experienced in the last two or so years is really that we can use AI to our advantage, not to increase risk, but actually to decrease risk. Because AI have this unique capability of being able to centralize all the rules, all the nuances, all the complexities that previously was solved by really very, very large agency teams that were supporting individual engagements.

Madeleine Glomsten: So we have really been using AI to not increase risk, but decrease risk, and then rethinking around how we're partnering with our regulatory and compliance partners in new ways of how we can create content. So it's not just creation faster, but actually going to market faster.

Esther Chung: Yeah. And I want to get back to that agency model. I'm going to go back to that topic a little bit later. But John, can you talk a little bit about regulated means for your industry, financial services?

Esther Chung: You obviously, everyone has bank accounts and it's important to have a level of confidence with the bank that you operate with.

John Dotto: And I was thinking a little bit more about this question earlier today. So we have a new CEO, she started this year and she has quarterly town halls in every single town hall that she comes on and she talks about the trust in the community and the trust for US Bank. Such an imperative that is driven from the top level down.

John Dotto: And when I think about Genai, there's certain ways legally that we have to talk to clients and to prospects, APY interest rates, et cetera. So that's sort of stuff that we can easily put into Genai platforms from a rules perspective. But then there's other stuff like around unconscious bias so that we're not like leaning into one race or one particular group when we're describing products. That's something. And we'll talk about this like human in the loop, why it's so important.

Ken Boney: For NetApp, data is our regulation, security is our big thing. So as we think about everybody here who has data and where our data resides and how that data's being used, I mean we have the highest level, I think of scrutiny probably right along with pharma. And for us it is definitely making sure, especially as it relates to AI, how is that data modeled? Where is that data going? Who's accessing that data? Who can access that data?

Ken Boney: And we want to make sure that as we think about it as a holistic standpoint from a marketing, especially as we deal with PII data, as we deal with, you know, what information is out there when we try to, well, when we, we launch campaigns and when we are doing email campaigns and when we have names and we're doing name capture on forms, how do we protect that information? So we're very, very highly regulated in our stance and in our governance of that data and of that information as it relates to AI.

Rethinking the Agency Model and AI Impact

Esther Chung: We're going to dig into that as well. I want to go back to what you were talking about, the agency model, because I think that's a really common thing that's changing and evolving with content marketing and marketing in general. Can you talk a little bit about how you've been rethinking the agency model?

Madeleine Glomsten: Yes. So just to give you a little bit of the background, I think it makes a lot more sense. So I joined NOFI just a year ago and when I spoke to what's now my manager, the ask was pretty clear. The ask was for me to come into the company and help them operationalize what we call internally it's digital pure play business model.

Madeleine Glomsten: In pharma, this means that the primary channel where we normally engage with both physicians and patients to some degree is through field reps, right? Think old fashioned field reps that walks into the hospital that makes sure that the physician have the information they need to treat their patients and so forth.

Madeleine Glomsten: So the ask was that we were going to eliminate this very expensive, the most expensive channel. The second ask was also that we do this in an agency free model. So this Ask was of course intimidating but also exciting. And I think why I love being in marketing. I think most of us that is in marketing, we went into this field because we like to be creative, we like to solve problems, we like to make sure that our customers get that amazing experience and feel excited and supported when they choose our brand or our products. Right?

Madeleine Glomsten: So as I accepted the challenge and we said set out to this journey, we eliminated the field pours and went completely agency free. So year over year, 2024 from 2025, we have cut 97% of our content creation costs. This is more than a quarter billion dollars in our world. Pharma spent a lot of money on agencies in the past, so a significant cost savings.

Madeleine Glomsten: We've also been able to accelerate our content production with 68% so significant time saving. But I think what's probably most important for me is also the work joy that we brought back to pharma marketing. For years and years in this old legacy kind of agency model, marketers, a lot of them was kind of project managers, was managing an agency, was managing a large agency team and a really significant budget. But they felt really disconnected between an idea they might have and if they the eventual outcome. That usually happened months and months after the fact.

Madeleine Glomsten: Now with all of this, we have seen my team's job satisfaction and joy go up exponentially. We probably the team internally, we have something called San Fe voice survey. We have one of the highest voice surveys within the company. These marketers are excited about coming to work. They're excited about feeling accountability and ownership to the that things they're delivering. And I think that's really what we have been trying to spread internally as we've been sharing our successes with the rest of the organization that it is possible that even though you might have operated within this older model for quite some time, there is an opportunity for you. Evolve, learn new skills as a marketer, take advantage of new technologies and actually find some work. Joy in doing things differently versus feeling intimidated, which I'm sure all of us in this room probably hear a lot about that every day from colleagues around. Is AI taking my job or am I going to like the new way that my job is maybe evolving? Right. So I think really showing that.

Madeleine Glomsten: So I've been using my team to kind of like roadshow around the company. I'm like, don't listen to me, talk to my team. How do they feel about their job today versus how they felt about it like 12, 15, 18 months ago?

Esther Chung: Are you adding, are you reducing or adding More headcount or is it obviously the introduction AI that's helping? Because this is content creation agencies. Right? These are people that are producing content for you.

Madeleine Glomsten: Yes. So interestingly enough, my team is supporting 31 pharma brands currently. That's a lot of scope and we have eight people. So we are a really small team supporting a lot of brands and a lot of content. So I think it's not just so much around only doing things faster, but doing things better and smarter, making sure that we're delivering quality outputs. Because of course we knew this was going to happen. Right. We're going agency free. We're going to get a lot of comparison. Like great. Madeline can create hundreds of pieces of content, but is it really good content?

Madeleine Glomsten: So we have done a lot of AB testing where we've actually had an agency develop an asset alongside us. Not just compare the cost but also the quality output and almost like putting it the blind tested internally in front of leadership saying this content or this content and every time they've selected our AI generated content because it's just better.

Esther Chung: Yeah, I think what you said is spot on because I think a lot of times when we think of scale, at least from a content marketing perspective, I'm like, oh my gosh, we have to do more and more and more. But it's not about that, it's about doing more better at a higher quality, more effectively for sure.

Madeleine Glomsten: Yes.

John Dotto: And I was going to say just to add, I mean we had similar experience at U.S. bank. You know, some of our content strategists, you know, been doing planning this year and into next year. They're pulling back on the agency spend for, you know, this mid upper funnel content briefs just because the output that they're seeing from Jasper is that good and that much quicker to market.

Governance, Trust, and Compliance in Content

Esther Chung: And actually related to that. I know you mentioned this in your intro as well. Trust is obviously the most important thing for financial brands. How are you ensuring that brand voice, regulatory accuracy is staying intact across the entire content lifecycle? As I'm sure you're dealing with a lot of steps of the process.

John Dotto: I mean the governance processes that we have set up haven't really changed. And when we onboarded Jasper or any Genai platform, Right. It's just additive to the process. So our content strategists are going in, it's actually helping them ideate the types of content that they are using and then obviously generating it in a very quick pace. But the product line reviews, the brand reviews, the compliance reviews and we still do Peer reviews, those activities still happen. So Jasper is just an additive component for us. It's not subtractive at all.

Esther Chung: Got it. Yeah, that makes sense. Ken, I know you have a very unique role because you on the panel because you sit at the intersection of marketing and technology. And I think this has been a burning question for the last couple years around who owns oversight, what the relationship? Is it like a frenemy relationship with it?

Ken Boney: I guess. How do you deal with that in such a highly regulated industry?

Esther Chung: We're being recorded, right?

Ken Boney: This is being sent to your IT team immediately.

Esther Chung: I think they have to approve some of this.

Ken Boney: Yeah, they will. But we have a very good relationship with our IT team. We have a very strong partnership. When we sought out to create the marketing technology office in this iteration, one of the things at the core of what we wanted to do is we did not want that stigma of shadow it. It's pretty big or was big in the previous administration, if you will. And when we structured the way that we perform today and the way that we go to market today within marketing we have different councils that we sit on. I sit on the Gen AI council and a couple of members of my team also situation on the Gen AI council as well and participate.

Ken Boney: But anything that we do from that end to end onboarding of technology, any martech that we do, we not necessarily have to get blessing through it, but we are definitely they sit in more of an informed role for us because they trust us, because we built that relationship with them. And that is one of the things that we really pride ourselves on as the marketing technology office is to build partnerships, build relationships because that's the people element for us in this day and time. So we have a great working relationship and a great trust factor there as well with them.

Esther Chung: I love the relationships. I think, I definitely think marketing, it's an underrated skill of like being able to build relationships across the organization. So can you tell us like how in practice you've built these relationships with it?

Ken Boney: Well, it's meeting with them and understanding that we also have a technical arm. So my team is made up or comprised of three different sections and one, we have our technical program managers on our team who go out and they do the contracting, they do the end to end onboarding of that life cycle management for the technology. Then we have the technical arm of our team who can speak the lingo for with IT as well. And then we have the AI arm which all of us marry it together and we kind of form a voltron of sorts with Only three arms. So with being able to speak to IT and meet them where they're at and speaking with the business and meeting them where they're at, we are kind of that bridge that bonds everything together from marketing and IT and even finance and even legal. And we speak with every single organization within the company and we're one of the very few teams at NetApp that do that and do it well.

Esther Chung: Do you find also Madeline, that Sanofi, that that's an important part of.

Madeleine Glomsten: Yes, I think the collaboration between marketing and digital and we also have like kind of like a marketing operations arm sitting separately is really crucial. Right. None of us is really going to accomplish this together. Like in our case digital. Digital a lot of times owns the software, in this case Jasper, but they're not the ultimate users of Jasper. Right. And understanding that they need marketers and they need marketers input what's working well and so forth. So we are the ones that just kind of like of course, maintain the relationship with Jasper because we're the hands on keyboards, not necessarily digital. But I think that collaboration is really powerful, especially as we're looking across the ecosystem broader than just content as well.

Esther Chung: John, did you have anything else to add?

John Dotto: No, I was just going to agree with Ken. I have pretty deep digital background. But having those people on your team that can challenge. I was going to say bullshit, but can challenge our task partners I think is helpful because it does create a good tension. Right. It's not to just completely accept what our technology partners are saying, but in partnership and challenge. And that happens both ways.

Esther Chung: Yeah, no, definitely we've heard from our other customers that that's probably one of the marketing. Being usually the first AI adopters in an organization, bringing everyone else along is really half the battle. And it sounds like it is, but I want to talk a little about some of the words we've been using is a little buzzwordy. Right. Governance or something. Responsible scale. So I wanted to hear a little bit how all three of you think about responsible scale in practice. Like are you. Do you have models in place, trainings?

Responsible Scale and Change Management

Esther Chung: Madeline, did you want to start?

Madeleine Glomsten: Yes, happy to start. And I think maybe pharma is a little bit unique in this sense. Right. Like governance or setting up like all these trainings was not necessarily a challenge to us because we are so heavily regulated. Right. Most marketers in this space is very familiar with that. We are under a lot of scrutiny. Right. So I think it's more. It was more around like because whether a piece of content is generated by an agency, a person or AI, it goes through the exact same regulatory step. We have a medical, a legal and a regulatory review for every piece of content. Right. So I think for me it was more around like how can we change the practice from marketers and get them to adopt AI as part of a new thought partner and a new partner of how to create content. And again and instilling that sense of ownership which many have not had for a long time because they have been relying on agencies. Because what we found is when the marketers feels a sense of accountability and ownership, they're also much more critical of the work. Right. They think about it differently and if something doesn't work, they also know how to fix it. Versus in the past they were very disconnected from that. Something didn't work, they had to go back to the agency that might be weeks or months to redo or fix something. Now they have Jasper or whatever tool they choose to use to address those issues. So for us I think the governance have always been in place because we have those processes already, but more around like making sure that the teams feel really comfortable and get to the point where they take accountability and ownership of the depth of content and bring that into how they use the tool.

Esther Chung: Got it. Is it pretty similar US bank, would you say? Or governance is already in place.

John Dotto: So from a training perspective, once we went through the pilot and onboarded and started to scale out, I established a sort of core working group because I mean we talked a little bit about how this technology and it just continues to scale and new capabilities are rolled out and there were about three of us that were who helped onboard Jasper and sort of like go through our pilot period. But after that I was like oh my gosh, all this other stuff that's going to be rolled out, it's just this was incremental work that this team was doing. So we, I created this sort of like core user group of about, I think it's like eight or nine or 10 people and we meet every three weeks and talk about different topics and I try to assign all I'm told folks within the group to take on some of this new activity, investigate things. We looked at translation services which Jasper was great. We ultimately decided we were going to continue doing what we were doing for whatever reason. But that was something that has been helpful also to sort of get the message out because I had representatives across the marketing organization. And then the other thing that we are doing is we're rolling out a community of practice to all of the users of Jasper. So we'll regroup on a quarterly basis, talking about what works, what doesn't work, and then just looking at some of the user reports on there. I want to go out and reward the people that are in there on a daily basis, our super users and sort of praise them within the system.

Esther Chung: Right, that makes sense.

Ken Boney: Yeah, yeah, yeah. I think very similar. One of the good things is we already had a good structure or good governance and place so that when AI came along we were already preparing and we actually, sorry, this is being recorded, beat it to the punch on a lot of the things that we were doing inside of marketing. We had like a very clear line of sight of where we wanted to go with AI and then also adding gen AI onto on top of that. So they actually adopted some, some of our practices and things that we had in place and they're still using them today. So we were excited about that and now we were invited to the table instead of being drugged to the table from an enterprise level. So yeah, we were very excited about that.

Governance in Optimization and A/B Testing

Esther Chung: I love that I want to switch topics because I know we talked a lot about optimization today in some of our presentations and that's a really, at least, you know, from a marketer's perspective. It's something that we don't talk about a lot, but it requires a ton of work and AI is obviously helping a lot with that. John, can you talk a little bit about how governance comes into play when you're doing AB testing, optimizing existing content? What does that look like?

John Dotto: Well, again, to sort of onboard Jasper we had a pretty rigorous process and the thank you to the folks at Jasper for helping me fill out quite a lot of that documentation. But as part of that pilot process we did about 15 ab or pre post tests. Some of the SEO stuff was pre post, but we were able to do some homepage banner ad testing, et cetera and it was great to see some of the results. I think at 70, 75% of human generated versus Jasper generated, Jasper was outperforming. So that was great and sort of did validation that we could circle back internally and get approval to roll this out. But what we're continuing to do is continue to do some of these A B tests. So I think one of the things that really drew me to Jasper's model was the differentiated LLMs on the back end. And you guys are sort of like looking on a regular cadence at which LLM is outperforming. I know you won't tell us what the secret sauce is, but I Think that's something that is appreciated but as a Czech and to be a responsible corporate citizen at U.S. bank, we need to do our own testing as well and see is it still materially holding up from an ad or whatever KPI that we have in place. So we continue to do that and latest rounds of tests. We just did some actually somebody else did it, so I was grateful for that. On our global homepage banner Jasper versus another provider versus human generated.

Esther Chung: Oh wow. So I want to go back to. You were saying that Jasper, you were testing Jasper content versus human generated content. Is that more of a baseline test or are you also testing Jasper versus Jasper? I guess. How are you thinking about that? Because I've been hearing, I actually just talked to Alex from LinkedIn about this, about how this has been more of a common ask from executive teams of what is actually outperforming each other and how are you thinking about that?

John Dotto: Yeah, I mean obviously it depends on the activity that you're doing. On a test like a global homepage, we're looking at KPIs then ultimately completed applications just to try and understand does this message work, does it not? So again it depends on the specific test, but on that global homepage one it was definitely click throughs and completed applications.

Esther Chung: Got it.

Globalization, Localization, and Content Quality

Esther Chung: Actually I do have a lot of questions up to me. So we're going to switch topics to globalization and we localization. I know Sanofi obviously operates in multiple different markets. How are you empowering your, how are you ensuring global brand alignment and then ensuring that you're empowering your local teams to also stay in line?

Madeleine Glomsten: Yeah, we get this question quite a bit and normally the way I try to answer this is global is not a place on earth. There's no global passport. Right. I happen to be lucky to have two passports, neither one of them are global. So I think the reality is that global needs to be more of a mindset. The reality is that any asset, any piece of content, if it's not actually deployed in market in country, is not reaching patients or healthcare providers in our case. So some years ago Sanofi had a much more of a global to local approach, meaning global teams pushed global contents down to markets. And I think what we learned is that local market either two things either happened. Either they spent a lot of time trying to localize said content, not just translating it, but changing up images, fixing formats, whatever deployment channels they were going to work in, or B they didn't do any of those things and just ignored it and did their own Content. So neither one of those was very productive, even though in theory it sounded great to have global content.

Madeleine Glomsten: So I think we are now switched approach and have a little bit more of what we call like a federated marketing concept where we global teams are working with a lead market for any kind of concept we develop. So whether that's the US or Italy or Germany or whatever it is, we develop the piece of content, the concept for that market so it's actually deployable. So like by the time we are done, it should be approved in market. And then of course markets that have similar type needs, we augment that piece of content. So instead of having maybe like a 30% of a content reuse rate, we're almost at 80%, meaning that we can create that piece of content and then our team can augment it for Italy, Germany, France, US whatever country might have a similar need. Right. So I think that's really then making sure that those local team can use the tool themselves to understand their specific, specific nuances. Right. Talked about earlier on with a lot of local rules, regulatory. In the US we have FDA that needs to approve content. In France, it's very different. They only do their approvals four times a year. There's a lot of those type of things that it's really difficult to keep track of unless you have those local teams truly empowered to support that content generation.

Esther Chung: Got it. And I guess part of that question, kind of a follow up is a lot of time we were talking about l' Oreal was talking about shitty content. Right. We want to avoid that. But when do you know it's good enough? What's your measurement for good enough? But I want to ask everyone, what's your measurement? Good enough. Quality content.

Madeleine Glomsten: Yeah. So starting at the very highest level, we are looking at treated patients. That's our ultimate goal. Right. Number of patients get treated with an appropriate drug that get appropriate treatment plan that I need. But as I'm thinking about content generation and we talked a little bit about AB testing, we actually don't do a lot of AB testing. It might be like a controversial topic here. Our primary channels we deploy in is digital. It's moving so fast, we can pretty easily put out a piece of content that we know pretty much within hours or max days if that content is resonating or not, and we can then pitch. So I think if a piece of content is compliant, factual and meet all the regular requirements, that's good enough for us to test it because the team also have the opportunity to pivot and come back and redo and deploy something that is slightly better.

Madeleine Glomsten: And I think the analogy was this. Like, I had the opportunity to go to Duke University a couple of weeks ago as part of a corporate training and I'm not necessarily normally a basketball fan, but being at Duke, we spent some time on the basketball court and had an opportunity to meet with the coaches and it was a really great experience. But I think what I learned about basketball, not being necessarily a sports fanatic myself, is that most games are won because of the rebound. The teams that have the ability and drive to catch those balls that doesn't necessarily hit the net and go at it again and again and augment and showing that ability. And that's really what I think where we are with marketing today, it's not necessarily get everything right every time, but having the ability of catching that rebound and shoot again and over and over again, that makes sense.

Esther Chung: I see you're nodding vigorously.

John Dotto: Well, I'm just thinking, I know we're going to talk about super users in one of the follow on questions, but just this use case in my head that really resonates to that question. So last year in California there was pretty devastating wildfires and we had an article that was maybe a year old or something like that on the site and we wanted to rework it and sort of like put it out there front and center. And we had a. I will call her Jess. She's a super user. She actually went into the platform and queried on the research, what is more information that's out there that we can sort of like bring into the article. And just that ability to sort of like pivot and flex in order to get best content I thought was just such a great example of how to leverage the platform to really bring, you know, critical information to our clients and just have it available on the site in pretty quick turnaround time.

Esther Chung: Yeah, yeah. And that makes sense. Pressure to move fast, especially in your industries.

John Dotto: Yeah.

Esther Chung: I'd be remiss if I didn't say that Ken's from North Carolina.

Ken Boney: I was kind of jealous because I live 30 minutes away from Duke and I've never been on the fort, like at all.

Madeleine Glomsten: I have some swag.

Ken Boney: I have some swag and maybe some sway. You know, I can get kind of, hey, Ken's coming to Duke today, but I think we have time for one more question, then we'll go to audience questions. But Ken, I'm going to start with you. So what's one thing you wish you knew before starting in terms of scaling AI budget Bigger budget relatable.

Ken Boney: Yes. I have to say that we're kind of like at this nexus to where we have established technology and we're trying to move it over to more AI forward AI first thinking technologies. So what we see is that investment is heavily into that established and a lot of time it's more bolt on, it's more how do I like shed this extra pounds or these extra pounds of technology and, and move into a place to where I can be more modern, more optimized, more lean in how I'm going to market and that's wish I wish I would have saved that money and listened to my parents and saved that money and now we're starting to come into this place to where we can reduce budgets in some areas while we start making bigger investments into our AI force technology. Stack makes sense, makes sense from the marketing tech.

Esther Chung: What about you John?

John Dotto: I guess again go with those super users. So as we piloted this it was interesting to see there was a lot of hesitation. This happened a year ago and there was a lot of hesitation from some of the content strategists who were just, this is icky. Is this going to take away my role, et cetera, et cetera. But we had a few users who just really dived into the platform and were curious about it. They seek to understood it. Understand it good English. They seeked to understand it content. Yeah, exactly. That would have been corrected. And as they figured out how the platform was working and were able to sort of scale, they were able to share it back and that sort of resonated and created this little wave of people coming along for the journey. So it really helped break down a lot of walls where people were initially hesitant about it. And I think in March of this year I had people sort of like knocking on my, you know, figurative door saying, hey, how do I get access to Jasper? Because we were just going through the final pilot approval phase and you know, just wanted to get on the platform right away.

Esther Chung: Wow. Amazing. Madeline.

Madeleine Glomsten: So I think choice, giving people choice I think is really important and I think this is where change management comes into play. Right. It's very easy to top down and ask teams to do something or use a particular tool, but if they don't have choice or say in that process, the change management is becoming pretty hard. So I think we were lucky enough that we thought about this actually. But as we were rolling out the pilot, I didn't just give my marketing team only Jasper. I gave them Jasper and several other tools. And I said we need to think about new way of doing things. And I want to make sure that you, you guys are shaping this like you guys are the marketers. You're here because you're excellent at what you do. This is just another tool in your toolbox. This is not somebody who's going to do this instead of you. You're still in the driver's seat. And I think that is so important. And of course, as all of us are scaling this to more and more teams across our organizations, that's become harder because it's more teams so forth. Like they get kind of like diluted. But I think in generally like start with humans, it's always very easy to get excited over a tool or a capability, right? Super excited. And digital gets really excited about incorporating it and integrating it and this and that. And then it's like, okay, but what about the people that we actually want to use this? So again, starting with those people that we expect to be hands on keyboard, making sure they feel vested, excited, showing them the art of the possible, being in the trenches with them. So important. I didn't ask my team to do any of this work that I was not willing to do myself. I think that is so important for them to feel like, okay, if Madeleine can do this, I can figure this out. We do this together, it's not going to be perfect, but we'll figure it out.