What's New In Data
A podcast by Striim (pronounced 'Stream') that covers the latest trends and news in data, cloud computing, data streaming, and analytics.
What's New In Data
Vector Search in the Aisles: How Morrisons Made Product Discovery Smarter
Peter Laflin, Chief Data Officer at Morrisons, shares how his team turned customer confusion into a cutting-edge vector search experience—bridging physical retail with AI-powered search. He and John Kutay dive into the practical challenges of implementing LLMs and real-time data pipelines at scale, the importance of starting with actual customer problems, and why the best engineering feels a little lazy (on purpose). A real-world look at what happens when modern search meets supermarket shelves.
What's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.
Welcome to what's New in Data. I'm your host, john Coutet. Today's guest is Peter Laughlin, chief Data Officer at Morrisons, one of the UK's largest grocery chains, serving millions of customers each week across nearly 500 stores. Under Peter's leadership, morrisons has been recognized for its innovative use of data and generative artificial intelligence to transform retail operations and drive customer loyalty. He does it while partnering with leading technologies like Google, cloud and Stream. Let's dive right in.
Speaker 2:Peter, how has real-time data improved your operations? Data improved your operations.
Speaker 3:We have a huge amount of data that we generate every second, every minute, every hour of the day, not least from the transactions that we have with our customers. So, whether it's in our store, whether it's online, whether it's through one of our partners like Just Eat or Deliveroo, we are collecting data points that tell us what people have bought, where they bought it and, crucially, how much money we have charged them. Now it's really important running a business that we can collate that data really quickly so that we can take decisions on that information, and so the first horizon really for the real-time data was to make sure that we had up-to-the-minute sales information, even slightly longer, to run batch processes to collect all that information from the tills, do the calculations, do all the aggregation and then present that back into a form that we can read and look at. But we have a number of pipelines in the business that are collecting those transactions together literally in real time. So, as we've been speaking in the last two minutes, I could go to our data set and see who's bought something within the last two minutes. What was it? Has there been an increase in a particular type of product? Are we seeing particular demand in a certain part of the country and, crucially, in a world of technology where we really care about making sure we have uptime, being able to use a stream of data to help sense the health of the network is also quite important Because, as we are processing all this information, we have a particular expectation about how much we'd see every minute, how much we might see every five minutes, ten minutes, so on and so forth.
Speaker 3:But being able to stream that data in real time means that actually we can pick up on on issues around us as soon as they happen, rather than having to wait for a few hours to realize that maybe there's a problem with a till somewhere or maybe there's a problem with one of our pipelines. So actually the real-time data starts to mean that we can make real-time sales decisions within a few hours. Do we need to think differently about promotion? Do we need to order more stock because the promotion is going even better than we thought it was? Other parts of the country, and certainly recently in the UK, we've had a number of big storm events where we did have to close a few shops for a few hours because the weather was so bad that the government advice was people had to stay home and not go out.
Speaker 3:In that context, we have that real-time information to understand just how our customers are interacting with us, to make sure that we can best service them, to make sure that we can best service them, and so that real-time data is super critical in helping keep our finger on the pulse of what's happening across our physical stores and online. Retail is about detail, but retail is also about what's happening now and what might happen in the next half an hour, the next hour, the next two hours, and so real-time data is super critical in that retail environment, because the numbers are so large that you don't want to get to the end of the day and feel like you were even two or 3% away from where you wanted to be, because that can be a material amount of money.
Speaker 2:Being able to be really close to understanding where we are at any point in time is super important in terms of how you can run the business absolutely, and the way you've deployed it is remarkable in how morrison's has optimized that for your internal operations and how you're able to respond to market changes and and even things like the weather and stock and things along those lines. And that's what's been incredible about partnering with your team, peter, which is just the obsession with customer success and making sure that your shoppers are having the best possible experience and the way. From Stream's perspective, you're using Stream Cloud multiple services and you have this vast, complex supply chain that we're pooling you know data literally from different physical locations and then massive volumes and ultimately applying that into these applications. I think what's made that so successful. Again, you know, yeah, the streaming infrastructure is great. The you know, the use of, you know vector databases is really innovative and it all comes back to the fact that you've applied it in a way that really helps the customers in the business.
Speaker 2:And you know I know there's a lot of data practitioners and data engineers listening to this, uh who are working on other amazing data pipelines. But just to to reaffirm just how important it is to make sure that the pipelines you're building are are feeding the the customers and the business and the people who are making those operational decisions, and that's that's really who you have to focus on, right, so I did. You know, morrison's is a very sophisticated operation with you know, thousands of farmers and suppliers and growers, and it's unique in how you're a grocery chain but you also control that kind of end-to-end supply chain for the food. I'd love to hear how you've built the data infrastructure to handle all you've built the data infrastructure to handle all.
Speaker 3:I think the key is to have a small number of patterns that you can replicate, but also recognizing that there's a huge amount of complexity that needs to be simplified. So the way I tend to think about it is that we have a super highway I think is probably what you call them in the US, isn't it? Or highway, at least we have these sort of big multi-lane roads that all go in the same direction and broadly they go quite fast. Whether they do in practice or not is probably a different point. Let's not get bogged down in traffic science, although there's a lot of really interesting science in traffic science but my point is you need a big pipe that is capable of transferring a lot of data incredibly quickly, and, and what you need is lots of processes that are catching the data. But, crucially, the data that you catch might be in a slightly different form to whether it's coming from a manufacturing site, whether it's coming from a supplier, whether it's coming from a store, and so one of the big areas of complexity that you could find is if we were to build individual pipelines for all of those, you would find you were doing a lot of bespoke work and you were doing a lot of work to just translate and map things doing a lot of bespoke work, and you were doing a lot of work to to just translate and map things. And the way that we've tried to build things is is to recognize that you need to do and I mean this in a really nice way, you need to do the least amount of work possible to give you the most amount of flexibility.
Speaker 3:Um across your, your sort of your data infrastructure, and therefore one of the things that um was always really high up our list of requirements was this idea of how do you catch the data and know where it needs to go, because actually I could catch the data, I could spend time thinking about where it's going and then I could send it on its way. But actually one of the things I like about where it's going and then I could send it on its way, but actually one of the things I like about where the technology is now going is that we can catch the data and we can worry about where it's going when it's on its way, because that cuts down the time it takes to do this process. It improves the opportunity for lower latency, but it also means I've got less work to do when I plug new data sets in, because if you have this sort of paradigm that says I've got a pipe that goes as fast as possible and it's incredibly intelligent so it can kind of figure out where things are going in the pipe and all I need to do is connect the data to the pipe and then make sure the pipe knows where it needs to ultimately get to at the other end. Now I hope that's not too too much of an abstract way of answering the question, but but really the the fundamental in the architecture has been we.
Speaker 3:If we built bespoke pipelines, every time we would need an army of data engineers and we would probably never get through the work. So what we've had to do is recognize that you have to build something that is flexible enough but intelligent enough to mean that our engineering team can be a bit lazy. Um, and they're not lazy. You know they're all incredibly hard working and there's always far too much to do. But you almost have to set out to make your teams feel like they are being lazy, because by doing that you're creating the right kind of behaviors in the sort of mindset for your pipelines and, like I say, I mean it's maybe not about getting into the whys and wherefores of how you can run AI on the data as it's streaming, but it is about that concept of ensuring that you can get things on its way and then eventually it'll figure out where it needs to go in an intelligent way, because that makes things very quick.
Speaker 2:Absolutely, and I love the way you phrase it because, honestly, the best engineers set themselves up so that they can be kind of lazy about to how Morrison's able to actually innovate so quickly, given the scale that you're running at, which is really incredible. And the other thing you said earlier in the episode was that retail is detail and you've run Morrison's now, which is an operation that runs at incredible scale, has, you know, full supply chain visibility, full shopper visibility, all the way from. You know the food being produced, so when it's purchased, and then you know being loyal to the customers and continue to provide a great experience for them. So what's your advice to other leaders in retail who are embarking on a similar journey of modernization and innovation with AI?
Speaker 3:Start with your customer and be very clear on why you are proposing what you're proposing. It's actually quite hard to get underneath the why? Question because, again, if you listen to your customers, they will. They will give you their feedback, they will give you a very clear steer in terms of how they feel, what they think is working well, what they think is not working so well. There's an art form, though, in being able to interpret that feedback and build a program of work that is able to talk to all of those um opportunities but do the right work in the right way, at the right pace to um to actually deliver against those those opportunities.
Speaker 3:Um, I mean, it's always the uh the henry ford example that comes to mind around. You know, if you ask people what they want, they wanted a faster horse back in the uh, the early, early 20th century. What they actually needed was a car, and so you do have to make sure that you're you're not over innovating, you're not over complicating um. You know our, our leadership team in morrison's, often talks about complexity and simplicity. We have a very complex business, but our job is to deliver that as simply as possible, and so be really quite clear on the problem you're trying to solve, because what fascinates me is, in a room full of technologists and a room full of incredibly clever data engineers, data analysis experts, data scientists even with the same question, even with the same observations from our customers, the solutions can be very wildly different. And therefore, keep asking why until you get to the point where you can absolutely crystallize I'm going to do this, we're going to do that because it's going to deliver this for our customers, and I think you've got to build your strategy around that.
Speaker 3:It might be that real-time data is super important for you and your customers. It might not be. It depends on your product, the way you set your business up, your customer value proposition and, equally, the use of ai to automate various things. There might be obvious things that you'd like to do because your competitors have done it, but challenge yourself to say why do we do that? Because we want to, or are we copying?
Speaker 3:And and I think, asking that really detailed why question can be really quite hard and, at times, can feel quite uh. It feels like you might slow yourself down, but in a world where you need to go fast, it actually speeds you up because at the point that you then start to deliver something. You're super clear on why and when it comes to then building the requirements and talking to the vendors and connecting all the technologies together, you can be really very focused on what you need. Now I think you know, if you look at the, the way that we sort of started to work with stream, I think we were very clear on what we needed and we were very clear that we needed a capability that you were able to provide.
Speaker 2:And I think you know, when you're setting these things up and working about where you go in the future and I, I think, being super focused on why, I'm super curious on why you think that that is the most important part, I think yeah, and that's that's one of the things that has been really incredible about partnering with with morrison's and you know, uh, you know working with you on your, on your uh implementation and rollout, because even when we're looking at streaming real-time pipelines, we benefit from knowing the business use case. Why? Because we're going to tune the pipelines accordingly. And you might say someone might say, hey, our use case is just having some reports that load nightly. And we're using stream because we want to do low impact change data capture so we're not pulling the databases and making that an expensive operation against the production databases and that's fine. So we'll do the real time change data capture but we'll batch and load the data into your warehouse, you know, every 24 hours, so you're saving the cost there. But when customers say you know we want to improve the customer experience by responding to real time signals, that's when we really say okay, then let's, let's go full throttle and help you optimize those pipelines. And you know it's been a great partnership and great teamwork. You know from from the, the, the folks here at stream that were working with Morrisons and then the counterparts at Morrisons who are really dedicated to building out this infrastructure. But I think it really does all come back to your vision and your leadership for what you wanted to accomplish, which made it easier and more focused for Stream and I'm sure Google Cloud as well to deliver what you're looking for, because, ultimately, when people know what they want, people will rally around them and help them get there. So that can apply to all data teams the more the data team knows about the business initiative and what they're really trying to accomplish and really who their customers Because all data teams, whether they know it or not, they have a customer.
Speaker 2:It might be an internal customer, but there's someone that's you know paying with their time and dedication and you know their precious resources to work with the data team in one way or another. And data teams just have to know who that customer is and why they're delivering. It could be like an end customer that's actually buying a product from the business, or it could be the finance team right, they could be doing revenue recognition with that data or it could be someone in the sales team is doing lead routing with it. But ultimately it all comes back to, peter, like you mentioned, just really understanding the business problems and how you're solving them. So that's what's been really fun about working with Morrison's, and now you're seeing all these incredible benefits and the customers are happy and the financial metrics all look great, and I think that's certainly no coincidence. And, peter, I also wanted to ask you there's so much changing in the industry as well. Just with all the innovation that's going on, what emerging trends in data and AI are you most excited about?
Speaker 3:I'm almost exhausted, to the point where it's hard to get excited and I'm joking, actually. I think there's so much going on, um that it's hard not to be excited, um, I. I think that the challenge is how do you deliver on the opportunity? Because as every week goes by, I think the opportunities get bigger and I think the hype gets bigger. If you go back 10, 15 years, there'd be new technologies that would emerge and you'd have to work really hard to go to your exec and say, on full circle, we now have board members asking for ai and asking for new developments, because these developments are now mainstream to the point where I don't think there's anybody who hasn't heard, either directly or indirectly, of a large language model um or a chat gpt, um, you know so. So I think the the really the really exciting bit, but actually the big risk, is how do you realize that the opportunity? Because it's clear now the opportunities are massive and the expectations of our stakeholders are that we will get our fair share of those opportunities. But how do you do it safely? How do you do it in a world where people are learning to walk? So you know, we've gone through various paradigms, haven't we?
Speaker 3:You know, if you look at the, the mid to late 90s, it was about how you enable computing in a business. And then the world of the you know the, the it sector as we know it then sort of grew around that, um, you know, data came probably 10 years later and now we have generative ai and sort of broader ai systems that are coming through and so we're going through probably that, you know, maybe second in my mind, third revolution you know these technologies are now as fundamental a change as it was to start to use a computer in your day job. I remember one of the things that inspired me to get into this world was um. One of the things that inspired me, um, to get into this world was um. I remember my. My grandfather was, uh, involved in in putting computers into schools in the mid 80s and he was massively inspired by the bit, but he retired at a point where he never got to use a computer at work and I do think the next generation that comes through. So if I look at my children, um, they're probably, you know, the first generation that will go into the world of work with an expectation that they do use AI routinely in what they do, and so this has been a huge cultural societal expectation of shift.
Speaker 3:It's massively exciting, but it is incredibly scary, because how do you, how do you, keep control of something which, inherently, is uncontrollable if you want to go after the value? And when I say it's uncontrollable, people are going to find their own ways to use this. So even if we put various regulatory frameworks in place in the business to say, please do this, but not this, I'm sure people will find ways around it, whether accidentally or otherwise, and therefore actually the governance of this is the thing that I'm probably most concerned about. But if we get the governance right, I think it is so exciting that things that took days, weeks, months to do will be done in seconds. And what does that do for our understanding as a, as a society, as a group of humans looking for ways to continue to live sustainably and effectively on a planet?
Speaker 3:Um, and therefore I, I, you know, I think how big do you go here? But um, ultimately, I think this is just another stepping stone into increasing our collective intelligence in such a way that we can then start to tackle all those questions that we still haven't solved, such as where do numbers come from and how do we tackle global warming and how do we cure cancer forever, and actually there's so many really valuable and exciting things that we could do with the technology. Um, but actually right now we just have to figure out how to use it effectively in a business to to mean that we don't break anything. Uh, because if there's a big bang where something goes wrong with generative ai, it might stop that. That development of technology, uh, in a way that probably would be would be bad for society but probably better for everybody in the short term. But I'm rambling now slightly.
Speaker 2:I mean, yeah, those are all you know. Yeah, like you said, the next generation is going to use AI. That's going to be their expectation. That's how they're going to work with everything. They're just going to chat with some robot, you know, some robot-like thing that just understands what they're trying to say, can derive the semantic meaning from anything.
Speaker 2:And you know, I think you put that, you phrased that very eloquently, peter, and you know that's the world we have to plan for now and it is sort of a race.
Speaker 2:And because you know companies are trying to outpace their competitors into who can roll out the best AI enabled service.
Speaker 2:Because you know, chat GPT already has tens of millions of users, right, so people are already just chatting with things and, uh, you know, getting getting the responses they want, uh, so it's such an exciting time. And that, tying this back to what you've already delivered, right, ai driven search, right, it really sounds like you know, you, you've kind of understood, so, the essence of the value of ai, which is kind of this, this, this, this buzziness, this little, you know, ambiguity of the way people describe things and knowing what they mean and giving them what they want based on that. So it's it's, it's really cool to see what you've already delivered there, with the power of getting the real-time inventory data doing the vector embeddings, surfacing that as an AI-driven customer experience. I just think that's so exciting. I mean, that's really what interests me. You can build all the AI infrastructure you want, but really what's most impressive is how teams like yours have actually rolled it out for the customer.
Speaker 3:You need to be left-brained and right-brained at the same time, because we're moving now to a world where the technology gives you a platform to be creative. And actually the exciting thing is how do you use this commoditized capability? Because it'll all become it'll all effectively become commoditized. Everybody's ai out of the box will probably be about the same as everybody else's, because, even if it's not, everyone will catch up. So let's just assume you know the base ai becomes kind of you know broadly a constant factor.
Speaker 3:How do you then build that into other processes in such that you are creating something new and valuable and that is a creative process. And therefore we um, we're shifting um from being a knowledge-based tech community to being a curiosity-based tech community, and actually the shift is going to be knowledge can be replaced or found in a large language model. The insight and the spark to think of something new, to apply it to, will become the valuable resource in the future. But it's all underpinned by the data and the pipelines. So I mean it's fascinating where it's going to go. I would love to have this conversation again in five years, john, to see where we've actually got to um. But you know, super exciting, bit scary um, but the journey is what's fun uh, yeah, it's, it is so much fun.
Speaker 2:I mean it is. You know, it does remind me when, you know, uh, mobile apps on the iphone and the app store kind of blew up and there's just this, this kind of blank slate of innovation that we know is going to happen. And you know, there was sort of a similar cadence with data, but now with AI, it's just so clear that there's going to be just a new generation of applications and use cases and ways this can be applied. Well, you know, peter, you know, at Stream, we've been so delighted to work with you on this five-year journey with Morrison's. I want to ask you about this a little selfishly, but what is it about Stream and our people that really made the difference for you?
Speaker 3:You worked with us really collaboratively and you co-created the solution with us. So I talk a lot about the difference between collaboration and co-creation with us. So talk a lot about the difference between collaboration and co-creation. Um, co-creation is the good one and what I'm looking for, um, which is where we kind of all jump in. We're one team, we're clear on the goal and we work together.
Speaker 3:I think, um, we often joked in in the project meetings that it was difficult to tell um the stream team from the the morrison's team, because it you just worked with us really really well. Everybody just knew what we were trying to do. It was hard yards, it was difficult stuff. I mean, I think it it can be quite challenging to sit back from this and go. We did a, we did a really big thing, um, and we did it really well, and the reason we did it really well is because you guys support us brilliantly and you know, I think, think the real difference was you have technology, but you have some fantastic people and working with you was easy, and so there were lots of real positives there, because it's really important that you can actually embed the opportunity, but you can only do that with the people, so your people are great. So yeah, thank you.
Speaker 2:Yeah, thank you to you as well, Peter. It's always great catching up with you. I hope to see you very soon next time I'm in the UK or vice versa. If you visit us here in Silicon Valley, you're always welcome to hang out at the office here. Peter Laughlin, Chief Data Officer at Morrison's. Thank you so much for joining today's episode of what's New in Data.
Speaker 3:Thanks, it's been a pleasure. Thank you, Bye-bye.