Leveraging Expertise for Creative Problem Solving

Sometimes addressing a structural issue takes some out-of-the-box thinking. Northeastern University, under the guidance of Jessica Liebowitz, Faculty Director of  the Student-Administrator Partnerships Initiative, brought together technically-accomplished students to serve as Fellows to assist administrators in addressing a serious or intractable institutional problem. In this episode we hear about one project at Northeastern tackled by two Fellows and the Registrar’s Office staff focusing on student demand for classes, bottlenecks in the curricular pathways, and ways these data might be used to improve the student experience.       

Key Takeaways:

• Each of us in higher education has the opportunity and the responsibility to work toward improving the student experience. Sometimes that will require being creative about ways to bring together different individuals to think differently about a problem, and to carve out time to work on solutions. 
• The Student-Administrator Partnerships Initiative at Northeastern is a great example of one way to leverage expertise on your campus. Giving capable grad students access, guidance, and resources can open a world of possibilities. 
• With any project, be open to where the project might take you. Sometimes what you learn along the way is more important than what you originally set out to accomplish.  

Host:

Doug McKenna, Ph.D.
University Registrar
George Mason University
cmckenn@gmu.edu   

Guests:

Jessica Liebowitz
Research Professor, Center for the Future of Higher Education and Work
Faculty Director, Student-Administrator Partnerships Initiative
Northeastern University
j.liebowitz@northeastern.edu

Ingrid Nuttall
(At time of recording): Deputy Registrar of Solution Delivery and Strategic Reporting, Northeastern University
(Current): Associate Vice Provost for Enrollment and University Registrar
University of Virginia
ingridn@virginia.edu 

Satya Pulipelli
Graduate Student
Northeastern University

References and Additional Information:

Service Excellence

Leadership

Enabling Academic Innovation

Project Management

Transcript

Copy

0:00

You're listening to For the Record, a registrar podcast sponsored by Acro. Hi, I'm Ingrid Nuttall. I'm the deputy registrar of Solution Delivery and strategic reporting at Northeastern University. Hi, my name is Jessica Leibowitz. I am research professor at the Center for the Future of Higher Education and Work at Northeastern University and faculty director of the Student Administrative Partnerships Initiative. Hi, I'm Satya, and I'm a graduate student at the College of Social Sciences and Humanities, Northeastern University. And this is leveraging expertise for creative problem solving. Hello. Welcome to For the Record. I'm your host, Doug McKenna, University registrar at George Mason University in Fairfax, Virginia. Registrars are pretty amazing, and it's said that we solve problems people didn't know they had before they knew they had them. And that is sometimes true. Other times we address structural issues that may be known but not well understood and find ways to improve related outcomes. That's the bucket we'll be focusing on in today's episode. I'm very excited to have a contingent from Northeastern University join us to talk about the ways that they have worked to identify bottleneck courses and how they've raised awareness of the interconnectedness of their curriculum. Ingrid, Jessica, Satya, welcome to the podcast. Thanks for being here. Thank you, Doug. Thank you for inviting us. Yup. Thank you for having us here. So there's a lot to unpack here today and I'm looking forward to getting into the thick of this conversation, but we should start with some introductions. So, if you would tell us who you are, what your role in this project has been, and then a little bit of About Northeastern University, I understand it's a small residential college just on the outskirts of Boston. That is, yes, Doug, thank you. I mean, you really took care of the introduction for us. I can handle that one. my name is Inger Nuttall. I'm the deputy registrar of Solution Delivery and strategic reporting at Northeastern University. Um, in the office of the registrar, I lead a fabulous team of folks working to remove barriers to student success through system implementation, system optimization, and Um, making data more transparent to people. I, uh, can give this overview of Northeastern University, as Doug said, it's a small residential university in Boston, um, started as an evening school at the YMCA, true story, um, but has since grown a little bit over the years, um, since the, the, its advent in the late 19th century. We have um over 20,000 undergraduate students and over 20,000 graduate students across 10 colleges. We are a global university. Uh, so we have 13 campuses across three countries and that is one of the things that makes us Really special and unique is we operate a highly mobile student experience across all of these different places and all these different regulatory contexts, which is sort of part of the fun, part of the, the registrar's office and the work that Satya um and Heather, who couldn't be with us, were involved in. And we are really rooted in experiential education. So it's integrated throughout the curriculum. We offer co-ops, um, to students, internships, internships, just learning kind of in the real world across the board, and that's one of the things that makes us super special. Thanks Jurgen. Jessica. It's great to be here. Thank you so much for inviting us. This is a very exciting example of a project that was part of the student administrative partnerships that I helped to start this year. So, I'm a research professor at the Center for the Future of Higher Education and Work at Northeastern. And, uh, specifically with regard to this program, I'm the faculty director of the Student Administrative Partnerships Initiative which just started this past fall. And in a nutshell, the idea is to bring together very accomplished, technically accomplished students who are selected for their technical expertise, but also for their capacity for rigor, creativity, and being able to navigate through unexpected, ambiguous, and complex situations. We bring them together with administrators who are very close to an intractable problem that's important in operations. Um, and what we're doing is we're bringing together the natural talent of the student base that we have, with the natural talent of the operational wisdom that middle-level managers have when it comes to universities being able to achieve their missions. Right on. Thank you. Satya, what's your role? Hi. Uh, again, thank you for having us here. And, um, I'm Satya. I'm a graduate student at Northeastern, uh, in Economics and Data Science at the College of Social Sciences and Humanities. I was one of the fellows on one of the projects of the Student Administrative Partnership, and I was working with, uh, the registrar's office, trying to identify enrollment bottlenecks, uh, And barriers to student success and helping the registrar's office move the university from a supply-based registration model to a demand-driven registration model. That's awesome. This is uh an incredible project and I'm excited to talk about it and learn more about it. So maybe Ingrid or Jessica, you introduced sort of the higher level kind of this is the systemic thing that we're trying to do, maybe drill down one step to say like how was this project selected, how did this come about? What was the impetus for um looking at the bottleneck courses. I can jump in a little bit there. So, as Satya mentioned, so this was one of a few different projects that were funded through this initiative that Jessica started. And there's another one that was happening that was really focused on course scheduling, like, how can we make sure we're offering the right courses for what students want as opposed to kind of the model of we're rolling over the sky. Schedule from year to year, everybody takes a look at it, like how can we provide more data. So, there was a, a complimentary project going on for one of the colleges that I want to note because I do think that that goes hand in glove with what we were looking at. So, they're doing that work. And when we had this opportunity to participate, um, I, I wanted to kind of pull back a level and start thinking about the earliest stages of how curriculum is just like conceived of at the institution. So we get questions kind of all the time for send me a list of Um, students in this program, I'm trying to find and what courses they've taken, I'm trying to find out acts in order to figure out what I should do with my curriculum, just kind of ambiguous statements like that. Sort of walking back from that and saying, what if there were a way to Have some kind of like a health check, and I think I came up with a different word for it satya. I'm trying to remember, but what if you could like have an idea and basically put it through a health check at the beginning with a little bit of information, yeah, diagnostic, exactly, so that you would know, OK, this idea that I have, um, is 50%, it, it like. It, it gets a score that shows that it's gonna really not work for students. Like it's gonna have all these barriers to student success. So it was sort of like a, a diagnostic for ideas at that beginning stage. And then it evolved from there, which was really the cool part of the project as um Satya and Heather Karamati, who isn't here because she has um a full-time gig, which is wonderful for her, but she and, and Satya, both of them together were a really important part of how this worked. Um, so, as they came together though, they really started looking at the data, which was an important part of the project to sort of flesh out that really high level idea of this diagnostic tool. So, that's how it came to be, um, in compliment. To that other project too that was happening with the fellowship. Cool. Well, let's be specific about what data we were looking at here, um, so that we're, we're not speaking in platitudes but in, in actualities. What, what was the project? Where were the data sources and how were they pulled together? I guess Satya, that's sort of a, it's a good spot for you to jump in. Yeah, uh, so, the, we were looking at 3 different data sources, uh, to begin with, we were looking at student enrollment history. And we were looking at program requirements, uh, of each program, and we were also trying to look at enrollment history of different courses over the last 3 years at Northeastern at all the different campuses. And the goal was to combine all these 3 data sources to get uh Connected holistic view of everything that was happening. In how students were navigating through each program. In real life rather than how it's written on paper. Yeah, and that's, those are common data sources in registrars' offices that actually so common that people might overlook the fact that they don't actually talk to each other in meaningful ways very often. And so what was that like trying to make sense of those three different sort of buckets. Well, that, that was a big chunk of what, uh, we did over the last one year, uh. At the very beginning, we realized that these 3 data sources were designed to serve their own administrative purposes, and they were never built to be interconnected with each other or talk to each other. So why would they be? Yeah, I mean, it was never done before, so why would they be, I guess. So the The majority of our time was dedicated to kind of standardize the data in a format that would help us connect all these three data sources so that the output or the outcome of data, whatever we're getting, makes sense and has analytical value that is useful for the admins who are using it. And I wanna jump in just on something that you said, Doug, because I do agree, it's like, why would these things be, why would everything be built to talk to each other? And I feel like sometimes the Inclination is I want like one ring to rule them all, like, give me a system that brings in all of that data and makes meaning for me. And what I thought was really illuminating about this project was that, that doesn't actually have to happen. Like, you can have purpose-built tools um if you can get at the data and have like a platform, and infrastructure and investment to Make meaning of it and tell stories with it, and that is really where, like, that is my passion for the registrar's office and for higher ed. It isn't, how can we buy the next like monolith system that brings together all of these things. Degree progress data, like student degree audit data is so incredibly powerful and important, and it is so hard to get access to it. So if I have a plea. For like all vendors and the universe and everything, it is, it isn't that you have to build additional functionality into these systems. It is, let's figure out ways to get at the data so that we can bring it together with amazing people like uh Heather and Satya to tell those stories. Yeah. What was the impetus, Ingrid, you talked a little bit about, you know, wanting to craft these stories. When this project was initiated, who was involved in scoping it? How did that, like what was the, the plan for the work to be done? Who was responsible for identifying sort of the metrics that were gonna be used, and then what was the end goal? What was the point of Of what you were trying to do? Was it to create just a narrative? Was it to increase information, or was it to create sort of, sort of a functional tool ongoing that then individual users across campus can engage with? I will start with part of that story and then I'm actually going to kick it over to Satya because it really is, we both, we had different roles, but they go together. I would say that as opposed to a traditional project where it's, you know, you're setting your scope, you're setting your objectives, you're doing your timeline, you're doing that. I mean, we knew how long we had to work together, but as Jessica has stated, and part of the goal of this was not to be prescriptive. In the scoping of the opportunity for the fellows, it was to say, we've got a problem, and here are some ways that I, Ingrid, think, here's some, here's some reasons why I think that problem might be happening. And here's some stuff I think you could look at. And here is what I would like to see at the end, like, here's an idea of the tool, as you say, or like the resource. And so, I scoped out that. I scoped out the problem and the opportunity and I put something specific in there just about this idea of a diagnostic tool. So that there was a thing. But that was pretty much it and the focus was intended to be narrow and, well, I suppose like big and narrow at the same time because it's a creative, this is like a creative opportunity. It's not just a data opportunity. So, I knew the data sources that they were probably going to need access to and I want to. Also say I knew that it was not going to be easy to get at that information. So, that was also part of the journey. The other thing I want to note though, is that at the beginning, this was the fellowship was supposed to bring one person, one fellow to the office. And I did the interviews. And I interviewed Satya and also Heather Carmani, and they, the questions that I was asking in those interviews were really about storytelling. Tell me about how you have told stories with data because that really was the central what of the scope was, let's surface data to tell stories about curriculum. And Heather. was very, had a lot of skills in front-end development, and she had done some work on her own, building a, I'll have to see if I can find it, but she built essentially um, an interface that talked about like bread and toast and when does, it was like something that was in the New York Times of like, What it's almost like that is a hot dog, a sandwich, or a hot dog thing, except with just with bread. So she like showed me, she talked to me about that and showed me this like cool thing she had done on her own. And then I met with Satya. And when I asked him that question, he was like, well, I've done some stuff on my own because I think that Apple gets a bad rap or iPhones get a bad. Rap. And so he had like, put together all of this data in order to show, like, I think why like Apple is good. And so they had both done this work, and I, yeah, you can talk about it yourself, but they had both done this work on their own to, like, play around, except Heather was really focused on the front end. And Satya was really focused on the data analysis and like the, like data research pieces. And so, I asked, so I will say the other thing I brought was I asked for the opportunity and the approval to hire two people, because I do believe very firmly in never working alone, ever. And so, I was hoping that part of the experience for them would also be to, like, get to be creative together. And fortunately, Jessica and Rebecca, um, approved the ability to do that. So, that's the scope and the story. And then, um, Satya, maybe you want to talk about how you, how you took that wonderful ambiguous opportunity and turned it into something magical. Well, yes, uh, I'll start by agreeing with you that in retrospect, having me and Heather on this project was definitely one of the reasons why it moved and what it became the way it did. But once we received the brief of trying to come up with a diagnostic score, For programs and trying to find student barriers, Heather and I worked together to figure out what data points or data sources we needed to kind of get an idea of what to even look at and how to understand the problem space. And in doing so, we also realized how, how many barriers there are to getting access to these data sources in higher air. Uh, that was one of the initial challenges that we faced. But once we were past that and once we had access to the data sources, we, we realized that in order to be able to come up with a diagnostic score for any program, we need to connect all these data sources which were not built to connect to each other. And one thing I will say about the scope of the project is It was evolving like any other typical project in the real world would because we went from coming up with a diagnostic score for programs to putting all courses in a pre-reg network graph and to analyzing enrollment trends and figuring out cross-program share for courses, and then, Putting it all into one single analytical tool that admins can use to figure out capacity planning, course offerings, and anything that is related to registration and program planning. So, what started out as a single metric of diagnostic score ended up being a functional tool for admins to use for uh capacity course and program planning. Let's talk specifically about the diagnostic tool. What data was used, how was it used, and then how did you create the score? What does that diagnostic score tell you about the program? Funny thing is, the diagnostic score never was never realized. It just evolved into a bigger tool that had multiple aspects to it, uh, and I'll talk about the tool itself. So, it's a dashboard that has 4 different tabs. Uh, the first tab is where admins can look at courses and programs in a network graph, where each, each node is a course itself in the program, and all the courses are connected by lines that are prerequisites for that course. Let's say course X is a prereq for course Y, they're connected with the line, which represents the prerequisite, and this enabled us to look at the program. And from a bird's eye view and see how courses connect and really figure out what the structural, structurally integral points are in a program, because one of the assumptions that we started out with was that we might find that all the intro courses or the introductory level courses might be the barriers that our students that are delaying student graduations. But it turns out once we looked at the network graph, the intro courses were not really the bottlenecks. The bigger bottlenecks were courses that were sitting in the integral points in a program where that was the only path forward for the student to access any upper level courses that they might need. And that was one of the, I would say, highlights of the initial phase of what we were trying to do. So that's the first step. The second tab is a program metrics tab where you can see different metrics of a particular program, including cross. Department share of courses, meaning what percentage of your courses are also shared by other programs. So you know how much dependencies you have or how much interconnected your program is.-- And when-- you say, I'm sorry, when you say are also used by other programs, do you mean required by other programs or just options within other programs? Was there a differentiation there? There's a toggle that Admins can use to figure out like to separate out required versus all courses in program. Yeah. Similarly, we also have a course metrics tab that is specific to a course. It's similar to the program metrics, but it's specific to a course, and you can see all the programs that require that course, what was the. Average enrollment trend of that course, how frequently was it waitlisted in the last 3 years? What was the average waitlist count, and very specific course metrics. The 4th tab, one of the things that we're very proud of, is it's, uh, we call it the redundancy analysis tab, but it's uh a tab that lets admins look at possible substitutes for a bottleneck course. Now, how did we get to the substitutes? We analyzed all course descriptions at Northeastern using semantic analysis and figured out, OK, this course and this course are pretty similar, and there could be substitutes. It might have a lower number of pre-regs that would make it an easier alternative, or it might have an equal number of alternatives, which could be fine as long as it's not a bottleneck. So we did not just stop at identifying bottlenecks. We identified the bottlenecks and then we presented the. With a solution that, OK, this is a bottleneck, but you can use this as a substitute for that bottling course. Yeah, here are potential options for opening those up. That's awesome. Yeah, so you say you used semantic analysis, I assume that you took course descriptions and fed them through something. Mhm. Uh, we used, uh, cloud API, PDF Plumber, and, uh, Rsex to come sort of, uh, understand the similarity between course descriptions. Right on, that's pretty great. I've had Zach Pardos from UC Berkeley on the podcast previously about some of the work that his lab is doing at UC Berkeley, specifically on course descriptions and mapping out sort of where the commonalities might be and, and so this is. It's a really interesting approach for understanding an institution's curriculum, and I think that this is one of those uses of AI that I am. Uh, supportive of, I'm very skeptical of AI overall, but I think that if there are places where we can feed a bunch of data and get meaningful outputs that we can then use that are actionable to say these might be other options, and I'm all for that. So this is a, an interesting approach. So those 4 tabs on the spreadsheet are now available to who at Northeastern? Well, I would like to say it's not a spreadsheet. Oh, I'm sorry,-- a dash-- dashboard. My bad, my, my bad. It's a, and I wish that I understand that a podcast is not a visual medium, but it's a really, it is like tells a visual story without you having to understand it. So just to kind of point out that no. You know, one of those graphs, you put it, you can put in a course and you can just see the relationship mapped immediately without having to even dive into any of those next details. So, and I can speak to the kind of department piece, Doug, which is I think where you like, how are people seeing this, how are people using it? So one thing though, before you have on the show notes page, we could include a screenshot or something if people want to take a look at what. It actually looks like if you want to send me something after like screenshots from that are probably all that there are, but um, so while, while Heather and Sachi were building this, I was still getting those quite like I got an email from a college saying, hey, I want to know, we're going to design some curriculum of course you are. I wanna put, I wanna know what courses are pre-req in what program before I do anything, and I was like, funny that you're writing me now because we have a thing. So, the, the one thing that we were able to do while it was still being built before the end of the fellowship was I put a call out and organized an open forum for colleges to kind of voluntarily attend. Reached out to people that had reached out to me, but I put it in our office's newsletter and was like, hey, if you want to come to this thing, let me know. So, we organized a demo and that was useful in a couple of ways. One, it generated more kind of questions and feedback and just contextualize how people might use it, so that was valuable for Heather and Satya to work with as they were sort of finishing the project. But also we, we did schedule time to show it to leadership in the provost's office, folks that work with curriculum to be like, hey, this is a, this is a thing, and wouldn't you like to, wouldn't you individually like to look at this thing as you are working with faculty on and colleges on curriculum. So, that was really the only way that it was made present and shown to people in the colleges. This has not been a completely funded operationalized enterprise project yet, which means that nobody's using it. And I think it is good, I want to say that because there are conversations underway trying to make connections with our central IT shop who has been fabulous in partnering with us on, How we can take kind of the next steps with the data analysis part and use university infrastructure more effectively for it. So, we are trying to make this an enterprise solution, but right now, it isn't. And I think that is one of the things I think for enrollment professionals, for folks working in kind of these types of projects, these almost like startup initiatives. It's like, Yes, you get the excitement and you get the energy and you get that wisdom and you get the talent and you get the time of all of these people and you know you have a thing, and the thing is not done yet. Like, I think Satya would say, like you had some goals for wanting to do work with this that was even more student facing that you didn't get to achieve, right? Um, but it takes, it isn't just a one and done thing, it takes commitment and investment. And so, I think it is worth balancing, like at the front end. Knowing that you're going to put your time and effort into something that is going to have legs when it's over. It's a challenge, um, but I think it, it's, it's worth calling out as an opportunity for all of our institutions to, it's not just a bright shiny object, like, let's keep it shining well beyond, um, whatever the sort of project phase is to make it operationalized. So, there is excitement, there is investment and that other project I talked about, the demand scheduling one coming from the college, that is. Currently being worked on by our central ITS department for that college. So we are trying to bring together these two pieces into some sort of future enterprise tool, but it's not there yet. And Jessica, is that a place where the initiative that you have started or working through are there to support, or what is the role there then for, for that part of the organization? Absolutely. So we started this thing uh with 100% external funding. And the interesting thing is now navigating through many parts of the university that are very excited about what Ingrid and Satya and Heather built, and for example, ITS is very excited and would love to be participating in moving it from fabulous prototype. To enterprise production. And they're working with many different parts of the university to make that work, you know, so it's challenging because it's innovative, you know, it's challenging because they identified something that had to be built in a different way than existing workflows and data management actually works. So people who see it and understand it want to help make it happen. And when they put their minds to making it happen, they realize, oh boy, we have to really restructure a lot of stuff. We have to Pull in people who are not really involved in this thing because of the organization, and this is not just Northeastern.-- My guess is it's-- anywhere is that the organization is structured with lanes that their project cuts across. And so the very methodological success of integrating data sets that have never talked to each other before, when they have succeeded at creating a prototype, now the next challenge is now. Do that again with humans in an organization because the same data sets that need to be integrated also represent divisions and teams and offices of people that also are not integrated or at least they know each other, but they don't have pathways, and it's the pathways for minimizing the friction to being able to get answers that is the next level of innovation. So when you say what's our program doing, we are working very hard. To support those people across different disparate parts of the university that see it, get it, understand it, and want to try to do the scotch tape to kind of put it all together at least to move forward. So we support them in sort of catch as catch can ways like someone doesn't understand it, they need to understand why it's valuable. Here we go. We'll tell you about that. Yes, we really should, right? But then more specifically, we're also now working to create student administrative partnerships, new scoped projects. A new scope that is kind of phase two that maybe instead of being defined for the registrar, it would be a project scoped in ITS with a student that would come or two students that would come with a very carefully targeted project scope to build on what Ingrid and Satya and Heather built to enable the cross-cutting lanes to make it possible to bring it from cool thing to central thing. This is one of those things where I love working in higher education because there's so many incredibly talented people who have so many amazing ideas, and then there are so many obstacles and barriers that we put up in front of everyone in order to do this kind of really exciting and interconnected work that makes things easier for everybody. I've, I continue to not understand why. Institutions aren't leveraging more of their students and you know creating these pockets where we can take some really interesting ideas, really challenging problems, and let creative, smart people work on them in meaningful ways and, you know, get out of their way to solve a problem, and I just I love that that's the approach that Northeastern has taken here and pulling the different groups together. Registrar's office, obviously, I'm, I'm a fan of registrars. I think we're important people at institutions. I think we know a lot. I think we have a lot of context. I think we have a lot of institutional knowledge, and we're overworked. And to your point at the beginning, there's not time. I would love to be able to sit down and And dedicate, you know, hours and hours on one particular thing. But if I can't do it, can I facilitate that? Can I enable someone else to do it, who is capable and creative and qualified to, to work through that kind of a thing and, and to come up with something that I could not even have imagined at the outset. So I just love that approach. I so appreciate this part of the conversation because it is illuminating that everybody who works in higher education has the opportunity to deliver the, like, promise and capabilities of higher education. And I think that registrars do know that the most valuable thing about this project in my mind isn't necessarily that we have this tool. It's not the thing, right? It's that Heather, well, so Heather and Satya have the experience and they have a thing to show for that they want to see continue, but they also have new capabilities from doing the work together and frankly, understanding how higher education institutions work under the hood. Like imagine if they had had that. Experience at the beginning of their graduate programs. So, I think that that, and yet the way that you get attention to do that is not by saying, you know what I want to do? I want to develop capabilities as a registrar's office of graduate students to lift up the hood of these systems and processes and find out where they're broken. That is a very compelling argument, Ingrid, a very compelling argument. It becomes about the tool, and that is why vendors continue to invest in. Like, uh, like, some of our vendors have some of this functionality that they built are in these tools that people use. Some of it, certainly things about figuring out where things show up in different areas of the curriculum. So, I think that there is, you, you know, you mentioned AI and your distaste for it, which I appreciate. There's a, um, there's an article in the Chronicle this morning, Doug, about that institutions that we're focusing a lot on what AI is going to take over instead of asking what people. People are going to need to continue to do. And I think that is the beautiful thing of this project is there was a lot of machine, machine learning and AI and things that went into it, but nowhere in here is there like a diminishing of what a Heather and a Satya are like essential for in this one little use case. So, if we can get commitment to investing in capabilities, in addition to investing in products, like I think we'll be OK. And the whole point of the project is to make things easier for people and, and so to, to be able to facilitate a better student experience, to be able to facilitate a more efficient administrative experience for the students so that they are not dealing. With the administrative part of higher education, they can deal with the sticky part of the education part of higher education. That's the part that we want the friction, where we want the friction to remain. We want to take the obstacles away from enabling those learning experiences for students. This is awesome. That was well said. Good job, mic drop. How will you know that this project has been successful? I mean, I do think we know already. In some of the metrics and goals that we sort of just talked about. So, I know it's successful because, as I said, and I had to, I had to record all of these videos for Heather and presentations that Jessica, Jessica made me do. It was all about that, like, what is the value? Like, how do you know it's successful? Those are all the questions. And pretty much every single time. I said that what made the work so powerful was that it changed the type of questions that we were asking about curriculum and it changed the way that we were thinking about how to approach problems. So all of those assumptions that Satya talked about at the beginning. Oh, well, we assume so many assumptions about we need to know what courses to offer and we're gonna know that if we know what students are planning to do. So, if we can just get at student plans, we'll know what students want, and then we'll know what to give them. And it was like, let's go ahead and just push all of that aside and let's start saying, what are people actually doing? And looking at this map and seeing the problem isn't that we're not offering enough courses here or that this is over-enrolled or whatever, it's that all roads lead to this particular popular course at this point when it's too late for students to necessarily have made a different, maybe they could have made a different choice 3 terms ago. And so, I think it changed that question, not of what are the things that. We should just be doing every term, but like, how should we be thinking about constructing curriculum and working with each other and collaborating in like interdisciplinary collaboration. So, for me, that's the success because to focus on the delivery of the tool, tools come and go, those capabilities will become part of things that we already own, but that, that moment I think is, was transformational. I want to thank you all very much for being here and for sharing this project overview and for introducing ways that higher ed administrators can think differently about problems, about pulling people and different resources together to come up with creative solutions, obviously with the overarching goal of improving student experiences throughout. Thank you for having us. Thank you, Jack McKenna. Thank you for having us. It's awesome. Thank you so much. Thanks to Ingrid, Jessica, and Satya for joining us today. I really like hearing stories about the innovative work people in higher education are doing to remove obstacles for students. I appreciate Ingrid's sentiment, especially about getting the institution to think differently or to ask different questions as a result of this work. Where are the opportunities for you and your office or your institution to bring some people together to creatively address an issue? Thanks very much for listening. You are a data center. You need lots of water. Also, be sure to stretch your legs, turn off your phone, spend some time outside. Until next time, I'm Doug McKenna, and this is for the Record.