How Mature Is Your Moodle? Why Evaluation Frameworks Matter with Dr Mark Glynn

Join Dr Mark Glynn (Head of Business Development at Catalyst IT Europe) and Jessica Gramp (Moodle Community Manager at Moodle HQ) as they discuss the Moodle Maturity Framework 

In this episode, they:

• Introduce the Moodle Maturity Framework and how it works.
• Share why evaluation frameworks are essential for strategic planning, quality assurance, and resource prioritisation.
• Explain how the tool aligns with JISC’s Digital Transformation Framework
• Explore the benefits for institutions, whether they’re just getting started with Moodle or looking to elevate their digital learning environment.
• Highlight why we’ve made the framework free and open to the global Moodle community.

Learn more here: https://www.catalyst-eu.net/service/moodle-maturity-framework 

Visit Moodle at Moodle.com

Transcript

Welcome to a new episode of the Moodle Podcast. Today we will discuss how mature is your Moodle? Why evaluation frameworks matter. To help us understand this topic, we're joined by Mark Glynn who is the head of Business Development for Catalyst IT with responsibility for the eu, Middle east and Africa regions.

Mark has worked within the education sector for over 20 years and for most of that time Mark has been involved with Moodle in some shape or form, using Moodle as a lecturer, providing staff development, Moodle workshops across the HE sector, managing an entire Moodle instance from Dublin City University and co chairing the Ireland and UK Moodle Moots for a number of years.

Mark is also an Advanced HE Principal Fellow which recognises his extensive experience of strategic leadership in higher education. Through leading teaching and learning initiatives in areas including blended learning, technology, enhanced learning, learning spaces, student retention and assessment. Mark has developed a local, national and international reputation for innovation and for leadership in teaching and learning.

Thank you for joining us today, Mark.

My pleasure, absolute pleasure.

So could you just start by telling us a bit about how you've been involved with Moodle over the past 20 years?

Yeah, you summed it up quite well. In fairness, I started with Moodle back in, I would say about 2005, 2006. So if that's Moodle 1.2 or 1.3 there or thereabout, and a little known fact which I remind Martin to Gammas of on several occasions, he started off being the first tech support for Moodle in Dublin City University many years ago.

So that's how long ago Dublin City University and indeed Moodle has been over on this side of the hemisphere. So a long time using Moodle and involved in a variety of different ways, as you said in the intro, as a lecturer, as a coordinator, as a trainer and then indeed owner.

Great, thank you. And could you explain to people about the Moodle Maturity Framework, what it is and how it works?

The framework is. It allows institutions to have a framework, have a structure, do a systematic evaluation of their lms. And that's what a lot of institutions are missing. They take it for granted. The VLE is going for so long, vle, lms, whichever term I appreciate they're interchangeable, everybody has their favorite, but it's been there for so long, people are taking it for granted and people will be giving out about it and people will be complimenting.

Everybody has their champions and so on. But a framework allows you to put structure around it, structure around your evaluation and see where it's good and where it's not, where it can be improved and where it's actually doing incredibly well. And a lot of people, when they think of a vle, they think of a document repository.

They don't think of quality assurance, they don't think of strategic planning, they don't think of scalability and sustainability. They're words that just don't come into their heads when they think of learning management. But if you think about it, it's the one system that students in your institution taught every single day.

Albeit some programs might have more involvement with the LMS than others, but it's very rare. You couldn't, for example, say to an institution, how much traffic do your students use to your website every day? They pump a small fortune in designing and upkeeping their website. They just take the LMS for granted, on the other hand.

So framework like this, the maturity framework, will allow an institution to systematically evaluate how good their LMS is. And that's what I think. That's where the key strength.

Right. And of course, LMS being learning management system, VLE being Virtual Learning Environment. Used in different places, the term is used differently. So could you maybe talk about some of the common pain points or challenges that typically drive organizations to explore and complete the maturity? The Moodle Maturity framework.

Yeah, the most common one is pretty much what, what I've said where people have taken it for granted, say we need to change our lms. That's it. It's rubbish. And it doesn't matter what one you're talking about. But the thing about we're not getting value for money for this.

Well, what is value for money? How are we measuring it? So the pain points really come in to inconsistent, no strategic vision, limited use of reporting and analytics. They want to know how students, which students are at risk of dropping out and disengage with. The VLE is a key indicator of that.

And just to put that into perspective, and I give this example all the time when talking about it. If you imagine each student's fee are €10,000, and I'm picking that figure just as an example, and it's easy for math, but if you get them in first year and they pay €10,000 in first year, great, happy days.

Then when it comes to second year, third year and fourth year, if they failed in first year, you're no longer getting their second, third and fourth year, so you're losing €30,000. So the pain point is actually the budget, hitting the budget, because for every student that drops out on first year, you're losing €30,000.

If you lose 10 students. 10 in a large institution is not a large number by any stretch the imagination. But if you lose 10 students, that's 300 grand a year an institution is losing. So that's a pain point. Never mind reputational damage, program damage, institutional planning. If you can save them that money, imagine putting that money back into the LMS or putting it back into marketing to attract new students.

So for me the pain point is at the coal face where they end up losing students. And if you can use your LMS with good solid use of reporting and analytics, give that insights, data, informed decisions as to where students are struggling and which students are struggling, engaging with them earlier, that will be the LMS is just a no brainer.

What typical signs or challenges would usually alert an institution that's time to evaluate the maturity of their Moodle platform.

Yeah, there's a good few points in that I would say for example inconsistent use across courses and departments even within a course, lack of pedagogical design or universal design for learning is a key buzzword at the moment to talk about inclusive design. But if you're courses are more used as document dumps, right?

Rather than a good structured learning environment and Moodle is a brilliant learning management system built on social constructivism, there's a solid pedagogical foundation to it. Sure, if you're just going to dump up documents, you might as well use Microsoft Teams or Google Drive where the elements come in so much better because they're grounded in pedagogical foundation.

So if you're using them properly with good pedagogical design, you will reap the reward. But this is where the pain points are. This is where the the alarm bells go off. They are just users document dump. You'll also find where staff are struggling with their confidence or indeed they just have lack of awareness or training around the uses of Moodle.

Like say for example, if you do a quick report on your Moodle and see how many quiz questions are being used across your platform, how many quizzes are there? But not only how many quizzes, what sort of question types? I would hazard a guess in a lot of the majority of the question types would be Even though there's 16 different question types in Moodle, the majority would probably be McQ.

So they're not aware of capability or they don't have the confidence or the training or the support available to them. So again, that's an alarm bell that goes off for me when you look at site analytics along those lines. And again, if a course isn't Structured, Right. To optimize you getting the use of the data out of it.

You're not going to be have those key analytics, those key insights that I referred to a while ago. And then the other thing I would say is no investment or no central governance or vision is always seen as an aside. It's not involved in the strategy. But to be honest, it's the biggest influence of your students or potentially the biggest influence of your students.

And I'm not just saying this for online students, I'm saying this for on campus, fully face to face courses. You have a gem at your disposal if you use your vle, if you use your LMS properly.

So I guess you start then by taking a step back and just doing a review initially to see where you're, where you stand before it's clear that you would need to go this way.

Yeah, 100%. You need to take that step back and evaluate it. And this is where the framework is really useful because it comes back to that word I said at the start. A systematic evaluation of your platform. It's not a, it's not a good feeling from a senior manager who saw something else shiny and says, oh, we need that new tool.

Everybody's talking about it. It's not a tool, it is not an IT tool, teaching tool, and that's what it could be viewed as and that's how it should be evaluated. Looking at, as I say, course quality, low engagement or completion rates, the digital skills of our staff, all of those sort of indicators should be looked at at a systematic way.

And it's not that you're competing with another university next door or down in terms of how mature your VLE is. It is how mature it is this year versus next year. Where are you in 2025? Where are you in 2026? Where do you want to be in 2030? So it allows you to set goals and target for yourself.

Again, we need to move away from the mentality that the document dump because you might as well use other tools of document.

Yeah, exactly. So the Moodle Maturity framework is aligned with the GISC Digital Transformation framework. Why was this alignment important and how does it enhance this tool's value?

There are a ton of frameworks out there and we could have chosen any. And for those not familiar with frameworks, a typical framework starts at the heading level. So you'll have, for example, headings 1, 2, 3, 4 and 5. And then underneath each one of those headings you'll have an area of.

And essentially you're raising yourself. Are you at this area of activity? Are you a gold? Are you a silver? Are you a bronze?

Are you?

And there are many activities, there are many frameworks out there and they all pretty much have that same structure. Why we chose GIF because it's so tightly aligned and a great reputation, a great organization. There's no point in reinventing the wheel so tightly aligned with other systematic evaluations that are done across the whole university.

If they end up using this GISK tool. It allows the element owner to start being involved in those strategic and start saying to the IT department, I know you're looking at the IT infrastructure across the college, well, this is what we've done using the same framework, looking at the LMS specifically.

So you're speaking the same language as those that are looking and doing the review. And JISC have done the job incredibly well, not only producing a framework with validated questions with tons of research going behind. Behind each one of the questions, they also have, well, they practice what they preach and they have that brilliant reputation across sector of knowing how to engage and successfully in digital transformation.

And that's what the LMS is. That's what we want this to be, a digital transformation the way it's used within the sector. So that's really why we aligned with JISC and we worked with them closely in the past. Catalyst have worked with them closely in the past. And this to me was just a logical next step.

So could you share some of the key frustrations or barriers that Moodle administrators and educators face that the framework is designed to help address?

Yeah, where do you go? Where do you start? So speaking as a former owner of the lms, not being involved in those conversations, not being involved in that strategic conversation where like you want to have universal design and in a strategy, you'd have universal design for learning embedded throughout it.

And you have the indicators and they don't talk about the elements where that's, that's like one of the biggest tools available for you to implement udl. But the conversations are the buzzwords that they're using. And they're talking about AI or they're talking about learning analytics, they're talking about udl and they're not involving the LMS or they're just brushing it over.

Whereas it's like every click you take on an LMS is tracked on Moodle is tracked. So that gives you a mass load of data that can influence

direction that can help you set those KPIs and move forward. Like in a former institution that we worked with, one of our clients, they had, in the strategy they had, we are taking active learning as our pedagogy of choice. So, and they talked all about active learning within, within this plan.

And it's great. It's a super idea. All about engaging students an awful lot more and moving away from the chalk and talk approach to teaching. But what we did was we worked with them, built an active learning competency framework using the Moodle architecture. And then we were able to get the lectures to highlight or to evidence using the Moodle competency framework language, highlight where they are using active learning within there.

So what happened there is Moodle became involved in conversation because they were able to the institutions, they valued active learning. This is what they did at the hierarchy. They valued active learning and we used Moodle to show them how they can evidence how they can measure active learning across the institution.

There's no good saying all of these buzzwords and not Being able to evidence it, be it for performance or programmatic accreditation, whereas now in that particular institution they're able to do that. And again, we set up similar frameworks for UDL and indeed for digital skills cross lectures. So it's where you get that conversation, bringing Moodle to the next level.

And that's the biggest frustration for owners, LMS owners. And the other thing I would say is because they're not involved in those conversations, you'll have investments made by the university that if a fraction of that money was put into the lms, could bring it to the next level, could bring it 10 level forward.

And the obvious one that springs to mind to me is the physical infrastructure that's put in place where you will have a new building built which cost millions, yet they'll try skimp and save on the lms, which costs thousands as opposed to millions. And that building will affect a fraction of the students on that university because not every student is going to be in that particular building immediately excludes or is of no value to the distance education students or the part time students.

The LMS affects every single student and allows you to reach so much further than your physical location. So again, not being involved in those conversations or not being concerned, considered at a strategic level is very, very frustrating.

And I also wondered, have you seen students involved in this process as well?

Yeah, if I go back to my old institution, dcu, they took a great student centered approach, students and partners as assessment and they looked at and again this was part of their tackling for academic integrity, which is a huge, incredibly important element of any and strategically important. But what DCU done was they brought the use of the VLE into that by actually engaging students, co creating rubrics, co creating assessments and having the flexibility that an LMS affords by allowing flexibility in choice of assessment or assessment mode.

But the students were front and center in those discussions. So again it enabled the LMS in there by stealth, for want of a better expression. Both the students were the focal point of it. And again, going back to an earlier comment, students, the current students that a university have, they engage with the LMS more than any other platform, more than any other system.

They don't engage with student record system every day, they don't engage with the college website every day, they engage with the lms. So they really need to be taken into consideration when it comes to design and when it comes to use of the various.

So why might an institution feel like their Moodle platform isn't meeting its full potential and how can the framework help uncover the root causes?

There's a lot of similarities really in the other questions where, like what are the pain points? What an institution. Some institutions might think we've just been using this for so long, time for change, whereas in actual fact they don't realize change. Do this digital transformation and change from one VLE to another actually costs an awful lot of money.

Never mind the platform change, never mind the technical migration of your course content from one to the other. What matters is the extra budget involves retraining the staff, readjusting all your systems and bringing their confidence level straight back up again. And the new platform. And what happens then when they don't have those conversations first, when they don't start evaluating the system first, it's only anecdotal evidence that comes to play, not data informed decisions.

A class example would be an institution not investing in the infrastructure within the college and they invest in Moodle for say or sorry, they would. The lecturer would use Moodle for an online quiz and they'll bring all of their students into the class for academic integrity reasons. They want to see all their students face to face while doing this online assessment.

And the quiz crashes. Now, the quiz doesn't crash because of Moodle. The quiz crashes because there's limited WI FI available, where you'll have 200 students into one room coming in. But Moodle, Moodle is what gets the damage. Moodle is what gets rip. Ah, well, Moodle let me down in that particular circle.

And again, not could be any VLE that's in there, but then that damage is done. But if you had data informed decisions, if you had analytics track the activity that's done on the site, why it went down, or if it went down at all, you're able to turn around the selection and say, actually no, it's your infrastructure that's let you down, it's your WI FI point.

You only have one Wi Fi point for 300 students is what lets you down. But again, the VLE, the LMS, whatever it's called in each of your institutions, that protocol that's actually blamed, but you have better data informed decisions, I think that would be. It allows a much more robust decision to be made about the lms and in some circumstances it might be time to change.

But I would challenge any institution who has changed their LMS over the last 10 years, use this framework and see if it has actually made an impact. Now, the only challenge they have is they don't have the historical data, but you got to start somewhere. So Institutions should start using this framework or other frameworks like it, but do have that systematic evaluation of their, of their elements.

And so for organizations struggling with engagement, scalability or aligning Moodle with their teaching goals, how can this framework provide clarity and direction?

I think it provides it in a few ways, but if you just look at the structure of the framework, so there's six different areas where you're asked these questions and for those not familiar with it, it's a self assessment tool. So we're not looking over your shoulder. Our Moodle HQ isn't looking over your shoulder.

It's a self assessment tool, but it's knowledge management and use, organizational digital culture, knowledge exchange and partnerships, knowledge creation, innovation, knowledge development, and then there's the digital and physical infrastructure. So you're asked these questions across such a broad range. It really gives you the holistic view of the LMS and allows you to see the gaps because in some cases you may have great stuff around knowledge exchange and partnerships, but you don't have that digital culture that's involved, you don't have that fault use of let's use the LMS for this.

Or how you manage your data, how you manage how you archive off your courses and how you create new courses every year, how you embed innovation within it, or is it the same LMS platform that it has been for the last 20 years? There's loads of different ways you can target it and when you look at it under those six headings, you get that holistic view and that's I think the power of the framework, because you get university wide view of how you can improve, of the status of your element, but also how you can improve.

And you've already touched on this. But how does the framework help make the case for more resources or investment in Moodle from senior leadership?

Well, if I go back to what I was saying earlier on, where a standard framework will have those questions, the areas of activities and then the questions within that area and you'll get an overall rating of a gold, a silver and a bronze. So and again, that's your standard framework.

We would use in this particular framework, emerging established and an expert with the same sort of principle, it allows an institution to look at those various different elements and say, right, in this area I'm a bronze, I'm quite happy being a bronze. Strategically, this is the direction where we're going.

Or in this area I want to be a silver or I want to be a gold. And because you then have this systematic evaluation, it allows you to tie it then to your strategic plan to say, right, well, in your strategy, the university says we are going to be a fully inclusive university, yet we've rated our use of the elements under knowledge development or under digital culture, and we're rated as a bronze.

We need to if we've to achieve this goal, we need to be a silver by next year or two years time, and we need to be a gold by five years time. That's how you then start putting your argument together. And that's where LMS owners tend to fall down, in my opinion, because it's not viewed as are taught about at level.

So therefore making those decisions to get additional investment is. Actually quite difficult or is viewed in a short term view. How many LMS owners or LMS departments? The department that would be responsible for the LMS have these short term contracts for their staff and they're just getting it in in piecemeal funding.

That's lack of strategic vision. If they're not optimizing the use of their LMS properly and they're not seeing the long term gain they can get from it. Again, simply going back to save 10 students through your VLE and it's 300 grand a year, it's a brainer for me. So that's how you start doing it with data informed.

Right? Okay. And catalyst it have been a Moodle certified partner for over 20 years now. How has your experience shaped the development of tools like the Moodle Maturity Framework?

It's been really good. As I said, we built ourselves in this particular framework on really strong foundations with JISC using their framework. As I said, there's no point in reinventing the wheel. However, the advantage or the extra bits that we were able to bring to it as a Moodle partner, the biggest higher education Moodle partner worldwide, we were able to go to our clients, get them to evaluate the questions that were going get them to pilot the use of those questions.

And while we taught, the question was very clear. When we got feedback from all of our clients or the majority of our clients, they were saying actually no, in our context this means something else. So we had to, they helped us through the research that they'd done or that we'd done, excuse me, with them that we were able to refine the questions and we couldn't have done that independently of ourselves.

We needed their expertise, we needed their local knowledge as to what it would be within their own institution, as to how it would make it most effective. So that validation stage of our question was key for us. And the other thing, again, we're very fortunate we have such a diverse, diverse array of HE clients that we were able to get that broad spectrum and make sure that the analysis or the self assessment tool is fully inclusive of institutions.

And I would argue now as well, based on that research that we could make the vast majority of the questions would actually be suitable for other VLA for other institutions that don't use, for whatever silly reason, don't use Moodle, it still would be applicable. There are bits specifically focused on Moodle where we talk about the power of plugins and the open community that exists and the nature of open source.

But the vast majority of it could actually be applied to other LMSs as well. So for me, going back to your question, the benefit or how Catalyst influence came in is the wide variety of clientele that we have and the fact that we were able to call upon on people within those institutions to contribute and interpret the questions for us and validate the questions.

Great. So it's really been a collaborative effort with your clients throughout this process by the sounds of it.

Yes, it most certainly has. And I look forward to with the remainder of our clients who weren't able to get involved in everybody's undertime pressure. So I'm not being critical at all, but I look forward to the remainder of our clients engaging with it and seeing how they can use it and influence it going forward.

We're always up for, like any good pedagogy person, we're always up for refinement and improvement and we'll always be looking for feedback on how to make it better. Whether it's our clients or anybody else. We would welcome that and it will be an iterative process.

And what trends are you seeing in the Moodle community that institutions should be preparing for and how does that Moodle maturity framework help institutions stay ahead?

The more trends that we're seeing, obviously everybody's talking about AI, can't wake up in the morning without seeing something AI related and so on. And we are unfortunately seeing trends where people are shying away from online assessment, for example, because they're concerned about academic integrity. So the way to address that for us, and we spoke about it recently at the Africa Moodle Mood and it was also part of the E Learning Africa conference, is how you can use the lms, how you can use Moodle to enhance, enhance academic integrity.

So we're trying to reverse that trend of moving away, back into the face to face classroom, exam hall situation. Say well listen, if you do X, Y and Z, which are Moodle platform, it can actually enhance it. So we're trying to work with institutions, with our clients and elsewhere to try and improve the pedagogical use of Moodle to tackle specific issues that institutions would have rather than generally talk about Moodle for how great it is, but to try solve the problems that institutions have.

So that's some trends that we've seen. We're also seeing higher usage of Moodle, which is great, but then we're seeing that institutions that don't have it set up properly can't deal with that intensity on the infrastructure. So Moodle crashes, it falls over. So what we're seeing now is institutions saying right how can we operate at scale?

How can we bring the VLE to make sure we have a robust and reliable infrastructure? Because it is thankfully being recognized by more and more institutions as an ideal vehicle to bring them into that world. That opportunity of blended learning that really can explode the use of technology across their institution.

And you mentioned before universal design for learning, udl, so all about improving accessibility for students and, and staff as well who might have disabilities. How does the Middle Maturity Framework kind of tie in with udl and how can it be used to improve that across the institution?

Yeah, what I love about udl, and I've been a strong advocate for it for years, what I love about it is the fact that it's not just for disabilities, it's for full inclusion. Right. Where you're talking about having content available online for somebody to look at just because maybe they just can't get on campus at that particular evening.

They're looking after their kids, nothing to do with disability. They're looking after the kids or they're looking after their parents. Transport costs are so expensive they can't afford to get into college every day of the week. So udl. And embracing UDL allows you to include all students and diversify your audience, be it part time or full time.

And with regards to Moodle, how it can be used simply at the simplest level, putting your notes up online in advance of the class. Loads of people always give us, you know, when, if I put my notes up, they won't come into class. If you're only as good as your PowerPoint notes that you're putting up.

You are so underselling yourself as a lecturer. You are so much more than that. So I wouldn't use that excuse at all. And you give them the value of you being them being there on campus or engaging with it. Also having the choice of assessment they can use, whether it's the group choice tool or whether it's the assessment where they choose a particular mode of assessment is made for them or they choose a topic.

So as a lecturer, you're not getting, if you have 100 students, you're not getting 100 versions of the same topic. Bit back out at you. If you give the students choice of what topic could be, you may be giving five different choices, so you're getting only 20 versions of the one topic.

But you're also giving them that sense of ownership given that, which is reducing the rationalization that they may have to actually do academic misconduct. So it all ties into a loop. Yes, it is udl. It's supporting udl, but it's also supporting students as partnership. It's also supporting academic integrity.

So when you take that holistic view of IT and knowledge development or knowledge exchange and partnership is a great way to do that. And what I love about the Moodle community is the openness of it. Where if you are talking about UDL and how you're using UDL within your institution via Moodle, presented the global moodleness.

Right. Or presented local Moodle users group, it's that exchange of knowledge or the way you might partner, you might come over, oh, I wish Moodle would do X, Y and Z. Well actually I don't have the money to do it completely. But if we club together we could actually form the that partnership, develop this plugin and release it to the community.

There are so many different examples that are there where something has been used, something has developed to enhance udl, but everybody benefits. So many different ways to go with it. And the last one, I'll say could talk about this all day, but the last one, the organization and culture, which is an element of the framework that will also bring you an opportunity to highlight UDL and where it could be improved within your organization, which is great for everybody.

It doesn't harm anybody whatsoever by including everybody.

Exactly. And that's a really good point that it's not just about disability. It's one of my favourite things about accessibility and universal design for learning is that making these changes does help everybody in different situations. And it's really about being more inclusive across the board. So that's great. Okay, I've got one final question for you.

For organizations that are unsure of where to start, what advice would you give to take that first step towards evaluating their Moodle platform?

Take the jump, reach out to us, or indeed to yourself and get the link to this

self assessment tool. We make it freely available. It's on our community site. Take the jump, reach out and we'll allow you have a shot at it. No cost, no hassle, no harassment from us. Afterwards you go do the self assessment tool. I would say there's six different elements of it.

So it's good to have a team approach to it. Doesn't necessarily have to be everybody involved. We did initially have the tool as 62 questions. I think in total. 62 questions all on one where you needed to have everybody around the table to answer those questions. Whereas based on feedback now we've split that up into the six different sections so you don't have to have everybody together around the table at one time, but I would strongly encourage it that when you do get into the mode of doing this, that you do open the invitation up to your relevant colleagues within the institution to answer those six different areas.

But the first step? Reach out. Mark Glynnatalyst eu.net Or just go onto the Catalyst website.

I've just done a quick Google for it myself, so just typed in Moodle Maturity Framework and your website came straight up with all the information. So that's great.

There we go.

Thank you, Mark Glynn, for your time today and for sharing so many valuable insights about the Moodle Maturity Framework. To access more episodes of the Moodle podcast, please visit podcast.moodle.com and from there you can listen, read the transcript, or access the show notes for each episode. Thanks for listening.