Leticia Britos Cavagnaro: Make learning visible

Dr Leticia Britos Cavagnaro is the co-Founder and co-Director of the University Innovation Fellows, a programme of the Hasso Plattner Institute of Design (d.school) at Stanford University. Leticia uses emerging technologies to empower learners to be self-directed, action-oriented, and reflective shapers of the future.

In this episode, she shares her view on why reflection is important, how to scaffold its practice for learners, and how to incorporate it into every learning experience. She talks about her upcoming book 'Experiments in reflection', published by Penguin Random House, and her recently published Riff chatbot.

Click through to the episode notes for the transcript.

Connect with Leticia

• LinkedIn
• Twitter

Resources

• Blog post introducing Riff: 'Reflecting with AI: A Tool to Develop Human Intelligence'
• Enchatted, company that did the technical work on Riff
• 'Teaching & Learning' programme at the d.school
• Bulk discounts of 'Experiments in Reflection'
• Full d.school series of books at a discounted price for the set

Reflective frameworks mentioned

• 'What? So what? Now what?' by Terry Borton
• 'I Used to Think... Now I Think...' by Harvard University's Project Zero

Scholars mentioned

• Ellen J. Langer: 'The Power of Mindful Learning' (1997)
• Guy Claxton: 'Learning to Learn – The Fourth Generation: Making Sense of Personalised Learning' (2006)
• Gavriel Salomon, David Perkins, and Tamar Globerson: 'Partners in Cognition: Extending Human Intelligence with Intelligent Technologies' (1991)
• Erica Estrada-Liou, MBA
• Meenu Singh, MS

Subscribe to the monthly newsletter about Mahara and portfolios.

Production information
Production: Catalyst IT
Host: Kristina Hoeppner
Artwork: Evonne Cheung
Music: The Mahara tune by Josh Woodward

Transcript

Copy Transcript Copied

Kristina Hoeppner 00:05

Welcome to 'Create. Share. Engage.' This is the podcast about portfolios for learning and more for educators, learning designers, and managers keen on integrating portfolios with their education and professional development practices. 'Create. Share. Engage.' is brought to you by the Mahara team at Catalyst IT. My name is Kristina Hoeppner. 

Kristina Hoeppner 00:27

Today I'm speaking with Dr Leticia Britos Cavagnaro to whom Helen Chen introduced me only a short while ago. Leticia is a scientist turned designer, who is the co-Founder and co-Director of the University Innovations Fellows, a program of the Hasso Plattner Institute of Design, better known as d.school at Stanford University. It was actually Helen, who took me on a 25-cent tour of the d.school a few years ago, having heard so much about it. 

Kristina Hoeppner 00:55

Today, while we are not talking about the d.school itself, the d.school is very much involved in it of course, and that is that we are going to talk about the work that Leticia has been doing in regards to reflection and using artificial intelligence in that. Leticia, thank you so much for making time for our chat today.

Leticia Britos Cavagnaro 01:14

Thank you, Kristina. It's a pleasure to be here.

Kristina Hoeppner 01:16

Leticia, I already mentioned a couple of things that you do at the d.school, but can you please tell me a bit more about yourself? What do you do as designer there?

Leticia Britos Cavagnaro 01:27

As you mentioned, I'm a scientist turned designer. So I started my professional career in an experimental lab in biology. That's where I got my bachelor's, my master's, and my PhD. Along the way, I discovered my passion for teaching and learning. And that's what led me to the d.school. I actually was fortunate to take one of the very first d.school courses. It was a very experimental one week intensive. And once I stepped into that space, and at the time, it was not the building that you toured with Helen, it was actually a double wide trailer. It was a temporary space that the group that were starting the d.school had gotten, but there was something so amazing about their vision, their vision of bringing together students, graduate students across all disciplines at Stanford. 

Leticia Britos Cavagnaro 02:26

So I was doing my PhD in developmental biology, and I found myself next to students studying history and literature and chemistry and engineering. And also the idea that we need to activate our imagination and our creativity, as much as we need to lean into our analytic and scientific skills or the scientific kind of thinking that for me was fascinating. I said, "Well, this is my place. I'm not leaving," and I have not. So I have been working full time at the d.school since 2016. 

Leticia Britos Cavagnaro 03:00

As you mentioned, I co-direct a programme there, called University Innovation Fellows, and I also have the pleasure of working with university educators and students through this programme across universities globally. So we reach students and educators from over 300 universities. And so I get to co-run experiments and play in the realm of pedagogy with all of these fantastic community of educators.

Kristina Hoeppner 03:29

That is amazing. I mean, I must say, when we did our little tour, it was the summer break. So I didn't actually see any of your there. It was a pretty deserted building, but it was fantastic to see the setup there with the moveable walls and the walls that you write on and that post-it notes are stuck on, and I really felt the creative energy and the way that you can think just differently because you could reconfigure the space very nicely, instead of just having everything just set up one way. So it's fantastic that you saw the evolution of the d.school.

Leticia Britos Cavagnaro 04:04

Yeah, and one thing that is important, if I may build on your comment about the space is that temporary spaces are very valuable because oftentimes, the institution doesn't care too much about them. So you can maybe tear down a wall and create something new that in those more permanent spaces is difficult. However, the d.school has been designed even in our current permanent space, it's designed to have an adaptive built pedagogy, if you will, because I think that this space is the built pedagogy of the d.school. We've built it in a way that we can do a lot of changes within that space to adapt to the needs of our students, to adapt to the needs of our teachers and our teaching so that's an important affordance of our space.

Kristina Hoeppner 04:52

Coming back to our topic today, I mentioned in the beginning that we are talking about reflection and artificial intelligence. So Leticia, what sparked your interest to study and research the use of reflection for learning?

Leticia Britos Cavagnaro 05:07

That's a great question, and actually, I often think about that, and I can pinpoint it to a specific moment. It was the first time that I entered a classroom as a teacher. I experienced first hand the complete failure of the transmission of information model of education. I had just finished my master's in cellular molecular biology, and I had taken a position teaching at a teacher's training institute. So I was going to teach biochemistry and genetics and cell biology to future science teachers in the K-12 space. I remember being very excited and saying, "Okay, great." My first class was about metabolism. So you know, it's like, "I know this stuff." I just dusted the textbook that I had used as a student in the university, I prepared my transparencies that can give you an idea of how long ago that was, it was not a PowerPoint, and I marched into a classroom with great enthusiasm, and I started telling the students "look at this beautiful set of reactions and how amazing all of these enzymes and regulation and the implications for pathology and all of that," and when I turned around, I see my students' blank faces. They're just not excited, actually falling asleep, you know laughs], being bored by what I was telling them. 

Leticia Britos Cavagnaro 06:27

For a time there, I was literally stuck. I was stumped by what's going on here. I learned. That's how my teacher taught me when I started studying this topic, and I learned. Why are my students not learning? I had a keen sight. After a while, I realised that I had not learned because of how I had been taught, but in spite of how I had been taught. I had, in a way survived bad teaching, and not because my teachers were not dedicated and passionate about the topic, but they had also inherited a pedagogy that was passed along from probably the time of Roman amphitheatres where there's people talking in the front and others listening. I realised, well, this is not a good metric because some students will survive anyway you teach them and others probably you're not going to connect with this model of just this is about me pouring information into your brain. That's not how we learn. 

Leticia Britos Cavagnaro 07:29

So that led me to really start diving into how do we learn? How does it happen? And starting understanding the different functions of the brain, and that's what led me to understand that in order to use our full brain and really learn reflection is key because not to go into the details of the anatomy of the brain, but we have our sensory cortex, where we're getting information, getting everything from our senses into the brain. And we have our motor cortex, which is about doing, moving, putting things out there. But in between, we have the integrative cortex, and that's where reflection happens. And that's where actually when you allow for the time, and it's a slower process of really making connections, creating new connections between the neuronal networks that represent your existing knowledge in assumptions and mental models and new information and creating those new neural networks, is that you learn. Reflection is the process that catalyses that. When we are not building reflection into the learning experiences that we create, we're, you know, basically, robbing our students of the opportunity to learn.

Kristina Hoeppner 08:42

How have you started introducing reflection? 

Leticia Britos Cavagnaro 08:45

In so many ways, but you know, in essence, for me, a learning experience always needs to have a component of reflection. We need to be mindful of that. It's not always at the end of an activity, it might be in different ways throughout an activity, different moments of let's pause and let's allow for the individuals to actually have a introspective moment, if you will, of stopping and making sense and then maybe turning to a partner talking about it, and maybe then we talk with the whole group and really process what's happening. 

Leticia Britos Cavagnaro 09:20

That's particularly important when you are talking in any kind of activity, but specifically when you're doing active learning, where you're putting the students into a situation where they have to do something, it's really important that we don't just have them do, do, do, and not make sense of what happened, what they observed, what they noticed themselves doing, what they noticed others doing, and really make that pause. 

Leticia Britos Cavagnaro 09:44

I did an informal experiment a few years that was pretty informative for me in my journey of helping students be better at reflection. I want to foreshadow that this insight might sound obvious to some listeners, and maybe it is, but for me it was important. I did this experiment in the context of an introduction to design class for graduate students across different disciplines. It was a quarter long class. So Stanford is in the quarter system, trimester, if you will, 10 weeks. In the second week, our teaching team asked students to reflect on a particular activity that they had had to complete as part of their first design project. And we asked for this reflection via a simple Google form that they submitted at the end of the week, and they continued to do that for every week. Then what we do is actually we anonymise the reflections and start the next class with an activity where everyone reads everyone's reflections. But that was not the part of the experiment. 

Leticia Britos Cavagnaro 10:39

In that second week, we asked students to reflect on this activity, it was an interview, a series of interviews that they had had to complete with stakeholders. We didn't give them any specific framework or way to scaffold the reflection. The third week, we asked them again to reflect on a project activity using the same Google form, but this time, we gave them a framework to think about the structure of the reflection and what components the reflection needed to have. 

Leticia Britos Cavagnaro 11:07

The framework that we gave them can be summarised as 'What? So what? Now what?' So, 'What': What did you notice? What did you observe in yourself, in others? How did you feel? All about noticing. The 'So what' is about: What might explain some of your observations, making inferences? And the 'Now what' is projecting yourself into the future. What implications does this have for the next time that you do a similar activity, for instance?

Leticia Britos Cavagnaro 11:33

Then what we did is we contrasted the reflections from both weeks. They were night and day. In the reflections where students had followed the framework, we could see much more specificity, much more nuance, much more depth of insights. So the learning for me, the insight for me was that giving students the opportunity to reflect is necessary, but not sufficient. They also need to understand and practice what a substantive, deep reflection looks like. So they need to practice and get good at the 'how of reflection' as well. So that's really what led me to go deeper into, okay, how do we build not only moments of reflection for a specific activity, but activities that have to do with practising getting better at reflection.

Kristina Hoeppner 12:21

That scaffolding I think, is really important because we can't just expect that students know how to reflect, similarly, how we cannot really expect that they know how to give good feedback, how to give constructive feedback, and not just 'Oh, mhh, I don't know, it was okay,' or things like that.

Leticia Britos Cavagnaro 12:36

Exactly. Or that's also the case of teamwork. It's not enough to say 'The four of you or a team. Go have fun,' but how do we give them frameworks, activities in which they practice what good collaboration looks like?

Kristina Hoeppner 12:51

Yeah. So Leticia, is that then actually quite a big component of your upcoming book 'Experiments in reflection' that is going to be published by Penguin Random House this November, so November 2023?

Leticia Britos Cavagnaro 13:05

Yeah, I'm so excited about that, yes. I can tell you a little bit about the book and how it in a way allowed me to synthesise all of my experiments, all of my experiences with reflection in the classroom, in my classes. Reflection is what Marvin Minsky would call a suitcase word. By itself, it doesn't really mean anything until you unpack it. So one of the goals of the book is to do just that. Unpack the meaning of reflection, a word that otherwise, if I said, like, "Oh yeah, I reflect" and you say, "Yeah, yeah, yeah," but we might mean completely different things. 

Leticia Britos Cavagnaro 13:40

The book unpacks the meaning of reflection, but in a way that does not intend to provide one more scholarly definition. In fact, when I started writing the book and doing research, there are so many definitions of reflection that it would take me just the whole book to discuss and contrast them and say, "Well, this one and that other one." So I took a different approach. The structure of the book itself embodies a definition that I think can help unpack dimensions of reflection in a way that are actionable for the readers, for the learners. That's what I call my non definitive definition of reflection, and I'm giving you a sneak peek on the book.

Kristina Hoeppner 14:23

Yes, please because it's not there yet. 

Leticia Britos Cavagnaro 14:25

Yeah, I know. I know. 

Kristina Hoeppner 14:27

I'm definitely interested in getting that sneak peek and what we can expect because already the blurb sounds really interesting that you give little activities on hand to guide us through to scaffold the introduction to how we can use reflection. 

Leticia Britos Cavagnaro 14:42

Yeah, so my non definitive definition of reflection is "reflection can be seen as a whole body process to transform experience into meaning to shape the future." So you can think about three different parts. The whole body process is about noticing, but noticing in a way that is comprehensive of all of your senses. We often tend to think only about the visual cues or some sort of stimuli. But how do we really get good at noticing a change in our perspective, at uncovering other aspects of what the reality that surrounds us, that we might not be noticing it, right? 

Leticia Britos Cavagnaro 15:20

Then the transformation of experience into meaning is really about making sense of what we notice. How do we take what we experience and we make sense of it? That has to do with activating ways of thinking that are normally not practised in most education systems: abductive thinking, creating inferences, thinking about what could be, not just what is. 

Leticia Britos Cavagnaro 15:44

And then the third to shape the future, for me is like, very, very important, because we often associate reflection with looking back, looking back, but we not only can reflect on the future, but we should reflect on the future. So it's crucially important that we reflect on the future because we need to be and help our students be shapers of the future. And when I say the future, actually, we should really be thinking about futures, possible futures, many possible futures, and thinking about how the future is not something that is going to happen to us, but we need to imagine a multitude of futures, and then figured out what can I do today to steer towards some of those futures that I want or that I envision as being better futures for myself and for others, more inclusive futures and what can I do to steer away from futures that scare me? I think that changing mentality about how we think about the future is an important part of the book. 

Leticia Britos Cavagnaro 16:50

The other aspect that I think is important, and as I mentioned earlier, I come from the world of science. I've been an experimenter [laughs] for a long time, is that the book is written as a collection of experiments. It's an invitation to learn by doing. I took many of the activities that I've included as part of my classes to help my students grow as reflective individuals and adapted them to make them suitable for doing them as an individual because, of course, unlike in the classroom, when I can guide my students through an activity, give instructions so that they experience unfolds, here, the reader has all the power. I have to write everything and hand it to them, and then they're on their own. But I think that in framing these experiments, these concepts, and these activities as experiments, I want to put people in the frame of mind of 'We all need to be experimenters.' It's not just for scientists. We need to be experimenters in our daily lives, as educators, as learners. And these are designed - also one little nuance - as private experiments. That means that you don't have to show your results, you don't have to show your ideas, and it feels a little bit more safe than when you're in the classroom, and maybe you're working with a team or you have to show your work to others, which I think is important, but also getting people used to experimenting can feel a little bit out of the comfort zone. So I want to provide with the framing of the book, a safe environment for getting into the habit of experimenting.

Kristina Hoeppner 18:25

I think that language also really nicely invites people to take what you have given as an activity in the book, and because it is seen as an experiment, then make little changes and tweak it maybe they do involve a colleague of theirs, or maybe they try it out in a class. 

Leticia Britos Cavagnaro 18:42

Absolutely. 

Kristina Hoeppner 18:43

But not see it as 'This is the rule of how you should do reflection, but here's an invitation, take it, adapt it to your own needs, for your own context, in your own culture. Make it your own. I'm giving you a framework, a starting point, and then let you run with it.'

Leticia Britos Cavagnaro 18:59

Absolutely. You got it. I think that's very important in education in general is that we frame things in more conditional ways. There's a couple of authors that brilliantly write about it. Ellen Langer from Harvard, Guy Claxton, about the importance of the language that we use and how we frame things for our learners. If we say, "There are three ways of understanding this." Then it's like, okay, there's not four. Normally, all interpretations [laughs] should leave space for other interpretations. If I say, "These are three ways in which we could understand this," and that's why for instance, I say, this is my non definitive definition of reflection, it's an invitation for you to explore reflection in this way. And as you said, through the activities it's an experiment. Try it and tweak it and what works for you, and then I'm going to give you a lot of more information of experimenters that have tried these methods but also data from research, from control experiments, data from, you know, the theory so that you can make the connections, but start with your experience. Because, I think, that's something that, for me, is important in my philosophy of education is invite students to not discount their experience. Oftentimes they feel that, well, everything that they're reading in a book or everything that the teacher says is more important than what they're experiencing. They have to reach a balance, right? It's not that they should not learn from sometimes years and years of experiments and work in development of theories that many in a particular field have done or the experience of their teachers, but they should not discount their first hand experience.

Kristina Hoeppner 20:39

Yeah. Leticia, recently, and that's what sparked this whole conversation between the two of us today, you launched Riff, which is a chatbot that takes your ideas of the experiment and also your three main components, the noticing, sense making, and envisioning the future very much into account and has those three elements at its heart. Can you tell us how does Riff work? Why did you come up with it?

Leticia Britos Cavagnaro 21:09

Let me start by answering the last part of your question, the why, why did I come up with it? For that, I have to take you back a couple of years before Riff was born or hatched, whatever we want to say that with bots or these kinds of tools because I think that can put things into a useful perspective. So at the d.school, we create transformative learning experiences for the students who enter our classrooms. And when I say students or learners, those could be Stanford students, those could be corporate executives, those could be people working in government, educators, we have the great fortune to work with folks at different stages of their lives, but we know we can create such transformative experiences when they enter our classrooms, which we call studios. When the pandemic forced us to teach virtually, we got creative, and we came up with a range of experiences that really unlocked the power of learning where you were, or where the learners were, learning in context. 

Leticia Britos Cavagnaro 22:12

That really pushed me to focus my attention more and more on how to support learners, not only where they are, but when they need to learn. So just in time learning as well. Because after all, we spend a great deal of our time outside of learning environments, be it physically or virtually. Sometimes we need to learn things when we're at the supermarket and we need specific information or at the gym, whatever it is. So I started experimenting with creating chatbots that would be available 24/7 and could guide students in learning where and when they needed it. We're talking about way before ChatGPT. 

Leticia Britos Cavagnaro 22:53

The chatbots that I really got good designing were scripted chatbots, meaning that their responses were pre written by me. If you've had experience with one of those customer support chatbots on websites, you're probably thinking, 'Oh, no, those are terrible. What are you talking about?' But actually, as with many things, it really depends on how you design them. For me, they became sort of like a language. If something could be an asynchronous on demand experience, I could design a chatbot for that. So I got deep into conversation design. 

Leticia Britos Cavagnaro 23:27

I discovered that I could create on demand learning experiences that not only felt authentic and engaging for the learner, but also had a few very interesting affordances. For one, they kind of like capture a trail of data and the flow of the conversation that I could then synthesise and analyse and even bring back to the learners. It is perfect for asynchronous activities, as I mentioned, like on demand. It scales: Once I design a chatbot, it can interact with hundreds or thousands of learners. And very importantly, it responds uniquely to each learner. Even if my responses are pre written, I could ask questions in which there's a fork: some students aren't going to answer A, some students are going to answer B, and depending on what they answer, I could pre write an answer for that that was different. So that would take them into different paths. There was a lot of differentiation in the learning. 

Leticia Britos Cavagnaro 24:27

In particular, I designed many chatbot led reflection activities that actually parallel the ones that I included in the book. When generative AI technologies like the ones powering ChatGPT, which is large language models, became available, I decided to try and see how can I use generative AI to power a reflection assistant? That's how Riff was born. 

Leticia Britos Cavagnaro 24:52

The idea behind Riff is pretty simple. You start with focusing the attention of the learner to an experience that you want them to make sense of. It could be an activity that happened in class or an activity that happened in a project, and Riff will ask the learner questions to help them examine that experience, follow-up questions. These questions are different than my [laughs] previous chatbots are not scripted. They are generated from the large language models based on the answers that the learner gives. In this case, the gist of the design work, which by the way, it's a collaboration with Stergios Tegos from Enchatted. He's a colleague with a PhD in chatbots in education, which has been an amazing collaboration.

Leticia Britos Cavagnaro 25:38

The design work has been to take those large language models and create a set of directives to steer Riff's behaviour towards asking the most productive types of questions, and steering it away from doing other things that we know large language models are actually not good at, like providing information or advice because we know that generative AI can produce what is called hallucinations, right, and very confidently shares information that is completely inaccurate. In this sense Riff sidesteps these issues because it's not providing information. It's asking you questions about your experience. You could say that there are no factually wrong questions. There could be questions that are more or less productive for you in conversation, but there's no really wrong questions. That's how Riff works.

Kristina Hoeppner 26:29

What I find fascinating with that is, as you say, it's not providing you with information, but it is asking you questions. It is very easy to get started with because all you need is that initial question. I wouldn't even call it a prompt because in contrast to using ChatGPT where you prompt something and then also explain how you want the chatbot to respond, the educator just needs to have their starting question which gets the students going with their reflection and all the science behind - I guess then how the chatbot is responding with the follow-on questions to get to the 'What? So what? Now what?' responses to your reflective framework is built in so that you're sidestepping, kind of getting off on the tension, but really getting toward that reflection and eliciting more of what the students really thought about the learning experience there?

Leticia Britos Cavagnaro 27:23

Yeah, absolutely. As you mentioned, here, my goal with Riff is not to replace the teacher, but to augment it. Riff can be a co-instructor of sorts. You can pull it in as a tool, a platform for educators to customise it. So they can change that initial prompt so that they put in context to whatever activity they're doing. So if they want the students to focus on a specific activity, they can ask them to provide their initial reactions to that activity, then Riff takes care of the rest, as you said, in terms of its behaviour has been trained to lead the students in reflection. One thing that is interesting, there's an old paper from 1991, 'Partners in cognition', I'm trying to find the complete title so I can give it to you. But I can also share it later.

Kristina Hoeppner 28:11

Is it 'Partners in cognition: Extending human intelligence with intelligent technologies'? 

Leticia Britos Cavagnaro 28:17

Yep, that one exactly. So this is a paper from 1991. This is way before we had this powerful artificial intelligence. Their point has to do with how we use technology and how we think about using technology and specifically for teaching and learning. And one key concept that they bring is distinguishing between two kinds of cognitive effects. The effects with technology that is basically the effect that you have in the moment that you're using the technology, but also the effects of technology by using the technology, are you learning some skills or some mindsets that then without the technology, you're able to do something or do something differently? Those remnants, they call it 'transferable cognitive residue', which sounds very academic. I hope that with Riff, we're providing both. That it's helping you sort of like go deeper in that particular reflection of where you're using Riff, but it's also training you to ask yourself better questions when you're faced with an experience that you want to make sense of so that Riff doesn't become a clutch, but it becomes training wheels, if you will, or a balance bike that then allows you to be a better reflected on your own.

Kristina Hoeppner 29:32

Leticia, you mentioned earlier that you want to augment the learning experience and not replace the instructors or tutors and educators. So what I find nice with Riff is that the teacher who sets up the chatbot can actually look at the log of the activity, so what the students have written, and then take that input, maybe present it back to the class anonymised or go further into that conversation with a student if they have a follow-on conversation. So that's really fantastic. 

Leticia Britos Cavagnaro 30:04

Yeah, absolutely. I think Riff provides a way for the teacher to see the thinking. It's like making the thinking visible, the learners' thinking visible. You can see the conversations, and then you can see what aspects stood out, what aspects were like being stuck for learners, they could also follow the evolution of students' reflections across different sections or activities. And we're currently implementing a feature where with a press of a button, the teacher can create a summary or a synthesis of all of the student's reflections. So you could very quickly see what's going on through the student reflections.

Kristina Hoeppner 30:42

Digital ethics, AI ethics, responsible AI are being discussed when we are using tools like ChatGPT, and since Riff is based on that, where does the data of the students go? Is it just the teacher who has access to it, or is that also being fed back to Open AI to train future models?

Leticia Britos Cavagnaro 31:03

That's a great question. I should say that the responsible development and the responsible deployment of a tool like Riff is very important to us. To go to your specific question, the data is not used for training the models. We use different large language models: GPT but also Claude, created by Anthropic, which is a very interesting language model because it has been trained differently than GPT. It has been trained with a constitution of ethical principles. But the student data is never used for training of the language models. The student data is only available to the teacher. 

Leticia Britos Cavagnaro 31:40

We have made some decisions in the development of Riff, for instance, well, it does have its own moderation module to detect specific problematic interactions that the teacher would be alerted to. We made some decisions, so for instance, not use it with young people under 18 because I think there's still a lot that we don't know that we don't know when it comes to generative conversational AI. So it's a little bit different when we're talking about generative AI when we're creating an image, when you're creating the image. And of course, that has other ethical [laughs] implications. But it's not the immediacy of the conversation. 

Leticia Britos Cavagnaro 32:20

So we wouldn't want to be in a situation where we discovered that an interaction of a young person who might not understand what they're interacting with and have a negative outcome in their development. We want to be very cautious in that and be cautious in how we're training and the instructions and the behaviours that we're creating that take into account all of those potential problematic interactions. 

Leticia Britos Cavagnaro 32:47

And as well, I think it's important that we make our pedagogy transparent so that we are transparent with our students that this is an AI tool that they're in charge, that they should never feel compelled to share anything that they're not comfortable sharing, that they can redirect a conversation always, and that they should treat it as a tool that is helping them so give the tool the information that is helpful for the tool to help them, but then ultimately, they are in charge.

Kristina Hoeppner 33:16

Thank you for being so transparent about where you're coming from and also what ethical considerations you have taken into account from the start because building that in at the beginning, while it takes quite a bit of thinking ahead What could the potential difficulties be, I think is really important so that it doesn't have to be bolted on later on because that oftentimes, it's way more difficult to do. 

Leticia Britos Cavagnaro 33:42

Absolutely. 

Kristina Hoeppner 33:43

Leticia you call the chatbot Riff, how did you select it?

Leticia Britos Cavagnaro 33:47

You can think it from the world of music when improviser is riffing on a song. It's the same that happens when you're thinking about your internal dialogue. When you're trying to figure something out, you might be riffing on ideas, but you could also say you might be riffing with someone going back and forth. 

Kristina Hoeppner 34:03

Thank you. That makes so much more sense now to know where the name comes from because I can see how that works out. Because as you said, the chatbot doesn't provide the answers. It asks you questions, and the fun thing that I noticed is I can even type in German, it's not quite there yet to answer in German. I did get the next question back in English. However, it did an internal translation, which was quite fascinating. Do you have any plans to incorporate other languages?

Leticia Britos Cavagnaro 34:34

Currently, there are language models that are good at multi language, namely GPT-4, for instance, but they're cost prohibitive at scale. They're cost prohibitive to use for many hundreds, thousands of chats. We might use GPT-4 for summarising, for instance and it does a better job than GPT-3.5, but 3.5 and Claude are very good for the purposes of guiding the behaviours of Riff, but are not as good as in responding in language. But as you said, you can do an internal translation and then respond in English. But I can see a near future in which it could be completely multi language.

Kristina Hoeppner 35:16

With whom have you already been using Riff so far, Leticia?

Leticia Britos Cavagnaro 35:20

Well, me personally, with basically all of the classes and learning experiences that I've done, including most recently, I did a workshop and keynote session with the Stanford summer session students, which is basically students all the way from graduate students, undergraduates from Stanford, visiting students, community college students, and high school students who are taking classes in the summer at Stanford. I've been using it and incorporating in my classes like that, but also, we offered access to other educators at the university level so that they can try it. We're working towards making that permanently available for educators to incorporate in their activities in person or in online courses, in learning management systems (LMS), etc.

Kristina Hoeppner 36:07

So that we can at some point then see an activity in the LMS that can be pulled into the course where you can set your reflecting questions or since we are talking about portfolios here on this podcast, that it might be incorporated into a journal activity.

Leticia Britos Cavagnaro 36:25

Exactly. You could have a conversation with Riff, then synthesise your key takeaway from that conversation, and that's what you capture. But the conversation is also something that the teacher can see. So gives it a little bit more transparency into the thinking process.

Kristina Hoeppner 36:43

You mentioned Borton's reflective framework of 'What? So what? Now what?' that it is nothing new. I think it came out in about 70s, I believe. So when I think back to one of the functionalities that we've had in Mahara through customization at some point, an organisation had said, well, we don't need to scaffold students' use of reflections. And they had come up with the idea of asking questions in a journal so that when a student selected a specific reflective task, they would have initial prompts. And of course, with 'What? So what? Now what?' there are heaps of websites that give you all of those starting questions that students can pick from. So what I find fascinating with Riff is that you don't need to give that slew of questions to the students, but you can have that starting question, be that a general one if students are left on their own to reflect on an inactivity, but where the teacher doesn't have insight into what that activity was, or something really specific to reflect on a particular project or class that you can then just select that starting question and riff off that and get those follow-on questions really tailored to the previous answers to get to that reflection more quickly, more easily, and in a more targeted way.

Leticia Britos Cavagnaro 38:02

Absolutely. The 'What? So what? Now what?' is just one scaffold. Riff will not ask you exactly that. It will ask you, as you said, targeted questions that have to do with exactly what you just answered. There's many other interesting frameworks for reflection. One that is a favourite at the d.school is 'I used to think and now I think'. It's honing into that moment when you change your mind, which is, I think, so powerful in an educational experience or in a learning experience when you revise an assumption that you had. So allowing students to hone in on those or direct them to hone in on those moments is a very powerful thing with just one simple prompt.

Kristina Hoeppner 38:47

Does Riff already support different reflection frameworks or is that something that you want to offer in the future?

Leticia Britos Cavagnaro 38:54

It takes quite a bit of testing or refining its behaviour. I could see creating different cousins that have different styles or that use different frameworks that could be interesting, for sure.

Kristina Hoeppner 39:06

Hopefully, our listeners will not just look at your book 'Experiments in reflection' once it comes out in November, but also take you up on the invitation to experiment themselves, either by themselves or with colleagues, maybe even have a study circle and try things out with his students. And then also give Riff a go because it's really, really easy to get started and see already the benefits even if you just use it on your own because you said that it helps you on your own to have that conversation with yourself and get started, but then also can use it in conjunction with others or have a conversation around those reflections. 

Kristina Hoeppner 39:44

That takes us now to our quick answer round, Leticia. Which words or short phrases do you use to describe portfolio work? We haven't really talked about portfolios much today but because of reflection is such a big part of portfolios, we kind of have been talking about portfolios.

Leticia Britos Cavagnaro 40:03

Exactly. And I have to say for me portfolios, because I come from the world also of design and design education, portfolios are such an integral part of our process and our learning and the ways our students, you know, we actually assess students to admit them into a programme, how students are presenting themselves in the marketplace, but I'm not by no means an expert in portfolios. But the way that I would describe portfolio work is making learning visible. That would be you know, one way at least.

Kristina Hoeppner 40:35

What tip do you then have for learning designers or instructors who create portfolio activities?

Leticia Britos Cavagnaro 40:42

To answer this question, I'm going to borrow from work that I've been developing for these past few years with two of my closest collaborators, Erica Estrada-Liou from the Academy of Innovation & Entrepreneurship in the University of Maryland, and Meenu Singh, also a lecturer at the d.school. We are the teaching team for our Teaching & Learning Studio, which is our programme for university educators. We have developed, as part of this workshop, a framework of pedagogical levers or levers. 

Leticia Britos Cavagnaro 41:13

Basically, the idea that there are all of these knobs or variables that you can play with, and you should play with, when you create a learning experience. So I think that really thinking about how many variables we can control as learning designers or instructors or creators have a learning experience that can include portfolios, but also more generally, such as the activities, what types, but also the sequence of activities. And there's so many counter-intuitive, sometimes non obvious, and sometimes the opposite of what you see [laughs] in most educational systems in terms of the types but also the sequence of activities that you create. 

Leticia Britos Cavagnaro 41:53

In that lever is also 'Where is reflection in your sequence and how is it embedded?' Assessment? What and how are we assessing? One big idea around that is you could think about a two by two kind of like four quadrants and one axis is thinking about, 'Are you assessing product or are you assessing process?' There's not one that is better than the other. Then the other axis, the Y axis, would be 'instructor led and learner led or student led'. Oftentimes, we tend to be in one quadrant, and I think we should expand the intuitive four quadrants and think about well, if we only assess product, then we're sending the message because assessment is basically sending a signal about what's important to the students, we're sending the message, all that matters is what you produce at the end, but that you are developing a process that you're being reflective on your process is not important. So we need to assess process as well. 

Leticia Britos Cavagnaro 42:50

Other levers - earlier, at the very beginning of our conversation - you mentioned space. How can you with portfolios, for instance, activate in context, out in the world learning so that students bring their own worlds into the portfolios that they create? Those are just some of the pedagogical levers, but really thinking about what can you control and oftentimes is way more than what we think we control.

Kristina Hoeppner 43:15

Last question for today. What advice do you have for portfolio authors, Leticia?

Leticia Britos Cavagnaro 43:21

Let your quirks shine through your work and your portfolios. So when you create a portfolio, even if it's for a class, even if it's been required by an educator, take that as an opportunity to really explore who you are, and who you want to become. Don't waste that opportunity. It's really about you as a learner.

Kristina Hoeppner 43:46

That's a fantastic finishing sentence there for our learners to not just do everything for the assessment and assume that they need to say what the teacher expects or think what the teacher expects, but to also bring in their own personality, bring in their culture, bring in the experiences because that's who they are as a learner. 

Kristina Hoeppner 44:09

Thank you so much for the conversation today, Leticia, giving us an insight into reflection, where you are coming from, where your research is, and also what you have developed over the last years, thinking of the book and now most recently also Riff, the chatbot.

Leticia Britos Cavagnaro 44:26

Thank you, Kristina. It's been such a pleasure.

Kristina Hoeppner 44:29

Now over to our listeners. What do you want to try in your own portfolio practice? This was 'Create. Share. Engage.' with Dr Leticia Britos Cavagnaro. Head to our website podcast.mahara.org where you can find links and the transcript for this episode. 

Kristina Hoeppner 44:43

This podcast is produced by Catalyst IT, and I'm your host, Kristina Hoeppner, Project Lead and Product Manager of the portfolio platform Mahara. Our next episode will air in two weeks. I hope you'll listen again and tell a colleague about it so they can subscribe. Until then create, share, and engage.