FOCUS AREA 1 SESSION |Problems of practice| October 19, 2019

For attendees and facilitators please access the full meeting minutes below.

In our second Problems of Practice session, Dr. Brian Beatty, Associate Vice President for Academic Affairs Operations at San Francisco State University (SFSU), has asked our community of experts to work through some ideas around the following challenge SFSU is currently thinking about. As a review, our 4-step process looks like this:

1. "Client" presents challenge

2. "Consultants" ask clarifying questions of client

3. "Consultants" workshop a solution

4. "Client" reflects on solution

THE WICKED PROBLEM OF PRACTICE

"What active learning strategies and technologies support large class sizes both online and in F2F classrooms? How can these strategies be designed to increase student engagement and performance, especially as it relates to closing achievement gaps associated with underrepresented minority students?"

OUR CLIENT

Brian: I’m dealing with engaging our students better and providing better resources for faculty who want to use different pedagogies, especially around active learning. Also, I'm developing faculty development opportunities and structures that support their movement towards a better pedagogy and using newer technologies that we're able to advise. One of our challenges is our relatively large class size. Our typical face-to-face classroom is usually capped with an enrollment around 49. For whatever reason 49, not 50. Our average class size is only 34 students, but anytime we want to do anything on campus, especially around turning classrooms into an active learning classroom or giving faculty more space for better approaches, we always run into this pushback saying that we can't reduce our seat size anywhere because that's really critical for, the business side of the university. In our large lecture classes, which are often general education courses, they typically run around 130 to 140 students.

As we know, and the literature kind of bears out, ideal active learning class size for one faculty without a lot of support is a lot less than 150. It's a lot less than 50, usually it's probably more in the 25 range. It's a challenge for us and we find the similar challenge, in our online classes. I'm looking at a lot of our online data and we're not really talking about the same kinds of technologies being used, but there are combinations of pedagogy and technology using in the online environment that can be also be used to support online learning or support active learning in an online environment. Those class sizes also are a relatively large.

Our average class size online is around 42. We have about 250 online classes, not a really large portfolio for a university of our size, but it's an ongoing challenge. We're actually just now starting to look at the data around student performance in these classes with an eye for what are our achievement gaps, specifically in our online courses, and it's pretty revealing that not surprisingly, the achievement gap is worse in online classes overall in general. Part of my job is to help figure out a path towards solving that for the, for the campus. We're interested in face-to-face pedagogies, but we're also interested in our online pedagogies and we have a lot of faculty who crossover between the two, and students who crossover between the two. It's really supporting the act of learning throughout the entire curriculum that our challenge is, with larger class sizes, and with an eye towards closing achievement gaps. The one that just is top of mind for me right now is the URM (under represented minority student) achievement gap, although there are likely other achievement gaps among student groups that we'd want to address as well.

CLARIFYING QUESTIONS

QUESTION #1

Kiho Kim (American University): I have a question for Brian. Do you know whether or not the achievement gap amongst the online students carries over when they are in face-to-face classes? So the question is, are students selecting into online classes that say something about that demographic that is different than the general population who tend to take face-to-face classes?

Brian: Yeah, that's a good question. What we are, I don't know the answer to that question, but that's a worthy question to explore because one of the things we're looking at is, on the student side of self selecting in. What we understand right now is that a lot of students, especially at the undergraduate lower division level, are choosing online courses for convenience. That may say something about their readiness to succeed in those online courses. We're also seeing that there's a self selection process on campus in most departments for faculty teaching online. We're also looking at the aspects of which faculty are choosing to teach online, what are their perspectives on online learning and active learning, et cetera. Do they follow that in any particular way? Is that an important factor? Clearly one of the things we know is that it's a complex issue, right? There's no single one answer to this, but it's gonna be a combination of factors that lead to a successful solution.

QUESTION #2

Don Merritt (University of Central Florida): Brian. I've got a question. Is there any faculty development required or offered for faculty before they start teaching online?

Brian: Yeah, that's another great question. We have a lot of faculty development that's offered and probably in 20 percent of the departments or colleges it is required. Most colleges and departments leave it up to the faculty to make that choice in practice. Whether or not the words say that on the the policies or not. We do have a lot available and actually it's one of the things we're tracking as well, but, we have about probably five percent or less of our classes are actually certified high quality for online. They've gone through a quality matters or a local quality process, but about 30 percent of the faculty teach online have gone through faculty professional development, but that's been almost all volunteering.

Dr. Butch Herod (Houston Community College): Brian, this is Butch. I have a question, it’s multipart actually. What kind of additional support mechanisms do you have in place to support the online learning environment to more effectively emulate which you would have in a classroom environment? So for instance, everything from augmented chat rooms to live sessions, Empire State has assigned a faculty mentor that actually follows each student. They're a different sized institution, but they assign a faculty mentor to follow each student and they stay with that student throughout the entirety of their four year experience until they graduate and they’ve seen a bump in success rates. I just wonder what kind of additional steps that you're taking in that area and especially how that relates to your underrepresented populations?

Brian: One of the things we have going on here is we have no fully online degree programs. All of our online courses are taught in regular or traditional kind of residential programs. There is some development going on for degree programs, but, in great part because there are no fully online programs, there are no true, fully online students who the university assumes may never come to campus. That also means we really haven't developed a typical online student support services effort, like we would expect to see once our online programs are actually launched.

What we end up relying on is the local offices and their ability to serve students at a distance compared to requiring students to come in to see a counselor, sign papers, etc. Same thing for the faculty. There are some departments that won't even let faculty who teach all online just have online office hours, will still require the faculty to have on campus office hours because they know the students are all regionally located. I guess the real answer to your question is there's not a lot there. There really is no specific, coordinated support services for the students in online classes. Alongside that is also with the faculty, we give faculty the same essentially faculty development services and options as we do all of our other faculty without distinguishing between fully online faculty or, our traditional face-to-face faculty.

Don: What about blended learning opportunities? Do you do any of them?

Brian: We do a lot of the blended learning opportunities, and this is one of the reasons why I'm so interested in helping solve the online course problem in part because a lot of our face-to-face classes use a significant amount of online activity as well. And it may be a traditional hybrid. You're in class, you're out of class, you're in class, you're out of class approach, but more across the board, 80 percent or more of our classes are using the online systems to augment what's going on in the classroom. Whether or not it's a reassigning time or not. That all that to say that, it could be that one, those kinds of blended classes or if the grade differentiation is not as bad in those classes, it could be that they're adding, they're including something like the relationship component, for example, that the online classes themselves are not doing well or something else. What we want to make sure is that even the online components of the blended program, which is pretty significant, aren't disadvantaging students in any particular way and that we're able to better support faculty when they're making choices to do things online versus in the class.

WORKSHOPPING A SOLUTION

WHY IS THIS CHALLENGE IMPORTANT TO AN INSTITUTION’S ACTIVE LEARNING INITIATIVE? DO YOU HAVE AN EXAMPLE?

Kiho: Let me give you a perspective from a small urban institution, a small liberal arts urban institution. This is somewhat unrelated to the the academic mission, but really we are constrained by space. We are ensconced in a neighborhood where they are really belligerent when it comes to expanding the footprint of the university, both in terms of the undergraduate enrollment and so we have to maximize our ability to work with students and so online is one opportunity. So the limitations on number of classrooms that we have really points to online learning where students could be either on or off campus but not having to utilize on campus space. This is an important part of our strategy to maximize our capacity to work with students, both undergraduate and graduate students. That's one the reasons why we think this is important.

"From a small urban institutional perspective, we are really constrained by space and online learning is one opportunity to maximize our ability to work with students."

Patrice Prusko (facilitator): Anyone else looking at online or blended as part of that strategy? I know that space is a common issue. I'm wondering if anyone else might be looking at or already have implemented a similar strategies.

Don: So we do here at UCF. I know we're big, and a lot of people remind us of that, but the challenge we have is space as well. We do not have enough physical space to put all of the students that we have and in fact, about 40 percent of our student credit hours are generated online. We have some programs that are entirely offered online and if it weren't for that online component of what we do, we would be probably two thirds the size that we are now. Because we do so much online we've been very intentional over the past 15 to 20 years about not only faculty development but ensuring expectations among the students when they come in, they know what they're getting into, they understand that the quality level is there. In some cases we often hear that the online classes are harder than the face-to-face classes. When we look at student success outcomes as measured by grade, our blended courses are up at the top, but only a few points behind them are face-to-face and entirely online courses. Both of those two being pretty much neck and neck with each other with grades as a success measure.

"when we look at student success outcomes as measured by grade, our blended courses are up at the top, but only a few points behind are F2F and entirely online courses."

Yanling Sun: This is Yanling from Montclair State University. We are facing a similar issue Don just mentioned regarding physical space. This year at Montclair the Provost’s office is starting to encourage large classrooms and large electronic classrooms as well. The partial reason is about space concerns and that also raises some concerns about learner outcomes concerns as well. It's quite a challenge to run 200-seat classes and so those classes are not blended or hybrid, they are actually face-to-face classes. I'm working with several faculty members to provide pedagogical recommendations to those classes. Like one of particular recommendation related to putting active learning in place. How to make instead of just lecturing, provide a lecture on how to make the learning spaces more active. How do you engage your students? A lot of times actually these are conducted online. The entire class is divided into different groups, to after class or before class discussion, or group work or project work, things like that. The discussion space or collaboration space and the faculty feel frustrated managing those classes. So it make me also think, it sounds to me it’s not quite effective to run an entire 200 student class in face-to-face the entire time. Could it better to run blended or hybrid formats for those large lecture classes? So I'm unclear where we're working with multiple such a type of classes. Hopefully for next year we can find some recommendation. Those type of classes, not only pedagogical perspective and also formats is the better face to face or hybrid.

"It sounds to me it's not quite effective to run an entire 200-student class in face-to-face the entire time. could it be better to run blended or hybrid formats for those large lecture classes?"

Patrice: Thank you. I'm wondering, I know sometimes when we think of active learning, we immediately think about physical learning space. How might we think about active learning when the learning space is a fully online classroom. Is that a challenge that your universities have been looking at?

Yanling: At Montclair, active learning activities or strategies have been pretty well implemented in online courses. I would say for online program courses, we do have several online, fully online programs, all of the courses are designed by our designers, through the collaboration with subject, faculty members, and during the design process we make sure we implemented the learning activities and strategies from the beginning to the end. We do also have faculty development programs as well. It's not required, we’re a union campus. They wanted to make it required but we're not there yet. However, all of the faculty members who are hired to teach online program courses are strongly recommended to take our faculty development programs. With that said, the online learning initiatives or strategies are implemented better in our online programs courses and assembled into the individual online courses. At Montclair it is not a required that all the individual courses that are part of those traditional programs, not in the fully online programs, that are not required to be designed by us. But an individual faculty member can actually take advantage of our services, have their courses either we designed their courses and we'll make sure those run new strategies to be implemented. We have our own pedagogical model for online courses. We have our own online course template as well. We also have course review process which is, with the faculty members, a strong designers and also academic, a representative from their own department or program.

Patrice: Well thank you. Does anyone else want to share if they have a strategy with fully online courses and active learning or your thoughts on why the challenge is, is it important at your university?

Mike Goudzwaard (Dartmouth College): This is Mike at Dartmouth. I would add to the space challenge, not just the amount of space but locations since we have a rural campus. I think that holds true for any institution that is trying to reach students and learners that are not physically located at their campus for whatever reason. I think one reason that it's important is that students will be working in these blended environments and maybe I'll go meta for a moment and take our group as an example. We do some work together, when we're at conferences together, but we're often working remotely and I think there's an intentionality, through this activity to have it be an active meeting strategy so that we're all getting input and Brian is getting multiple perspectives from the group.

Kim Pulford Westemeier (American): Yeah, that's similar to what I was going to say in that the online environment has several unique advantages that can really push the active learning into a whole other level. So an example would be a separating the participants of a class into different groups and so it feels more personalized and it's also less overwhelming. It feels like they're not getting lost because a lot of these classes ended up being these foundational classes where it's their first semester, second semester in school, and so like there's an aspect of student retention in here too, but I think that even having a blended or flipped in our model allows for more interaction between students and they are able to contribute more of their knowledge to the class as well.

What might success look like?

Patrice: Moving onto our next question, what might success look like? So if we were able to solve this challenge for Brian, what might success look like? Some of the things that we talked about and the challenges were space issues. We might ask, does anybody have any thoughts on how we might define success from the standpoint of solving the space issue?

Katie Kassof (American): I know that it's not so much of a problem now, but a few years ago a lot of faculty were using virtual computing options in the classrooms and there wasn't enough bandwidth to make it possible, let alone trying to do other collaborative like using online collaborative software. Our IT department has actually worked very hard to beef up the infrastructure and now like every classroom has its own Wifi router or hotspot. So they have a lot better connectivity. They can do these active things. They can use wireless projection and stuff, but that was an infrastructure piece that had to be worked out, network engineering to be able to have these things in the classrooms and have them work more reliably.

Patrice: Thank you, Katie. If we think about some of the things we discussed about online learning and talking about increasing access for students, and some of the issues related to student drop-off or students not completing. If were to incorporate active learning into online courses to help with some of those things, what would be important? What would be important to you as you're measuring success? if you were to incorporate some type of active learning intervention into an online or blended course, what are some of the things you might think about if you were collecting data or, trying to come up with a way to measure success?

Kiho: This is Kiho from American University. For us, I think retention is. I mean, one of the things that AU prides itself is the idea that faculty and students have very strong touchpoints and connections and that is reduced when students are taking online classes at increasing rates. We want to assure that pushing students off into the online space, does affect the retention rate.

"We want to assure that pushing students off into the online spaces does affect the retention rate."

Butch: I want to second that. Of course, at my institution, retention and ultimately completion are ongoing goals that we struggle with because we lose a number of our students. The online learning was particularly brutal in the initial phases where we had huge failure rates in terms of completion. We've worked to improve those, but I mean, the importance of this, shouldn't be underestimated, especially for institutions like mine. You don't mind me storytelling just for a moment. The last online class I taught, I asked the students, why are you taking this online as opposed to, basically showing up at one of our 23 different campus locations across the city, Houston area. The first six people that answered that questions said the following, "I am a single mother and the only way that I can attend college and do better for my family essentially is that if I am able to take this course in this fashion," and they were all from underrepresented populations. So as necessity, this dictates that we have to be able to be successful in online learning environments, active learning techniques and active learning practices are a lever that we can use to help accelerate and enhance the success rates of these students because they're going to be more fully engaged. So, that’s what I got.

Patrice: Thank you. Has anyone incorporated online courses for their residential students are either from a sense of, I'm thinking about like a sense of wellbeing for students. One way to measure success might be if we were to offer some online courses, would that give students more options? And we have a lot of students who maybe are working a job so they can afford to go to school or might be an athlete. How might we measure the impact just on student wellbeing, and also I was thinking about graduation rates. So for example, if it's a class that is fills up very quickly, if were to offer the fully online courses that might enable students to graduate on time or sooner because they can get the courses they need, especially for transfer students. If nobody is thinking about student wellbeing from that standpoint is anyone else measuring success by retention rates, persistence of underrepresented students or just access and equity.

Don: Well, we're certainly looking at all of those things. Our online students and blended learning students tend to stay on board longer, better than others, better retention there. Our freshman retention rate just crossed 90 percent this past year. We're pretty pleased with that and looking at some of the programs we've got related to online and making sure that we've got access. Yeah we're, I mean, everything we do is geared towards retention, a better representation, etc, whether it be face to face or online.

"Our online students and blended learning students tend to stay onboard longer, better than others, with better retention rates."

Patrice: Before we bring Brian back into the conversation, does anyone else want to share anything, an example at your school or a wondering, maybe something that you're thinking that this discussion has made you start to think about?

Don: I've got something I'd like to share that actually one of our engineering faculty is really keen on. He's and I've got something I can show here in a second too. His challenge is that he teaches sometimes in large classrooms, 300-400 students, sometimes it's small classrooms, but he's really keen on getting them to work together in groups. With engineering students, that isn't always an easy thing to socially engineer. He has found a solution that actually works really well and we've been looking at it for awhile and kind of wondering what other applications it might have . Is anyone familiar with Etherpad? Okay. It's basically an online note pad system that allows multiple people to be working on a document at the same time. It's a lot like google docs, but it comes out of Mozilla. I'm pretty sure it's the Mozilla project. Our faculty member started incorporating this into his classes sometime ago.

Don: Okay. Let's see if I remember how to do this. He explains how he uses Etherpad here really quickly, but essentially what he does is create multiple of these little groups and then has the students break into groups and start doing problem solving together through the text. One of the great things about it is they might be sitting next to each other in that large classroom, or they might not, but they're still working together in a group. He and his teaching assistants are monitoring these because they're also part of these groups. They're watching each one of these conversations and he can jump in at anytime and give suggestions. Like for instance, here, are they taking care, are they using the correct units as they're doing their equations. He can see how the conversation flows, etc. He's found it extremely useful and the more that he uses it, the better improvements he has seen on student grades later. Obviously, you've got a record here of what they've done, how well they've done it. At the end of it all they have everyone has everyone come out and someone, the person, the group that got the answer correct first comes up to the front of the class and kind of explains how they got it and goes through their thought process. It really kind of gets the students working together collaboratively, but it also gets them to engage with the rest of the group later. He's worked it out so that, if I recall correctly, and it's been awhile since I've talked to him, but, there's always a concern some people are going to work better than others in the group. He got some pushback when he first started doing this from the students who were the better students, hey, I'm going to be carrying these other folks, this and that. Each group, elects or votes on, an MVP if you will, and that person gets a bonus point or two for the project so that gives them some incentive to all work towards it. If there is one who is doing more work, they get recognized for that too. It occurred to us that this works particularly well in some of the large classrooms and he's got video of them doing this kind of group work and you can see sometimes they're sitting together. Sometimes there's one person spread out, depending on how the room works. There's no reason you couldn't do something like this synchronously in an online environment either. So, just some things to think about. Maybe the tools that we're using in our physical spaces would translate to our online virtual spaces if we just think differently about how we're asking them to come together and work together. Now if the course is supposed to be asynchronous, and in the state of Florida by law, no more than I think it's 20 percent of the content can be synchronous if it's an online course. That doesn't mean the students can't work together synchronously on problem solving. So, I mean, there may be some creative ways to use this. The group can show how they work together, the faculty member or TA's can see how they did that group work, and then find some way to do some kind of call out or sharing towards the end too. So it's just an idea. He's very energetic and excited about what he's done with this and I thought it was something that I would bring up to this group in thinking about this problem too.

REFLECTION + FINAL INSIGHTS

Brian'S REFLECTION

Well, thank you. I appreciate all of the comments and the suggestions and the idea sharing. Let me go through a few things that I noted. Just like you space is an issue here. I apologize for the background noise here, that's our local waste collection group going by, they'll be gone soon. We have some expectations for enrollment growth and we have no space for them obviously. One of the things we'll be needing to do is to do a better job with this space we have. It includes pressure for online courses, although it's not an overt, explicit pressure that I'm aware of coming from the top. It's more of a colleges and departments feeling, if we're going to add to our enrollment, we're going to have to use more online courses. What we want to do is make sure that when we do that, or as we do that, we're doing that in the best possible way. Don made some mention about grades being lower, or slightly lower, in the fully online courses similar to face-to-face courses compared to the blended courses and I think that's consistent with most of the literature in the field. We're finding that to be the same thing, that in general, our online students have a slightly lower GPA and they're slightly higher gaps. When we look at the differences between grades in our courses taught with faculty who have gone through certification or professional development, we don't see consistent differences in those metrics. Which is starting to lead me to the belief that just looking at the grades and the distribution of grades and the grade based retention is clearly not the metric we need to be looking. However the challenge with it, because what we really want to do is look at student experience and faculty experience. Are students learning more, are they more engaged in the class, do they feel better about the class? All those kinds of things in the long run that should have an impact on retention, that should have an impact on grades. Yet we find that a lot of our courses that go through certification process actually become more rigorous. One of the things we hear back from students, or the faculty sometimes, this was a lot harder than my friends said it would be, because it had changed. It used to be, I don't know, listen to a lecture or watch a video and take a quiz. Now there's all this stuff you have to do. From the designer's perspective, much better course, students are learning a lot more. Yet in courses that they may not value that the same way as the faculty or an external reviewer might. They have a challenge with, and oftentimes grades are slightly lower. The metrics of just looking at grades to look at quality and the impact of online learning is they're not sufficient. As a little aside in our system, we're working with 11 other campuses, addressing a student body of potentially a couple hundred thousand students, and looking at the great differences in online courses, but also looking at doing some student surveys and faculty surveys to assess that next level of what's the quality, what does quality really mean in this kind of environment. So we've got that.

Brian: Let's see. A couple of other things I noted down was this idea of, I think we all have examples on our campuses where some faculty have done this really well. So they've gotten some really good active learning approaches.

• The example of the Etherpad approach I thought was pretty interesting. That's really interesting, especially in that face-to-face classroom using an online tool in that way.
• I've been kind of searching for a higher level like design principles for student success in these very large classes that are leveraging active learning pedagogies. I don't know if that kind of a meta approach is out there yet. I see lots of those in online learning and face to face learning, but they're almost always on the kind of the more traditional, smaller class sizes, smaller being less than 50 students. If you know of anything like that, any work that's been done in that area, please let me know. I'm thinking maybe there's a dissertation out there too that might be useful.
• The comment about online students, I think in Houston at the community college, about students who are taking the online course because it's the only way they can possibly get to the opportunity. I think that one of the things that tells me as a university and as a university structure here in the United States is that I think we have to be very purposeful and intentional about engaging these students knowing the challenges that they're coming from. If we're like an open access institution, if you meet basic minimums and you are in our region, you're a part of our community. We have to be very intentional about serving them well and serving them just as well as our face to face students. I think that is a bit of a cultural shift on campus. Even a campus like mine, the kind where the bread and butter of our mission is equity and social justice. As we develop these approaches, I think there's an opportunity for the university to actually be more intentional about this in our online programs, or our blended programs, or wherever the students are actually getting to us.

Kiho: You were talking about evaluating students. I'm assuming that students do end of semester evaluations of their classes on the fact they did. Do you incorporate specific questions for online classes?

Brian: That's a great question, but the answer is no. Unfortunately we have six standard university questions that are asked of all courses, at all levels, at all sizes. Any additional questions, it can be added by departments or colleges, but they're not used in our university wide data and reported. Interestingly though I think our senate is actually opening up that process this year to revise it and that is certainly something that should be part of the conversation as different kinds of questions sets, especially for different delivery modes or other kinds of things that would be relevant. Do you do things differently there?

Kiho: We do have a set of questions all faculty ask and then we are asked to provide additional four questions and typically they are related to unit objectives and so we have a set of objectives for your science department for example, and each of the classes has a particular set of outcomes and then they are turned into questions such as, did you have an understanding of how policy and science interact ? So we add those in at the time when the SET or the student evaluation of teaching are being formulated.

Brian: Do you have any specific questions for like an online only courses.

Kiho: I don't know, Kim might know that.

Kim: I don’t know. It could be that they do elect to have a few that are added to the student evaluations as part of being a unique, classroom setting. I can check on that if you'd like and let you know if we do.

Brian: Yeah, I think it'd be useful if anyone has sets of questions that they use for differential kinds of classes, whether it's online or a active learning or hybrid classes. I think that'd be interesting to hear as well because we know that the questions that students are asked at the end of the semester, sometimes will help drive faculty behavior, at least at an initial stage of getting them interested in doing something differently. Because for some of our faculty, they're coming back every year. It kind of depends in some places pretty heavily on their student evaluation scores.

The lack of "here's what we use" and here's what it tells us. That tells me there's a conference presentation out there, a good panel discussion for active learning in large classes, including maybe hybrid and online classes, or maybe it's two different conversations. There's there's an opportunity there.

Grace: Definitely. I was going to say Entangled had recently going back to what, Butch had given us a comment in some of your additional questions on learners that are underrepresented Entangled just did a report on parent learners, and I can send it out to the group after this, but it talks specifically about some of the challenges and how online learning can help bridge that gap. I can send that out to the group.

Patrice: I was just going to mention at University of Wisconsin, Milwaukee, you may know Tanya Joosten. They developed a toolkit called the DETA toolkit and I can also embrace and share the link in there. She's devised some different types of questions that you can ask to look at some of the things that were talking about. For example, if you want to look specifically at course design or student experience, different aspects of the course, so that might be a helpful place, Brian or others to start with as you're thinking about what questions you might want to add to a survey. When I was at Cornell, we did redesign one of our large, it was a large 700 student class and I worked with the faculty member to take that course and redesigned for a fully online course and we used some of those questions as part of the survey the students were going to take, at the end of the course. Before anybody asks me how it turned out, I completed that project just as I was leaving cornell. So I don't know.

Kiho: I think one of the challenges for teaching in big classes is how to integrate technology. Everything from multiple screens showing different things, and so we have some of our, one of our big rooms has two separate screens or multiple screens, but they all show the same thing. To project different things at different times is something that I'm interested in, but also thinking more broadly about what sorts of technologies like Google Docs or Ether Pad, what sorts of technologies can be easily integrated into large classrooms to create those small group environments working collectively at the same place. Those are the things that I look for when I teach some of the bigger classes and we don't have that big of a class. The biggest class I’ve taught is 110 students. Those are pretty rare on our campus.

Katie: Kiho, can I just really quickly say, not related to this group at all, but anywhere where we have multiple screens, you can show different things on the screens. People just don't know how to, but you've always been able to. But that's actually brings up a good point is that we've put a lot of additional technology into spaces that people can't figure out how to use even when we try to make it intuitive. That's constantly something we're trying to address so that people can embrace it and move forward.

Grace: What kind of training do you do for the new technology implementation or the new technology that you're putting into the rooms?

Katie: Sometimes when we have new, really complicated rooms open up, I'll email the faculty who are going to be teaching in that room that semester. But I can be emailing over 100 faculty and four of them will come. As far as training, it's really difficult. A lot of our training happens like on the fly if you call for help, and then there's not a lot of time to go into things and brainstorm about how you could utilize this technology in creative ways.

Grace: It's more reactive at that point.

Katie: Exactly, that's just simply because, before classes start nobody's on campus so it's hard for people to get there and then when classes start we don't have space to offer training in. They're all being used. That's like that's a huge problem for us.

PARKED: QUESTIONS WE STILL HAVE

• Technology training: best practices and how you get faculty to engage in training?
• Lots of questions continue to surround how to scale active learning within large classrooms as well as how to measure improvement within these large online courses.
• How do you translate space connected to this online courses to better support them?

SPECIAL THANKS TO DR. Brian beatty FOR This month'S WICKED PROBLEM.

We'll be taking off next month's PROBLEMS OF PRACTICE for the holidays and will reconvene in january 2019. DETAILS WILL FOLLOW VIA EMAIL.

Questions or comments? Please reach out to Grace at grace@entangledstudios.org or Patrice at patrice@entangledstudios.org.