Usability Testing

One design problem, 6 students, and an eager first-time user experience tester. The result? Not what I expected…

A Design Problem

When we first began using Moodle, we chose the fan-favorite Essential theme. Although this was a great starting place, we soon realized how easy it is for students to get lost in the labyrinth of buttons and clicks. (I think it took something like 8 clicks for students to view their first course.) The design problem became this: how can we make Moodle easier to navigate and reduce the amount of “clicks” a user makes in the site? The idea of a Moodle redesign was born.

Many people choose Moodle because it is free and open source. However, there are hidden costs, one of which may include the site’s usability. Fortunately, we are in a position to work with an external company for the redesign. This company uses an agile approach to design, which means they make prototypes based on our ideas and feedback. We go through several phases like this, until the project is finished. This approach helps both parties develop a project without getting too far ahead. We can make changes as needed, without slowing down the entire build or going in a different direction than anticipated.  

Usability Testing FTW

One of the top priorities of an instructional designer is to get feedback from the people intended to use the product. In my case, the majority of people using Moodle are students. So my usability testing group needed to include our students. For simplicity’s sake, I will refer to this group as a “focus group” from now on.

To prepare for testing, I did my research. First, I wrote down stumbling blocks that I knew students and I struggled with in the original Moodle site. I wanted to make sure the old issues weren’t still a problem in the new site. I used these issues to write a short list of tasks for the focus group to complete. I made the tasks simple and straightforward, but there was still opportunity for them to explore the new site and provide general feedback.

I also thought about how I wanted to run the focus group. I am a huge fan of Steve Krug, whose passion and ability to explain usability testing is enviable. I cannot recommend Rocket Surgery Made Easy enough. I used his script to introduce the idea of usability testing to students. I think it made them feel comfortable with the process, and helped them realize I wasn’t testing their abilities. Mainly, I was interested in their experience with the new Moodle site! As it happened, I had six students in the focus group. Five of them had never even heard of Moodle before, whereas one of them had actually taken a couple classes in our old Moodle site. It could not have worked out better!

The Nitty Gritty

To run the focus group, I set up 6 tablets in a booked meeting room and loaded the Moodle site onto each. Thank goodness I brought my laptop, because one of the tablets died. Side note: Always prepare for tech failure!

I didn’t offer them too much direction, other than using parts of Krug’s script and providing context about how and why we use Moodle. I made it clear to students I was available to answer their questions, and that I would walk around the room observing their actions. When I noticed they got stuck, I asked them questions and didn’t immediately offer answers. I found this approach worked quite well; I wrote notes based on their informal feedback. At the end of the meeting, I asked if they had any general feedback. This strategy didn’t work very well. Although I had provided them with pens and paper, none of them had taken notes. After 25 minutes, they seemed ready to pack it in. Plus, it is always difficult to offer feedback in front of peers! So no one really said anything at the end.

Final Thoughts

Overall, the testing was a good experience, and it made me realize we don’t do this enough when it comes to projects and curriculum planning. And to be honest, preparing for the usability testing and setting up the focus group was not very difficult. This process could easily be replicated for many different deliverables. It’s a good reminder to get feedback from the end users at various points of development and design.

I also really enjoyed getting feedback from staff members. I sat down with 3 co-workers, because some of them will be taking Moodle courses. Plus, it’s always good to get a second opinion. They caught things that I and the focus group had missed, and they asked good questions to help me consider why we were adding or taking away certain elements from the Moodle site. Don’t forget to have a Marketing team member review the build before delivery!

Finally, this experience was a good reminder that while it’s important to get feedback from end users, it’s necessary to weigh those opinions with sound instructional design principles. There were a few times when I felt tongue-tied after receiving feedback. I felt like I had to respond right then. A better strategy is to collect feedback, ask questions when necessary, and then sit with the feedback.

For Next Time

  1. Screen record testers as they complete the tasks. This allows you to share feedback with developers or re-watch the feedback to make crucial decisions.
  2. Give members of the focus group a bit of privacy. I think some students felt pressure to “keep up” with others. What was meant to see the barriers of usability turned into a bit of a competition (as depicted in the “high jump” photo from the beginning of the post.)
  3. Make it easy for the focus group to share findings. This idea came to me right after the testing was over (of course), but next time I will provide each member of the group with 3 Post-It notes, so they can easily share a few ideas. This way, there’s no awkward, non-existent discussion at the end!

Digital Badges

It’s the beginning of 2019, it’s my first blog post, and I’m writing about…badges. Why is that a thing?!

Are Badges Making a Comeback?

Last year, I created a digital badge system for a Moodle course. My research began with the Mozilla Open Badge project, where I learned that a single badge can be chock-full of usable data. Open Badge is wonderful if students are collecting badges from different organizations and wish to showcase them on a site like LinkedIn. Actually, LinkedIn has its very own badge system and could be an interesting case study on the success of badges (especially in conjunction with

For my purposes, Mozilla Open Badge wasn’t necessary. Many learning platforms are developing their own badge systems and ways to display them, which I found to be the case for Moodle. Used within internal systems or courses, I think badges can be highly motivational and informative. It’s no secret that badges reached “oversaturation status,” the last couple years, and it’s unclear how they compare to other types of credentials. But when used intentionally, I do think they can be very helpful wayfinders!

My aim was to develop badges by assigning them to specific competencies and then using them to guide students through a Moodle course. In the past, online students have had difficulty recognizing when they have successfully completed a course. This is in part because a Moodle course can be a grouping of disparate items like lessons, textbooks, images, etc., or it can be a collection of modules that students complete to finish one course. As a student, it can be quite confusing!

We used the badges to help students self-manage their progress throughout the course. Badges signalled the completion of a course section, usually a lesson and quiz combo. After successfully getting a certain number of badges, students knew they had completed the course. They were also aware of the specific competencies they were gaining by receiving the badge. Another important way we used the badges was to align with the operator manual we created. The badges could help motivate and remind students to complete an online section, review the manual, and then proceed to the next section of the online course.

What I learned is that badges must always serve a purpose, otherwise they will be seen as a gimmick, or worse, a distraction. Being transparent about what it means to achieve a badge is crucial to the badge system’s success.

Some might think badges are “been there, done that,” but I would suggest reconsidering. As a learner, seeing a badge can be a gentle reminder you are on track, while adding spontaneity and fun to a course.

Check Out My Badges

I would suggest using a professional tool to create badges; I used Adobe Illustrator. Canva is a fantastic free tool if Adobe products are not an option. When I first developed the badges, I came up with a random colour scheme. I’m not really sure what I was thinking, because badges should be uniform and also adhere to the company’s brand guidelines.

Here is the before:

All 10 badges used the following colour scheme. I am really pleased with the after:

A Helpful Resource

This is the resource I used to create the badge in Illustrator (the orange and red badge template was my inspiration):–cms-19971

Final Thoughts

Most people who saw the badges were impressed by their look and function. Some even suggested they be included in every future online course build. To me, this has the opportunity for overkill. I think it depends on the type of course and whether or not badges can be used to support learning and motivation. Using badges shouldn’t begin the conversation, but be considered an option if and when the time is right.

For Next Time

Here is what I will keep in mind the next time I create a badge system…

  1. Add random badges to the course. Some of the soft skills we teach students are timeliness and productivity. Since these are habit forming, I think it would work well to surprise students with a badge after they handed in an assignment on time/early or spent a concentrated time on a task. This would reinforce positive behaviour and also add an element of curiosity. Perhaps it could be worked out so that when students accumulate a certain number of badges for soft skills, they move to “expert” status.
  2. Create one badge and upload it to Moodle. Check that the badge does not appear cut off, and that the text is easy to read. I made this mistake the first time, so I spent extra time fixing the badge dimensions and text size.

Thanks for reading!