It may seem like overkill to make a rubric to evaluate ed tech. But I found the process really helpful. Here’s my experience…
Rubrics are our Friends
In my master of educational technology studies, one of our tasks was to design a rubric to evaluate LMSs. We began by brainstorming common LMS features and then grouped them under the following categories: functional, technical, and administrative. Our goal was to create a rubric that assessed common features without pigeon-holing people using the rubric. This proved to be more difficult than we thought it would be! However, I think having a rubric is better than having no rubric at all. It’s nice to have a set of common criteria to evaluate different platforms.
Plus, rubrics can be used at multiple stages of evaluating a product:
1) A rubric helps people consider criteria they may not have thought of.
2) It opens up the dialogue and gets people talking about criteria that are most important for different audiences using the product.
3) It raises important questions for people to consider after purchasing a platform. I love the reflective aspect of rubrics: Did we achieve what we wanted? Did expectations match reality?
Our group was happy with the end result, but the rubric was still quite long and onerous to fill out.
Rubric on the Job
Fast forward a year, and I am working as an instructional designer who’s tasked with trialling new educational technologies and finding the best instructional fit. I also have to persuade others that certain technologies are worth the money. I decided it would be worthwhile to create a rubric similar to the one I created for LMSs. I could share this with people to make informed decisions, and also teach them how to fill it out for future decisions.
I had learned a few things from the first go-around. The administrative, functional, and technical headings made sense to our group when we were designing the rubric. But users found it a bit confusing, so I nixed that idea. I changed all the criteria when I created my own rubric for evaluating educational technologies. But it was still nice to review the original rubric and remember the conversations from all those group meetings. I also focused on simplifying the terms and using plain language to describe the criteria.
I created my first rubric in Excel. It was clunky looking and not very user-friendly. I had a couple people review this rubric. The best takeaways were 1) Put the information into a fillable PDF and 2) Consider using the SECTIONS framework to organize criteria.
The SECTIONS acronym is really helpful when thinking about how to select technology for learning. It stands for Students, Ease of Use, Cost, Time, Interaction, Organizational Issues, Networking, and Security and Privacy. It’s a useful acronym because basically anyone can understand it without having a background in ed tech or ID.
Once I started using the SECTIONS framework, everything started falling into place. I wrote all the criteria in a Word Doc and then turned it into a fillable PDF. I also replicated the rubric for people to reflect on how the product was living up to expectations. This way, they could compare their “before” rubric to “after” using the product. This was overkill and made the document incredibly dense. So I took that out, and now when I share it with people, I just tell them to consider filling out the rubric again after they’ve purchased it.
Here it is:
I’m happy with the end result because it’s both easy to read and fill out. It was also a good opportunity to learn how to create a fillable form. And again, I am reminded that having other people look at your work is so rewarding! It’s never been the case where this hasn’t paid off for me.
For Next Time
- Consider a weighted system to make the most important aspects worth more. Someone asked me the point of using the rating system. It’s best practice to compare at least 3 products before making a decision. The final number can help someone make a decision. I could see a weighted system returning a more precise number.
SECTIONS model image from Tony Bates: https://opentextbc.ca/teachinginadigitalage/chapter/9-1-models-for-media-selection/