Tuesday, March 3, 2009

March 3, 2009

I've made more headway on my poster, and I thought I'd share in more depth the software evaluation rubric I've come up with. As I mentioned last time, I've come up with three categories of criteria: Ease of Installation/Setup, Ease of Use, and Features. I started doing some research into software evaluation, to try to see if I could find a vetted rubric to use.

In my literature search, I discovered that software evaluation can be an extremely intense process, punctuated by algorithms and equations. I think this type of evaluation actually centers on the code. I also found some information on ISONORM 9241/10, described here. This came closer to what I was looking for, but centered too much on ergonomics and usability of the software dialog. It didn't cover some of the more subjective aspects I wanted to measure. More relevant to my task were books and articles talking about how to perform a needs assessment and then devise a rubric to suit your own requirements. The down side is the subjective nature of the rubric. But in the end, I decided that for the purposes of my fellowship, where no replicable results are needed, this would work best.

So here's the rubric I've developed. (It's on the wiki, so you'll need access to that before it will come up.) The idea is that each tool will be assessed based on the criteria in each of the 3 categories. The numbers that correspond to the criteria for that tool would be added up to a total. The lower the number, the less conducive it would be to use in UARMS at this point in time. I've written out some explanations for some of the criteria below. For instance, here are my definitions of the criteria in the "Available Documentation for Installation" scale:

None: No documentation available
Sparse: Highly technical and brief
Adequate: Easy to read, mimics common install guides, may be out of date
Robust: Up-to-date, includes troubleshooting section

The "Available Documentation for Use" is similar:

None: No documentation available
Sparse: Some online documentation, may be interspersed with installation information
Adequate: Dedicated user's manual
Robust: User's manual plus built-in help

Finally, the "Knowledge Required" section deserves some clarification:

Program Administrator: Requires a program administrator
UNIX Savvy Librarian: Knows basics of a UNIX system
Technology Savvy User: Feels comfortable doing basic troubleshooting on Windows and Mac platforms
Out of the Box: Feels comfortable doing out of the box installations

Again, I know these are subjective. But I've tried to make them as explicit as possible, and also to select the characteristics I feel have the most impact on whether or not a tool might be adopted in the UARMS setting.