Tuesday, March 24, 2009

March 24, 2009

What a lapse in posts! The good news is that I've been busy preparing for the DigCCurr conference and trying to wrap up installation of the last tool to try (Web Curator Tool). Installation and setup of WCT has proven to be the most onerous of all, and beyond my own technological skill. Two new(ish) computers later, though, and things seem closer. I'll keep you posted.

My poster is practically finished. I ended up doing some modification to the rubric I discussed in the last post. I decided to move some of the criteria on documentation and support to their own category, Documentation/Support, because of the importance of having good explanations for how to install and use these tools. In my experience, lack of documentation can be common with open source tools. But when those using the tools are new to the field or to that type of technology, it becomes all the more important.

I changed the order of criteria for Interface. After some more thought, it seemed to me that simply giving access to a GUI (graphical user interface) is less ideal that offering access to a GUI and to the command line. That category was revised to reflect as much.

I also added some additional features, including bandwidth allocation and the ability to make alterations to harvested materials.

I won't post the results here until after DigCCurr 2009, just to keep up the suspense. Being able to test and thoughtfully evaluate a range of tools is surely a luxury for most institutions. I'm glad I have been able to help UNC in this way.

Tuesday, March 3, 2009

March 3, 2009

I've made more headway on my poster, and I thought I'd share in more depth the software evaluation rubric I've come up with. As I mentioned last time, I've come up with three categories of criteria: Ease of Installation/Setup, Ease of Use, and Features. I started doing some research into software evaluation, to try to see if I could find a vetted rubric to use.

In my literature search, I discovered that software evaluation can be an extremely intense process, punctuated by algorithms and equations. I think this type of evaluation actually centers on the code. I also found some information on ISONORM 9241/10, described here. This came closer to what I was looking for, but centered too much on ergonomics and usability of the software dialog. It didn't cover some of the more subjective aspects I wanted to measure. More relevant to my task were books and articles talking about how to perform a needs assessment and then devise a rubric to suit your own requirements. The down side is the subjective nature of the rubric. But in the end, I decided that for the purposes of my fellowship, where no replicable results are needed, this would work best.

So here's the rubric I've developed. (It's on the wiki, so you'll need access to that before it will come up.) The idea is that each tool will be assessed based on the criteria in each of the 3 categories. The numbers that correspond to the criteria for that tool would be added up to a total. The lower the number, the less conducive it would be to use in UARMS at this point in time. I've written out some explanations for some of the criteria below. For instance, here are my definitions of the criteria in the "Available Documentation for Installation" scale:

None: No documentation available
Sparse: Highly technical and brief
Adequate: Easy to read, mimics common install guides, may be out of date
Robust: Up-to-date, includes troubleshooting section

The "Available Documentation for Use" is similar:

None: No documentation available
Sparse: Some online documentation, may be interspersed with installation information
Adequate: Dedicated user's manual
Robust: User's manual plus built-in help

Finally, the "Knowledge Required" section deserves some clarification:

Program Administrator: Requires a program administrator
UNIX Savvy Librarian: Knows basics of a UNIX system
Technology Savvy User: Feels comfortable doing basic troubleshooting on Windows and Mac platforms
Out of the Box: Feels comfortable doing out of the box installations

Again, I know these are subjective. But I've tried to make them as explicit as possible, and also to select the characteristics I feel have the most impact on whether or not a tool might be adopted in the UARMS setting.