Thursday, September 25, 2008

September 25, 2008

This week I have been looking at some of the tools available for archiving websites. These include tools which address part of the entire archiving process like HTTrack (an offline browser that can be used to harvest site) and AONS (an "automatic obsolescence notification system" that checks file format registries against the files in your collection, notifying you when some are in danger of going defunct). There are also multi-function tools that can perform multiple tasks. The elusive Web Archived Workbench from OCLC is one of these, although there is very little OCLC-sponsored documentation regarding this software online. You're welcome to see my annotated list of these and other tools at the wiki.

The Internet Archive's "Archive-it" is another example of a multi-function tool. This is a fee-based service, although I couldn't find any specific pricing structure. If you subscribe to the service, you can use Archive-it to schedule and scope your harvests. Unlike the general Wayback Machine web crawls, Archive-it users can create collections of their web resources (the common complaint about the Wayback Machine is the access method, which is solely through full-text search or known-URL selection). The Internet Archive hosts and stores your data, and if you terminate your agreement it will keep the information, adding it into its general web collection. A number of university entities are taking advantage of the IA's services, including the Carolina Center for the Study of the Middle East and Muslim Civilizations here at UNC. Among our peers, the University of Southern California, the Latin American Information Network at UT-Austin, and the Stem Cell Research Project at UW-Madison are also IA partners. There is definitely an attraction to having an already developed and technologically accessible alternative, but there are also several possible downsides:
  • This is a for-fee service.
  • The IA crawl is still only a "snapshot." Accessing deep and dynamic information isn't part of this picture.
  • Although the IA has dispersed redundant storage, you are still placing the stewardship of your materials into the hands of another agency.
  • All IA materials are public. The site mentions that you can make materials private by contacting the IA for a special arrangement. There's no mention if selected materials can be made private, or if it's an all-or-nothing deal. Again, this is another area in which control will have to be handed over to an external agency.

In a sense, this boils down to the general debate over using your own solutions vs. outsourcing. Although the IA does give you autonomy over collecting and organizing, you're still beholden to their technological and business changes. Based on my general observations and investigations, it seems that more universities are choosing the IA option at this point in time. Whether or not they are exploring in-house alternatives as well is less clear to me.