Tool investigation continues! After contacting Paul Koerbin at the NLA and receiving a quick and informative response, I've learned that PANDAS is not being offered broadly as an open source solution for other institutions. This is primarily due to resource constraints and the fact that several other good open source tools are available. Paul suggested the Web Curator Tool (WCT) and also the Netarchive Suite (NAS), both of which were listed in my previous blog.
Looking at the possible implementation of these two tools, I've discovered to my dismay that I can't forge ahead on my own. System administrators will be required both for server installation and/or setting up a Linux box. It also makes me realize that, even if I had all of the technological knowhow to understand the installation requirements, someone with knowledge of the system in place here would still need to lend me their time and talents. This really highlights the fact that collaboration with IT is more and more necessary. I can see why librarians and archivists might get nervous at this dependence on an external department. It requires trust and patience.
Monday, September 29, 2008
Thursday, September 25, 2008
September 26, 2008
One of the ways I've come to categorize tools is by "accessibility for non-technical users." Of the web archiving tools I've looked at, I would rate them in the following way, from most accessible to least accessible.
- Archive-It (Internet Archive) (discussed in the previous post)
- WCT - Web Curator Tool (National Library of New Zealand + British Library)
- PANDAS - PANDORA Digital Archiving System (National Library of Australia)
- Web Archives Workbench (OCLC)
- Netarchive Suite (Danish Royal Library and State and University Library)
- IIPC Toolkit (a group of separate tools like Heritrix and DeepArc)
September 25, 2008
This week I have been looking at some of the tools available for archiving websites. These include tools which address part of the entire archiving process like HTTrack (an offline browser that can be used to harvest site) and AONS (an "automatic obsolescence notification system" that checks file format registries against the files in your collection, notifying you when some are in danger of going defunct). There are also multi-function tools that can perform multiple tasks. The elusive Web Archived Workbench from OCLC is one of these, although there is very little OCLC-sponsored documentation regarding this software online. You're welcome to see my annotated list of these and other tools at the wiki.
The Internet Archive's "Archive-it" is another example of a multi-function tool. This is a fee-based service, although I couldn't find any specific pricing structure. If you subscribe to the service, you can use Archive-it to schedule and scope your harvests. Unlike the general Wayback Machine web crawls, Archive-it users can create collections of their web resources (the common complaint about the Wayback Machine is the access method, which is solely through full-text search or known-URL selection). The Internet Archive hosts and stores your data, and if you terminate your agreement it will keep the information, adding it into its general web collection. A number of university entities are taking advantage of the IA's services, including the Carolina Center for the Study of the Middle East and Muslim Civilizations here at UNC. Among our peers, the University of Southern California, the Latin American Information Network at UT-Austin, and the Stem Cell Research Project at UW-Madison are also IA partners. There is definitely an attraction to having an already developed and technologically accessible alternative, but there are also several possible downsides:
The Internet Archive's "Archive-it" is another example of a multi-function tool. This is a fee-based service, although I couldn't find any specific pricing structure. If you subscribe to the service, you can use Archive-it to schedule and scope your harvests. Unlike the general Wayback Machine web crawls, Archive-it users can create collections of their web resources (the common complaint about the Wayback Machine is the access method, which is solely through full-text search or known-URL selection). The Internet Archive hosts and stores your data, and if you terminate your agreement it will keep the information, adding it into its general web collection. A number of university entities are taking advantage of the IA's services, including the Carolina Center for the Study of the Middle East and Muslim Civilizations here at UNC. Among our peers, the University of Southern California, the Latin American Information Network at UT-Austin, and the Stem Cell Research Project at UW-Madison are also IA partners. There is definitely an attraction to having an already developed and technologically accessible alternative, but there are also several possible downsides:
- This is a for-fee service.
- The IA crawl is still only a "snapshot." Accessing deep and dynamic information isn't part of this picture.
- Although the IA has dispersed redundant storage, you are still placing the stewardship of your materials into the hands of another agency.
- All IA materials are public. The site mentions that you can make materials private by contacting the IA for a special arrangement. There's no mention if selected materials can be made private, or if it's an all-or-nothing deal. Again, this is another area in which control will have to be handed over to an external agency.
In a sense, this boils down to the general debate over using your own solutions vs. outsourcing. Although the IA does give you autonomy over collecting and organizing, you're still beholden to their technological and business changes. Based on my general observations and investigations, it seems that more universities are choosing the IA option at this point in time. Whether or not they are exploring in-house alternatives as well is less clear to me.
Thursday, September 18, 2008
September 18, 2008
I've unearthed another university that, through partnerships, has been working on archiving websites. Although it is not considered a peer of UNC, given the very few universities that are far enough along with web archiving to have incorporated it into their online presence (and, by few, I mean one), I decided that the University of North Texas Libraries definitely deserves note.
I discovered their efforts because they're a member of the IIPC - the only university in the current list of North American members. Their rather morbidly named CyberCemetery "provides permanent public access to the websites and publications of defunct U.S. government agencies and commissions." ("Permanent" - very ambitious) In addition to searching the site, you can browse by agency, date of expiration or branch of government. The browsable links I clicked were broken due to a recent system change, but the search mechanism seems to work fine. Sites come up with a simple, small textual header of "Archive." It's a straightforward, easy to navigate system.
According to the IA's Web Archiving Blog, I found out that UNT will be archiving the end-of-term presidential websites, partnering with the following acronym parade: IA, CDL, the LoC, and the USGPO.
It's interesting to me that UNT hasn't made archives of their own website available, but is focusing on external government sites. UMich will have to be the closest model for archiving our own sites, for now.
I discovered their efforts because they're a member of the IIPC - the only university in the current list of North American members. Their rather morbidly named CyberCemetery "provides permanent public access to the websites and publications of defunct U.S. government agencies and commissions." ("Permanent" - very ambitious) In addition to searching the site, you can browse by agency, date of expiration or branch of government. The browsable links I clicked were broken due to a recent system change, but the search mechanism seems to work fine. Sites come up with a simple, small textual header of "Archive." It's a straightforward, easy to navigate system.
According to the IA's Web Archiving Blog, I found out that UNT will be archiving the end-of-term presidential websites, partnering with the following acronym parade: IA, CDL, the LoC, and the USGPO.
It's interesting to me that UNT hasn't made archives of their own website available, but is focusing on external government sites. UMich will have to be the closest model for archiving our own sites, for now.
Monday, September 15, 2008
September 15, 2008
Just when I thought we were behind...
I've begun looking at peer institutions' (see August 21 post below) websites to try to discern what others are doing in the way of website archiving. The answer: very very little. My judgment is based on slim pickings after simply trying to dig up archived websites somewhere on the university library or archives websites, through their catalogs or finding aids, or even getting a hint of them from a records retention and disposition schedule. Out of all of the peers, the only one that had any semblance of a web archive was the University of Michigan at Ann Arbor. They not only address archiving websites in their policies and procedures, but archived websites appear directly in their catalog. (Here's an example of a record for an archived website.)
So many of these sites don't even mention electronic records in any format, with retention guidelines still dictating how to pack boxes and shred paper. It's amazing. A number of sites had digital collections, but they all focused on digitized materials, not born digital. The occasional institutional repository (IDEALS at UI-UC, and eScholarship at the U of CA, to name two examples) tickled the edges of electronic material collection, but still didn't present anything comprehensive.
I'm alternately aghast and encouraged. There's a tad more information located on the wiki regarding what I found. I suspect some sort of brief email survey out to these institutions might turn up what (if anything) else they've considered in the way of electronic records, let alone archiving websites. As for finding a peer who's already established procedures and standards, other than UMich (applause for them), I will definitely have to cast a wider net.
I've begun looking at peer institutions' (see August 21 post below) websites to try to discern what others are doing in the way of website archiving. The answer: very very little. My judgment is based on slim pickings after simply trying to dig up archived websites somewhere on the university library or archives websites, through their catalogs or finding aids, or even getting a hint of them from a records retention and disposition schedule. Out of all of the peers, the only one that had any semblance of a web archive was the University of Michigan at Ann Arbor. They not only address archiving websites in their policies and procedures, but archived websites appear directly in their catalog. (Here's an example of a record for an archived website.)
So many of these sites don't even mention electronic records in any format, with retention guidelines still dictating how to pack boxes and shred paper. It's amazing. A number of sites had digital collections, but they all focused on digitized materials, not born digital. The occasional institutional repository (IDEALS at UI-UC, and eScholarship at the U of CA, to name two examples) tickled the edges of electronic material collection, but still didn't present anything comprehensive.
I'm alternately aghast and encouraged. There's a tad more information located on the wiki regarding what I found. I suspect some sort of brief email survey out to these institutions might turn up what (if anything) else they've considered in the way of electronic records, let alone archiving websites. As for finding a peer who's already established procedures and standards, other than UMich (applause for them), I will definitely have to cast a wider net.
Tuesday, September 2, 2008
September 2, 2008
"The combination of both automatic and manual collecting is the only way to ensure that, no matter what the technical wrapping, the most relevant Web content is found and archived" (Masanes, para 16).
Adrian Brown, in his book Archiving Websites (2006), discusses selective vs. unselective archiving. These two options carry with them many of the same selection and appraisal decisions long debated with paper records, like judgments regarding scope, evidential value, informational value, etc. Selective archiving, taking in some materials and not others, may lead to loss of contextual information. It involves more human interaction with the harvested materials, and consequently more resources. Unselective archiving, on the other hand, brings in (almost) everything, but ... it brings in (almost) everything. In the end, both still require some degree of human effort; the differences are in the amount of intervention and at which part of the process that intervention takes place.
Only having dipped my toes into the lake of theoretical literature as well as the practical examples of web archiving, I'm already seeing one trend: librarians and archivists seem to agree that unselective archiving, a la the Internet Archive, is not adequate in and of itself for long term preservation and access. The WebArchiv, as one example, does broad crawls of an entire domain as well as selective harvesting of sites targeted by curators. It also develops thematic collections. I get the impression that part of the desire to go beyond unselective archiving stems from our long history of adding value to collections through assignment of metadata and organization through cataloging. Another part is being shaped by the nature of the web. Though the material to be harvested is vast and changeable, a multi-pronged approach, which includes human quality control as well as scheduled broad harvesting, seems to be the only way to achieve the quality and quantity desired by many cultural institutions.
Coufal, L., & Žabička, P. (2007). Strategies and approaches to building thematic collections in WebArchiv. INFuture2007: Digital Information and Heritage, Zagreb.
Masanes, J. (202). Towards continuous web archiving: First results and an agenda for the future. D-Lib Magazine, 8(12) Retrieved from http://www.dlib.org/dlib/december02/masanes/12masanes.html
Adrian Brown, in his book Archiving Websites (2006), discusses selective vs. unselective archiving. These two options carry with them many of the same selection and appraisal decisions long debated with paper records, like judgments regarding scope, evidential value, informational value, etc. Selective archiving, taking in some materials and not others, may lead to loss of contextual information. It involves more human interaction with the harvested materials, and consequently more resources. Unselective archiving, on the other hand, brings in (almost) everything, but ... it brings in (almost) everything. In the end, both still require some degree of human effort; the differences are in the amount of intervention and at which part of the process that intervention takes place.
Only having dipped my toes into the lake of theoretical literature as well as the practical examples of web archiving, I'm already seeing one trend: librarians and archivists seem to agree that unselective archiving, a la the Internet Archive, is not adequate in and of itself for long term preservation and access. The WebArchiv, as one example, does broad crawls of an entire domain as well as selective harvesting of sites targeted by curators. It also develops thematic collections. I get the impression that part of the desire to go beyond unselective archiving stems from our long history of adding value to collections through assignment of metadata and organization through cataloging. Another part is being shaped by the nature of the web. Though the material to be harvested is vast and changeable, a multi-pronged approach, which includes human quality control as well as scheduled broad harvesting, seems to be the only way to achieve the quality and quantity desired by many cultural institutions.
Coufal, L., & Žabička, P. (2007). Strategies and approaches to building thematic collections in WebArchiv. INFuture2007: Digital Information and Heritage, Zagreb.
Masanes, J. (202). Towards continuous web archiving: First results and an agenda for the future. D-Lib Magazine, 8(12) Retrieved from http://www.dlib.org/dlib/december02/masanes/12masanes.html
Subscribe to:
Posts (Atom)