While we're waiting for the test server to be set up, I've decided to start investigating policies, guidelines and workflows.
I've started two new wiki pages. The first is a place to aggregate mission statements, informing policies, and example web archiving policies. I've begun by collecting information from the University Library, University Archives and Records Management Service, the North Carolina Collection, North Carolina General Statutes (Public Records Law), and information from the UNC IT department on website policy. I've only found one bona fide web archiving collection policy thus far (thanks to the British Library), but I think more are out there. It's unfortunate that more libraries and archives don't put their collection policies online. I'm having to cobble together ideas based on articles I've read.
The other new page is a place to start formulating a workflow for web archiving. To help me make sure I'm accounting for all parts of the process, the workflow will be informed by the infamous OAIS reference model. Right now, what I've written is at a very high level, and I'll slowly add in more detail, especially once I can do some test harvests.
Friday, October 24, 2008
Monday, October 20, 2008
October 20, 2008
We're currently in a holding pattern here, waiting for access to a test server hosted by ibiblio. It looks like this server has sufficient functionality to begin testing at least one, if not more of the web archiving tools I'm interested in. This will be a great opportunity to see what these tools can do, and to determine which ones seem best for our situation.
In the meantime, I've been doing some more reading and adding to the wiki. I've also been performing a basic site analysis of several high-profile and highly desirable UNC websites. This analysis includes trying to get a sense of how deep and broad the sites are, what sort of file formats are included on the site, how much they link out, if their pages include dynamic content, and who might be interested in the site's information. I've gotten a number of sites that I think would be viable for test harvest.
I'm now looking at a list of North Carolina sites not affiliated with UNC that have been identified as high priority for harvesting. This presents more of a challenge for a number of reasons. First, many of these sites have more dynamic content, like calendars and blogs, and a greater number of links out to related resources. Another additional step that with these sites involves clearing copyright and/or obtaining permission to archive. Many web archives operate with a blanket "take down" policy (Internet Archive and MINERVA, among them), meaning they harvest first and then post a general statement notifying those who object to their harvest that they will remove the materials in question. But these policies are generally used when the harvesting is done on an unselective basis, because of the infeasibility of trying to identify and contact all of the copyright holders. Best practice for selectively harvesting sites really dictates contacting the different agencies and at least attempting a good faith effort at requesting permission to take their materials as well as to provide access to them. Some even suggest that it might heighten your archive's profile and increase its scope (Žabička, 2006, p. 3). Archives like PANDORA and WebArchiv both follow this protocol. Negotiating for harvest not only serves to avoid contentious encounters with webmasters or agency representatives, but also shows that you are attempting to be a responsible steward with your materials.
Žabička, P. (2006). Digital cultural heritage and cooperation of memory institutions. Paper presented at the 1st Colloquium of Library Information Employees of the V4+ Countries, Banska Bystrica, Slovakia.
In the meantime, I've been doing some more reading and adding to the wiki. I've also been performing a basic site analysis of several high-profile and highly desirable UNC websites. This analysis includes trying to get a sense of how deep and broad the sites are, what sort of file formats are included on the site, how much they link out, if their pages include dynamic content, and who might be interested in the site's information. I've gotten a number of sites that I think would be viable for test harvest.
I'm now looking at a list of North Carolina sites not affiliated with UNC that have been identified as high priority for harvesting. This presents more of a challenge for a number of reasons. First, many of these sites have more dynamic content, like calendars and blogs, and a greater number of links out to related resources. Another additional step that with these sites involves clearing copyright and/or obtaining permission to archive. Many web archives operate with a blanket "take down" policy (Internet Archive and MINERVA, among them), meaning they harvest first and then post a general statement notifying those who object to their harvest that they will remove the materials in question. But these policies are generally used when the harvesting is done on an unselective basis, because of the infeasibility of trying to identify and contact all of the copyright holders. Best practice for selectively harvesting sites really dictates contacting the different agencies and at least attempting a good faith effort at requesting permission to take their materials as well as to provide access to them. Some even suggest that it might heighten your archive's profile and increase its scope (Žabička, 2006, p. 3). Archives like PANDORA and WebArchiv both follow this protocol. Negotiating for harvest not only serves to avoid contentious encounters with webmasters or agency representatives, but also shows that you are attempting to be a responsible steward with your materials.
Žabička, P. (2006). Digital cultural heritage and cooperation of memory institutions. Paper presented at the 1st Colloquium of Library Information Employees of the V4+ Countries, Banska Bystrica, Slovakia.
Thursday, October 2, 2008
October 2, 2008
While reviewing the proceedings from the recent iPRES conference, I stumbled across a very interesting JISC-sponsored project: PoWR. First, the acronyms: JISC stands for the Joint Information Systems Committee, whos mission "is to provide world-class leadership in the innovative use of I[nformation and] C[ommunications] T[echnology] to support education and research" (JISC website). PoWR stands for Preservation of Web Resources. It is a short-term project funded to "organise workshops and produce a handbook that specifically addresses digital preservation issues that are relevant to the UK H[igher] Education] /F[urther] E[ducation] web management community" (from the PoWR website). The main method of communication for the PoWR project is through the blog linked above. Many of the posts are directly relevant to the challenges of archiving websites at an institution of higher education, including sections on records management, case studies, and web 2.0. I also listened to Brian Kelly's presentation, which gives a great overview of the project in 13 minutes. Check out the case study at the University of Bath when they used an innovative approach to try to "motivate the techies" by drawing parallels between their print prospectus (or catalog, for us Yanks) and the website.
Even better, PoWR has released a draft of their handbook, which they will be finalizing after they receive feedback. I haven't read through the entire thing yet, but I've already gotten to the section on making a case for web archiving at your institution. In scanning the handbook's contents, it looks like a list of the same issues I've been considering during the past weeks. It's quite exciting and I recommend a peek.
Even better, PoWR has released a draft of their handbook, which they will be finalizing after they receive feedback. I haven't read through the entire thing yet, but I've already gotten to the section on making a case for web archiving at your institution. In scanning the handbook's contents, it looks like a list of the same issues I've been considering during the past weeks. It's quite exciting and I recommend a peek.
Monday, September 29, 2008
September 29, 2008
Tool investigation continues! After contacting Paul Koerbin at the NLA and receiving a quick and informative response, I've learned that PANDAS is not being offered broadly as an open source solution for other institutions. This is primarily due to resource constraints and the fact that several other good open source tools are available. Paul suggested the Web Curator Tool (WCT) and also the Netarchive Suite (NAS), both of which were listed in my previous blog.
Looking at the possible implementation of these two tools, I've discovered to my dismay that I can't forge ahead on my own. System administrators will be required both for server installation and/or setting up a Linux box. It also makes me realize that, even if I had all of the technological knowhow to understand the installation requirements, someone with knowledge of the system in place here would still need to lend me their time and talents. This really highlights the fact that collaboration with IT is more and more necessary. I can see why librarians and archivists might get nervous at this dependence on an external department. It requires trust and patience.
Looking at the possible implementation of these two tools, I've discovered to my dismay that I can't forge ahead on my own. System administrators will be required both for server installation and/or setting up a Linux box. It also makes me realize that, even if I had all of the technological knowhow to understand the installation requirements, someone with knowledge of the system in place here would still need to lend me their time and talents. This really highlights the fact that collaboration with IT is more and more necessary. I can see why librarians and archivists might get nervous at this dependence on an external department. It requires trust and patience.
Thursday, September 25, 2008
September 26, 2008
One of the ways I've come to categorize tools is by "accessibility for non-technical users." Of the web archiving tools I've looked at, I would rate them in the following way, from most accessible to least accessible.
- Archive-It (Internet Archive) (discussed in the previous post)
- WCT - Web Curator Tool (National Library of New Zealand + British Library)
- PANDAS - PANDORA Digital Archiving System (National Library of Australia)
- Web Archives Workbench (OCLC)
- Netarchive Suite (Danish Royal Library and State and University Library)
- IIPC Toolkit (a group of separate tools like Heritrix and DeepArc)
September 25, 2008
This week I have been looking at some of the tools available for archiving websites. These include tools which address part of the entire archiving process like HTTrack (an offline browser that can be used to harvest site) and AONS (an "automatic obsolescence notification system" that checks file format registries against the files in your collection, notifying you when some are in danger of going defunct). There are also multi-function tools that can perform multiple tasks. The elusive Web Archived Workbench from OCLC is one of these, although there is very little OCLC-sponsored documentation regarding this software online. You're welcome to see my annotated list of these and other tools at the wiki.
The Internet Archive's "Archive-it" is another example of a multi-function tool. This is a fee-based service, although I couldn't find any specific pricing structure. If you subscribe to the service, you can use Archive-it to schedule and scope your harvests. Unlike the general Wayback Machine web crawls, Archive-it users can create collections of their web resources (the common complaint about the Wayback Machine is the access method, which is solely through full-text search or known-URL selection). The Internet Archive hosts and stores your data, and if you terminate your agreement it will keep the information, adding it into its general web collection. A number of university entities are taking advantage of the IA's services, including the Carolina Center for the Study of the Middle East and Muslim Civilizations here at UNC. Among our peers, the University of Southern California, the Latin American Information Network at UT-Austin, and the Stem Cell Research Project at UW-Madison are also IA partners. There is definitely an attraction to having an already developed and technologically accessible alternative, but there are also several possible downsides:
The Internet Archive's "Archive-it" is another example of a multi-function tool. This is a fee-based service, although I couldn't find any specific pricing structure. If you subscribe to the service, you can use Archive-it to schedule and scope your harvests. Unlike the general Wayback Machine web crawls, Archive-it users can create collections of their web resources (the common complaint about the Wayback Machine is the access method, which is solely through full-text search or known-URL selection). The Internet Archive hosts and stores your data, and if you terminate your agreement it will keep the information, adding it into its general web collection. A number of university entities are taking advantage of the IA's services, including the Carolina Center for the Study of the Middle East and Muslim Civilizations here at UNC. Among our peers, the University of Southern California, the Latin American Information Network at UT-Austin, and the Stem Cell Research Project at UW-Madison are also IA partners. There is definitely an attraction to having an already developed and technologically accessible alternative, but there are also several possible downsides:
- This is a for-fee service.
- The IA crawl is still only a "snapshot." Accessing deep and dynamic information isn't part of this picture.
- Although the IA has dispersed redundant storage, you are still placing the stewardship of your materials into the hands of another agency.
- All IA materials are public. The site mentions that you can make materials private by contacting the IA for a special arrangement. There's no mention if selected materials can be made private, or if it's an all-or-nothing deal. Again, this is another area in which control will have to be handed over to an external agency.
In a sense, this boils down to the general debate over using your own solutions vs. outsourcing. Although the IA does give you autonomy over collecting and organizing, you're still beholden to their technological and business changes. Based on my general observations and investigations, it seems that more universities are choosing the IA option at this point in time. Whether or not they are exploring in-house alternatives as well is less clear to me.
Thursday, September 18, 2008
September 18, 2008
I've unearthed another university that, through partnerships, has been working on archiving websites. Although it is not considered a peer of UNC, given the very few universities that are far enough along with web archiving to have incorporated it into their online presence (and, by few, I mean one), I decided that the University of North Texas Libraries definitely deserves note.
I discovered their efforts because they're a member of the IIPC - the only university in the current list of North American members. Their rather morbidly named CyberCemetery "provides permanent public access to the websites and publications of defunct U.S. government agencies and commissions." ("Permanent" - very ambitious) In addition to searching the site, you can browse by agency, date of expiration or branch of government. The browsable links I clicked were broken due to a recent system change, but the search mechanism seems to work fine. Sites come up with a simple, small textual header of "Archive." It's a straightforward, easy to navigate system.
According to the IA's Web Archiving Blog, I found out that UNT will be archiving the end-of-term presidential websites, partnering with the following acronym parade: IA, CDL, the LoC, and the USGPO.
It's interesting to me that UNT hasn't made archives of their own website available, but is focusing on external government sites. UMich will have to be the closest model for archiving our own sites, for now.
I discovered their efforts because they're a member of the IIPC - the only university in the current list of North American members. Their rather morbidly named CyberCemetery "provides permanent public access to the websites and publications of defunct U.S. government agencies and commissions." ("Permanent" - very ambitious) In addition to searching the site, you can browse by agency, date of expiration or branch of government. The browsable links I clicked were broken due to a recent system change, but the search mechanism seems to work fine. Sites come up with a simple, small textual header of "Archive." It's a straightforward, easy to navigate system.
According to the IA's Web Archiving Blog, I found out that UNT will be archiving the end-of-term presidential websites, partnering with the following acronym parade: IA, CDL, the LoC, and the USGPO.
It's interesting to me that UNT hasn't made archives of their own website available, but is focusing on external government sites. UMich will have to be the closest model for archiving our own sites, for now.
Subscribe to:
Posts (Atom)