Tuesday, November 25, 2008
November 25, 2008
I got the impression that many who spoke up on the list were focusing on government documents. This reflects what I've found in the literature - most published accounts of web archiving in the US also seem to target government sites. This makes sense, considering (1) those sites have, what is considered to be, vetted information (2) have information libraries often collect without heavy selection procedures (3) are perceived as more stable than others.
Of those who responded, the most often mentioned service or tool was the Internet Archive's Archive-it. Others include HTTrack, Context Miner, Fedora, OCLC's Web Archives Workbench, and del.icio.us. This forum was also where I learned about Zoetrope (see my November 20 post).
Manual vs. automated harvesting also seemed to be equally represented. Most still feel that curating the materials is the valuable service that librarians can provide over and above a broad approach like that of the Internet Archive.
Although many stated they weren't actually harvesting web content yet, some of those who were mentioned that they have looked for copyright statements on pages and also sought permission to archive. I think that, because so many expressed their newness to this topic, and because of the newness of web archiving in general, this topic hasn't yet gained a critical mass of experience or attention.
Another really interesting thing was the number of respondents who said they're creating MARC records for freely available website resources, without actually harvesting those resources. This really astonishes me. Most cited storage and technological limitations as barriers to harvesting. I understand that the librarians are interested in getting their users as much information as possible, and hope to do so through catalog searches, but with the ephemeral nature of web content, not to mention unstable URLs, I would imagine these links rot quickly and it would be incredibly demanding to maintain. I also wonder if a list on an archivists' listserv would garner different reactions and methods. More interesting was the claim that they only enter "stable" sites. Sort of an oxymoron. From reading between the lines, I gather that these are sites they feel have a lower possibility for changing URL, but I'm not sure. If there's one thing I've learned in my time at SILS (there's more than one thing, but I'll keep it to one for now), it's that digital objects are anything but stable and that it's best to never think of them as such.
On a related note, many also talked about collecting e-books and e-prints versus the entire website. This must be a difficult decision when you're looking at a document that you formerly collected in print, which is now available online. Many still seem to look for the online equivalent of books or pamphlets for pulling into their repositories. Drawing this distinction allows repositories to draw boundaries in the boundary-less web, but it also may mean they're missing valuable information that has been converted to a "non-book" format.
Again, these are just my thoughts and interpretations of the conversations from the forum and shouldn't be taken as a complete synopsis!
Thursday, November 20, 2008
November 20, 2008
http://uwnews.washington.edu/ni/article.asp?articleID=45255
Zoetrope looks like a fascinating way to actually access harvested web material. Developed by University of Washington and Adobe, it allows you to compare a page over time, multiple pages over time, or even sections of multiple pages (using "lenses") over time. It will even graph changes in a numerical entity (think product prices or temperatures). Although it harvests sites VERY often, I would hope it would still be useful for a site harvested only monthly or bimonthly. They mention eventually using the Internet Archive as a source for harvested sites, which may mean it can read and display .arc files. Even better - they plan to make it available for free.
Monday, November 10, 2008
November 10, 2008
This is a really hard admission for me to make. I've been struggling to try to wrap my head around installing some web archiving utilities, like Netarchive Suite and WERA. Thanks to the generous help of ibiblio.org folks, the first tool is now up and running (albeit in a trial version) on our test server and I've completed my first harvest. I'm really excited about that major hurdle. Now, I just need to get a viewer for the .arc files created by Netarchive Suite.
I've had to admit that, despite my open mind and eagerness to tackle these things myself, I simply do not have the technological background to strike out on my own. This hit home when I was sitting down with an ibiblio.org staff member, who deftly and quickly took care of an issue I had been struggling with for over a week. I've been trying to figure out why this bothers me so much. Perhaps its because I've never really met with something within the scope of a job that was so far beyond my skill level that I knew I couldn't just bootstrap myself up. Maybe it's because everyone seems to need IT support, in low supply, and this project will undoubtedly fall to the bottom of the stack. Or maybe it's knowing that, without installation of these tools, I'm stuck. I can't go tackle a different set of boxes.
My frustration in part comes from the fact that installation instructions for the full version of Netarchive Suite, or the Web Curator Tool, or even WERA, are all written for a system administrator. Even the user manuals often state "your system administrator will do X..." The set of skills that these documents assume is already beyond what I have. It's a matter of encountering one term, only to be met with multiple other terms that I'm unfamiliar with.
This has been an eye-opening experience. Despite the fact that I feel pretty comfortable and familiar with technology, there is an entire realm of technological depth that supports a lot of the open source tools many librarians and archivists are advocating. To use these tools, I cannot operate without system administrator support. Period.
Friday, October 24, 2008
October 24, 2008
I've started two new wiki pages. The first is a place to aggregate mission statements, informing policies, and example web archiving policies. I've begun by collecting information from the University Library, University Archives and Records Management Service, the North Carolina Collection, North Carolina General Statutes (Public Records Law), and information from the UNC IT department on website policy. I've only found one bona fide web archiving collection policy thus far (thanks to the British Library), but I think more are out there. It's unfortunate that more libraries and archives don't put their collection policies online. I'm having to cobble together ideas based on articles I've read.
The other new page is a place to start formulating a workflow for web archiving. To help me make sure I'm accounting for all parts of the process, the workflow will be informed by the infamous OAIS reference model. Right now, what I've written is at a very high level, and I'll slowly add in more detail, especially once I can do some test harvests.
Monday, October 20, 2008
October 20, 2008
In the meantime, I've been doing some more reading and adding to the wiki. I've also been performing a basic site analysis of several high-profile and highly desirable UNC websites. This analysis includes trying to get a sense of how deep and broad the sites are, what sort of file formats are included on the site, how much they link out, if their pages include dynamic content, and who might be interested in the site's information. I've gotten a number of sites that I think would be viable for test harvest.
I'm now looking at a list of North Carolina sites not affiliated with UNC that have been identified as high priority for harvesting. This presents more of a challenge for a number of reasons. First, many of these sites have more dynamic content, like calendars and blogs, and a greater number of links out to related resources. Another additional step that with these sites involves clearing copyright and/or obtaining permission to archive. Many web archives operate with a blanket "take down" policy (Internet Archive and MINERVA, among them), meaning they harvest first and then post a general statement notifying those who object to their harvest that they will remove the materials in question. But these policies are generally used when the harvesting is done on an unselective basis, because of the infeasibility of trying to identify and contact all of the copyright holders. Best practice for selectively harvesting sites really dictates contacting the different agencies and at least attempting a good faith effort at requesting permission to take their materials as well as to provide access to them. Some even suggest that it might heighten your archive's profile and increase its scope (Žabička, 2006, p. 3). Archives like PANDORA and WebArchiv both follow this protocol. Negotiating for harvest not only serves to avoid contentious encounters with webmasters or agency representatives, but also shows that you are attempting to be a responsible steward with your materials.
Žabička, P. (2006). Digital cultural heritage and cooperation of memory institutions. Paper presented at the 1st Colloquium of Library Information Employees of the V4+ Countries, Banska Bystrica, Slovakia.
Thursday, October 2, 2008
October 2, 2008
Even better, PoWR has released a draft of their handbook, which they will be finalizing after they receive feedback. I haven't read through the entire thing yet, but I've already gotten to the section on making a case for web archiving at your institution. In scanning the handbook's contents, it looks like a list of the same issues I've been considering during the past weeks. It's quite exciting and I recommend a peek.
Monday, September 29, 2008
September 29, 2008
Looking at the possible implementation of these two tools, I've discovered to my dismay that I can't forge ahead on my own. System administrators will be required both for server installation and/or setting up a Linux box. It also makes me realize that, even if I had all of the technological knowhow to understand the installation requirements, someone with knowledge of the system in place here would still need to lend me their time and talents. This really highlights the fact that collaboration with IT is more and more necessary. I can see why librarians and archivists might get nervous at this dependence on an external department. It requires trust and patience.
Thursday, September 25, 2008
September 26, 2008
- Archive-It (Internet Archive) (discussed in the previous post)
- WCT - Web Curator Tool (National Library of New Zealand + British Library)
- PANDAS - PANDORA Digital Archiving System (National Library of Australia)
- Web Archives Workbench (OCLC)
- Netarchive Suite (Danish Royal Library and State and University Library)
- IIPC Toolkit (a group of separate tools like Heritrix and DeepArc)
September 25, 2008
The Internet Archive's "Archive-it" is another example of a multi-function tool. This is a fee-based service, although I couldn't find any specific pricing structure. If you subscribe to the service, you can use Archive-it to schedule and scope your harvests. Unlike the general Wayback Machine web crawls, Archive-it users can create collections of their web resources (the common complaint about the Wayback Machine is the access method, which is solely through full-text search or known-URL selection). The Internet Archive hosts and stores your data, and if you terminate your agreement it will keep the information, adding it into its general web collection. A number of university entities are taking advantage of the IA's services, including the Carolina Center for the Study of the Middle East and Muslim Civilizations here at UNC. Among our peers, the University of Southern California, the Latin American Information Network at UT-Austin, and the Stem Cell Research Project at UW-Madison are also IA partners. There is definitely an attraction to having an already developed and technologically accessible alternative, but there are also several possible downsides:
- This is a for-fee service.
- The IA crawl is still only a "snapshot." Accessing deep and dynamic information isn't part of this picture.
- Although the IA has dispersed redundant storage, you are still placing the stewardship of your materials into the hands of another agency.
- All IA materials are public. The site mentions that you can make materials private by contacting the IA for a special arrangement. There's no mention if selected materials can be made private, or if it's an all-or-nothing deal. Again, this is another area in which control will have to be handed over to an external agency.
In a sense, this boils down to the general debate over using your own solutions vs. outsourcing. Although the IA does give you autonomy over collecting and organizing, you're still beholden to their technological and business changes. Based on my general observations and investigations, it seems that more universities are choosing the IA option at this point in time. Whether or not they are exploring in-house alternatives as well is less clear to me.
Thursday, September 18, 2008
September 18, 2008
I discovered their efforts because they're a member of the IIPC - the only university in the current list of North American members. Their rather morbidly named CyberCemetery "provides permanent public access to the websites and publications of defunct U.S. government agencies and commissions." ("Permanent" - very ambitious) In addition to searching the site, you can browse by agency, date of expiration or branch of government. The browsable links I clicked were broken due to a recent system change, but the search mechanism seems to work fine. Sites come up with a simple, small textual header of "Archive." It's a straightforward, easy to navigate system.
According to the IA's Web Archiving Blog, I found out that UNT will be archiving the end-of-term presidential websites, partnering with the following acronym parade: IA, CDL, the LoC, and the USGPO.
It's interesting to me that UNT hasn't made archives of their own website available, but is focusing on external government sites. UMich will have to be the closest model for archiving our own sites, for now.
Monday, September 15, 2008
September 15, 2008
I've begun looking at peer institutions' (see August 21 post below) websites to try to discern what others are doing in the way of website archiving. The answer: very very little. My judgment is based on slim pickings after simply trying to dig up archived websites somewhere on the university library or archives websites, through their catalogs or finding aids, or even getting a hint of them from a records retention and disposition schedule. Out of all of the peers, the only one that had any semblance of a web archive was the University of Michigan at Ann Arbor. They not only address archiving websites in their policies and procedures, but archived websites appear directly in their catalog. (Here's an example of a record for an archived website.)
So many of these sites don't even mention electronic records in any format, with retention guidelines still dictating how to pack boxes and shred paper. It's amazing. A number of sites had digital collections, but they all focused on digitized materials, not born digital. The occasional institutional repository (IDEALS at UI-UC, and eScholarship at the U of CA, to name two examples) tickled the edges of electronic material collection, but still didn't present anything comprehensive.
I'm alternately aghast and encouraged. There's a tad more information located on the wiki regarding what I found. I suspect some sort of brief email survey out to these institutions might turn up what (if anything) else they've considered in the way of electronic records, let alone archiving websites. As for finding a peer who's already established procedures and standards, other than UMich (applause for them), I will definitely have to cast a wider net.
Tuesday, September 2, 2008
September 2, 2008
Adrian Brown, in his book Archiving Websites (2006), discusses selective vs. unselective archiving. These two options carry with them many of the same selection and appraisal decisions long debated with paper records, like judgments regarding scope, evidential value, informational value, etc. Selective archiving, taking in some materials and not others, may lead to loss of contextual information. It involves more human interaction with the harvested materials, and consequently more resources. Unselective archiving, on the other hand, brings in (almost) everything, but ... it brings in (almost) everything. In the end, both still require some degree of human effort; the differences are in the amount of intervention and at which part of the process that intervention takes place.
Only having dipped my toes into the lake of theoretical literature as well as the practical examples of web archiving, I'm already seeing one trend: librarians and archivists seem to agree that unselective archiving, a la the Internet Archive, is not adequate in and of itself for long term preservation and access. The WebArchiv, as one example, does broad crawls of an entire domain as well as selective harvesting of sites targeted by curators. It also develops thematic collections. I get the impression that part of the desire to go beyond unselective archiving stems from our long history of adding value to collections through assignment of metadata and organization through cataloging. Another part is being shaped by the nature of the web. Though the material to be harvested is vast and changeable, a multi-pronged approach, which includes human quality control as well as scheduled broad harvesting, seems to be the only way to achieve the quality and quantity desired by many cultural institutions.
Coufal, L., & Žabička, P. (2007). Strategies and approaches to building thematic collections in WebArchiv. INFuture2007: Digital Information and Heritage, Zagreb.
Masanes, J. (202). Towards continuous web archiving: First results and an agenda for the future. D-Lib Magazine, 8(12) Retrieved from http://www.dlib.org/dlib/december02/masanes/12masanes.html
Monday, August 25, 2008
August 25, 2008
Although I haven't looked at many yet, I've been looking at one particular site that seems to do a great job of presenting archived websites in a user-friendly and straightforward way. It's Our Digital Island by the State Library of Tasmania. Adrian Brown mentions the site in Archiving Websites as one which provides excellent browsing functionality. Their selection policy is also available online.
Thursday, August 21, 2008
August 21, 2008
- Keep in mind: DigCCurr submission? Deadline September 30
- Started poking around the web, and came up with a great site on Web Archiving from PADI. Very comprehensive. Some portions (case studies) haven't been updated in a while, though.
- Spoke with John. We decided to do separate blogs so that we could keep them more informal. We'll set up a pbwiki page where we can post more formal drafts and products, as they're developed.
Peer Institutions
Duke University
Emory University
Johns Hopkins University
The University of Texas at Austin
University of California - Berkeley
University of California - Los Angeles
University of Florida
University of Illinois at Urbana-Champaign
University of Michigan-Ann Arbor
University of Pennsylvania
University of Pittsburgh-Main Campus
University of Southern California
University of Virginia-Main Campus
University of Washington-Seattle Campus
University of Wisconsin-Madison
August 19, 2008
Thoughts for general timeline and how to proceed:
Research
Goals:
- What is the current idea of a responsible web archiving strategy, including preservation, metadata, access?
- Work to refine scope - how much do we want to keep/is feasible to keep
- Identify what has been tried, is being tried
- Compare these with UNC's situation (resources/collections)
- Review current policy and procedure
- Begin blog
- Meet with Carolyn Hank
- Review notes from Elec Recs class
- Find related books, articles, websites, theses, etc.
- Identify peer institutions: what have they been (thinking of) doing thus far? Possibly send out short survey?
- Locate any suggested standards
- Develop list of resources with annotations
- Check with Cal about possible avenues
- Investigate technology: iRODS, Archivist's workbench, Archivit
Suggestions and Policy
- Identify target sites
- Get list of contacts, especially those who have been amenable to rec mgmt activity in the past
- What kind of metadata will we collect? records/series?
- How often should we harvest?
- What about intranets?
Trials/Practice