The American Library Association Association for Library Collections and Technical Services recently held a 3 day email forum on web archiving. I was clued in to this forum by one of the archivists here, and I'm glad I got to lurk a bit. Although the conversation was a little disjointed, and dominated by several respondents, it was interesting and enlightening. I thought I'd put down some of my thoughts here.
I got the impression that many who spoke up on the list were focusing on government documents. This reflects what I've found in the literature - most published accounts of web archiving in the US also seem to target government sites. This makes sense, considering (1) those sites have, what is considered to be, vetted information (2) have information libraries often collect without heavy selection procedures (3) are perceived as more stable than others.
Of those who responded, the most often mentioned service or tool was the Internet Archive's Archive-it. Others include HTTrack, Context Miner, Fedora, OCLC's Web Archives Workbench, and del.icio.us. This forum was also where I learned about Zoetrope (see my November 20 post).
Manual vs. automated harvesting also seemed to be equally represented. Most still feel that curating the materials is the valuable service that librarians can provide over and above a broad approach like that of the Internet Archive.
Although many stated they weren't actually harvesting web content yet, some of those who were mentioned that they have looked for copyright statements on pages and also sought permission to archive. I think that, because so many expressed their newness to this topic, and because of the newness of web archiving in general, this topic hasn't yet gained a critical mass of experience or attention.
Another really interesting thing was the number of respondents who said they're creating MARC records for freely available website resources, without actually harvesting those resources. This really astonishes me. Most cited storage and technological limitations as barriers to harvesting. I understand that the librarians are interested in getting their users as much information as possible, and hope to do so through catalog searches, but with the ephemeral nature of web content, not to mention unstable URLs, I would imagine these links rot quickly and it would be incredibly demanding to maintain. I also wonder if a list on an archivists' listserv would garner different reactions and methods. More interesting was the claim that they only enter "stable" sites. Sort of an oxymoron. From reading between the lines, I gather that these are sites they feel have a lower possibility for changing URL, but I'm not sure. If there's one thing I've learned in my time at SILS (there's more than one thing, but I'll keep it to one for now), it's that digital objects are anything but stable and that it's best to never think of them as such.
On a related note, many also talked about collecting e-books and e-prints versus the entire website. This must be a difficult decision when you're looking at a document that you formerly collected in print, which is now available online. Many still seem to look for the online equivalent of books or pamphlets for pulling into their repositories. Drawing this distinction allows repositories to draw boundaries in the boundary-less web, but it also may mean they're missing valuable information that has been converted to a "non-book" format.
Again, these are just my thoughts and interpretations of the conversations from the forum and shouldn't be taken as a complete synopsis!
Tuesday, November 25, 2008
Thursday, November 20, 2008
November 20, 2008
Talk about a neat and shiny tool. Julien Masanes forwarded this link to a web archiving listserv I've subscribed to. It is definitely worth the time to watch the video:
http://uwnews.washington.edu/ni/article.asp?articleID=45255
Zoetrope looks like a fascinating way to actually access harvested web material. Developed by University of Washington and Adobe, it allows you to compare a page over time, multiple pages over time, or even sections of multiple pages (using "lenses") over time. It will even graph changes in a numerical entity (think product prices or temperatures). Although it harvests sites VERY often, I would hope it would still be useful for a site harvested only monthly or bimonthly. They mention eventually using the Internet Archive as a source for harvested sites, which may mean it can read and display .arc files. Even better - they plan to make it available for free.
http://uwnews.washington.edu/ni/article.asp?articleID=45255
Zoetrope looks like a fascinating way to actually access harvested web material. Developed by University of Washington and Adobe, it allows you to compare a page over time, multiple pages over time, or even sections of multiple pages (using "lenses") over time. It will even graph changes in a numerical entity (think product prices or temperatures). Although it harvests sites VERY often, I would hope it would still be useful for a site harvested only monthly or bimonthly. They mention eventually using the Internet Archive as a source for harvested sites, which may mean it can read and display .arc files. Even better - they plan to make it available for free.
Monday, November 10, 2008
November 10, 2008
I am not a system administrator.
This is a really hard admission for me to make. I've been struggling to try to wrap my head around installing some web archiving utilities, like Netarchive Suite and WERA. Thanks to the generous help of ibiblio.org folks, the first tool is now up and running (albeit in a trial version) on our test server and I've completed my first harvest. I'm really excited about that major hurdle. Now, I just need to get a viewer for the .arc files created by Netarchive Suite.
I've had to admit that, despite my open mind and eagerness to tackle these things myself, I simply do not have the technological background to strike out on my own. This hit home when I was sitting down with an ibiblio.org staff member, who deftly and quickly took care of an issue I had been struggling with for over a week. I've been trying to figure out why this bothers me so much. Perhaps its because I've never really met with something within the scope of a job that was so far beyond my skill level that I knew I couldn't just bootstrap myself up. Maybe it's because everyone seems to need IT support, in low supply, and this project will undoubtedly fall to the bottom of the stack. Or maybe it's knowing that, without installation of these tools, I'm stuck. I can't go tackle a different set of boxes.
My frustration in part comes from the fact that installation instructions for the full version of Netarchive Suite, or the Web Curator Tool, or even WERA, are all written for a system administrator. Even the user manuals often state "your system administrator will do X..." The set of skills that these documents assume is already beyond what I have. It's a matter of encountering one term, only to be met with multiple other terms that I'm unfamiliar with.
This has been an eye-opening experience. Despite the fact that I feel pretty comfortable and familiar with technology, there is an entire realm of technological depth that supports a lot of the open source tools many librarians and archivists are advocating. To use these tools, I cannot operate without system administrator support. Period.
This is a really hard admission for me to make. I've been struggling to try to wrap my head around installing some web archiving utilities, like Netarchive Suite and WERA. Thanks to the generous help of ibiblio.org folks, the first tool is now up and running (albeit in a trial version) on our test server and I've completed my first harvest. I'm really excited about that major hurdle. Now, I just need to get a viewer for the .arc files created by Netarchive Suite.
I've had to admit that, despite my open mind and eagerness to tackle these things myself, I simply do not have the technological background to strike out on my own. This hit home when I was sitting down with an ibiblio.org staff member, who deftly and quickly took care of an issue I had been struggling with for over a week. I've been trying to figure out why this bothers me so much. Perhaps its because I've never really met with something within the scope of a job that was so far beyond my skill level that I knew I couldn't just bootstrap myself up. Maybe it's because everyone seems to need IT support, in low supply, and this project will undoubtedly fall to the bottom of the stack. Or maybe it's knowing that, without installation of these tools, I'm stuck. I can't go tackle a different set of boxes.
My frustration in part comes from the fact that installation instructions for the full version of Netarchive Suite, or the Web Curator Tool, or even WERA, are all written for a system administrator. Even the user manuals often state "your system administrator will do X..." The set of skills that these documents assume is already beyond what I have. It's a matter of encountering one term, only to be met with multiple other terms that I'm unfamiliar with.
This has been an eye-opening experience. Despite the fact that I feel pretty comfortable and familiar with technology, there is an entire realm of technological depth that supports a lot of the open source tools many librarians and archivists are advocating. To use these tools, I cannot operate without system administrator support. Period.
Friday, October 24, 2008
October 24, 2008
While we're waiting for the test server to be set up, I've decided to start investigating policies, guidelines and workflows.
I've started two new wiki pages. The first is a place to aggregate mission statements, informing policies, and example web archiving policies. I've begun by collecting information from the University Library, University Archives and Records Management Service, the North Carolina Collection, North Carolina General Statutes (Public Records Law), and information from the UNC IT department on website policy. I've only found one bona fide web archiving collection policy thus far (thanks to the British Library), but I think more are out there. It's unfortunate that more libraries and archives don't put their collection policies online. I'm having to cobble together ideas based on articles I've read.
The other new page is a place to start formulating a workflow for web archiving. To help me make sure I'm accounting for all parts of the process, the workflow will be informed by the infamous OAIS reference model. Right now, what I've written is at a very high level, and I'll slowly add in more detail, especially once I can do some test harvests.
I've started two new wiki pages. The first is a place to aggregate mission statements, informing policies, and example web archiving policies. I've begun by collecting information from the University Library, University Archives and Records Management Service, the North Carolina Collection, North Carolina General Statutes (Public Records Law), and information from the UNC IT department on website policy. I've only found one bona fide web archiving collection policy thus far (thanks to the British Library), but I think more are out there. It's unfortunate that more libraries and archives don't put their collection policies online. I'm having to cobble together ideas based on articles I've read.
The other new page is a place to start formulating a workflow for web archiving. To help me make sure I'm accounting for all parts of the process, the workflow will be informed by the infamous OAIS reference model. Right now, what I've written is at a very high level, and I'll slowly add in more detail, especially once I can do some test harvests.
Monday, October 20, 2008
October 20, 2008
We're currently in a holding pattern here, waiting for access to a test server hosted by ibiblio. It looks like this server has sufficient functionality to begin testing at least one, if not more of the web archiving tools I'm interested in. This will be a great opportunity to see what these tools can do, and to determine which ones seem best for our situation.
In the meantime, I've been doing some more reading and adding to the wiki. I've also been performing a basic site analysis of several high-profile and highly desirable UNC websites. This analysis includes trying to get a sense of how deep and broad the sites are, what sort of file formats are included on the site, how much they link out, if their pages include dynamic content, and who might be interested in the site's information. I've gotten a number of sites that I think would be viable for test harvest.
I'm now looking at a list of North Carolina sites not affiliated with UNC that have been identified as high priority for harvesting. This presents more of a challenge for a number of reasons. First, many of these sites have more dynamic content, like calendars and blogs, and a greater number of links out to related resources. Another additional step that with these sites involves clearing copyright and/or obtaining permission to archive. Many web archives operate with a blanket "take down" policy (Internet Archive and MINERVA, among them), meaning they harvest first and then post a general statement notifying those who object to their harvest that they will remove the materials in question. But these policies are generally used when the harvesting is done on an unselective basis, because of the infeasibility of trying to identify and contact all of the copyright holders. Best practice for selectively harvesting sites really dictates contacting the different agencies and at least attempting a good faith effort at requesting permission to take their materials as well as to provide access to them. Some even suggest that it might heighten your archive's profile and increase its scope (Žabička, 2006, p. 3). Archives like PANDORA and WebArchiv both follow this protocol. Negotiating for harvest not only serves to avoid contentious encounters with webmasters or agency representatives, but also shows that you are attempting to be a responsible steward with your materials.
Žabička, P. (2006). Digital cultural heritage and cooperation of memory institutions. Paper presented at the 1st Colloquium of Library Information Employees of the V4+ Countries, Banska Bystrica, Slovakia.
In the meantime, I've been doing some more reading and adding to the wiki. I've also been performing a basic site analysis of several high-profile and highly desirable UNC websites. This analysis includes trying to get a sense of how deep and broad the sites are, what sort of file formats are included on the site, how much they link out, if their pages include dynamic content, and who might be interested in the site's information. I've gotten a number of sites that I think would be viable for test harvest.
I'm now looking at a list of North Carolina sites not affiliated with UNC that have been identified as high priority for harvesting. This presents more of a challenge for a number of reasons. First, many of these sites have more dynamic content, like calendars and blogs, and a greater number of links out to related resources. Another additional step that with these sites involves clearing copyright and/or obtaining permission to archive. Many web archives operate with a blanket "take down" policy (Internet Archive and MINERVA, among them), meaning they harvest first and then post a general statement notifying those who object to their harvest that they will remove the materials in question. But these policies are generally used when the harvesting is done on an unselective basis, because of the infeasibility of trying to identify and contact all of the copyright holders. Best practice for selectively harvesting sites really dictates contacting the different agencies and at least attempting a good faith effort at requesting permission to take their materials as well as to provide access to them. Some even suggest that it might heighten your archive's profile and increase its scope (Žabička, 2006, p. 3). Archives like PANDORA and WebArchiv both follow this protocol. Negotiating for harvest not only serves to avoid contentious encounters with webmasters or agency representatives, but also shows that you are attempting to be a responsible steward with your materials.
Žabička, P. (2006). Digital cultural heritage and cooperation of memory institutions. Paper presented at the 1st Colloquium of Library Information Employees of the V4+ Countries, Banska Bystrica, Slovakia.
Thursday, October 2, 2008
October 2, 2008
While reviewing the proceedings from the recent iPRES conference, I stumbled across a very interesting JISC-sponsored project: PoWR. First, the acronyms: JISC stands for the Joint Information Systems Committee, whos mission "is to provide world-class leadership in the innovative use of I[nformation and] C[ommunications] T[echnology] to support education and research" (JISC website). PoWR stands for Preservation of Web Resources. It is a short-term project funded to "organise workshops and produce a handbook that specifically addresses digital preservation issues that are relevant to the UK H[igher] Education] /F[urther] E[ducation] web management community" (from the PoWR website). The main method of communication for the PoWR project is through the blog linked above. Many of the posts are directly relevant to the challenges of archiving websites at an institution of higher education, including sections on records management, case studies, and web 2.0. I also listened to Brian Kelly's presentation, which gives a great overview of the project in 13 minutes. Check out the case study at the University of Bath when they used an innovative approach to try to "motivate the techies" by drawing parallels between their print prospectus (or catalog, for us Yanks) and the website.
Even better, PoWR has released a draft of their handbook, which they will be finalizing after they receive feedback. I haven't read through the entire thing yet, but I've already gotten to the section on making a case for web archiving at your institution. In scanning the handbook's contents, it looks like a list of the same issues I've been considering during the past weeks. It's quite exciting and I recommend a peek.
Even better, PoWR has released a draft of their handbook, which they will be finalizing after they receive feedback. I haven't read through the entire thing yet, but I've already gotten to the section on making a case for web archiving at your institution. In scanning the handbook's contents, it looks like a list of the same issues I've been considering during the past weeks. It's quite exciting and I recommend a peek.
Monday, September 29, 2008
September 29, 2008
Tool investigation continues! After contacting Paul Koerbin at the NLA and receiving a quick and informative response, I've learned that PANDAS is not being offered broadly as an open source solution for other institutions. This is primarily due to resource constraints and the fact that several other good open source tools are available. Paul suggested the Web Curator Tool (WCT) and also the Netarchive Suite (NAS), both of which were listed in my previous blog.
Looking at the possible implementation of these two tools, I've discovered to my dismay that I can't forge ahead on my own. System administrators will be required both for server installation and/or setting up a Linux box. It also makes me realize that, even if I had all of the technological knowhow to understand the installation requirements, someone with knowledge of the system in place here would still need to lend me their time and talents. This really highlights the fact that collaboration with IT is more and more necessary. I can see why librarians and archivists might get nervous at this dependence on an external department. It requires trust and patience.
Looking at the possible implementation of these two tools, I've discovered to my dismay that I can't forge ahead on my own. System administrators will be required both for server installation and/or setting up a Linux box. It also makes me realize that, even if I had all of the technological knowhow to understand the installation requirements, someone with knowledge of the system in place here would still need to lend me their time and talents. This really highlights the fact that collaboration with IT is more and more necessary. I can see why librarians and archivists might get nervous at this dependence on an external department. It requires trust and patience.
Subscribe to:
Posts (Atom)