Thursday, January 29, 2009

January 29, 2009

Here's an update on what I've been working with in the last few weeks. It's been difficult trying to figure out where to go this semester, especially while I'm waiting for some technology-related projects to move forward.

1. iRODS: I've been thrown head first into iRODS in an incredibly short but very interesting class. I've gotten the client installed and my group is now working on identifying policies we can use to create rules.

2. Web Archiving Policies: In continuation from last semester, I'm trying to draft some web archiving policies using the Collection Plan Template from the Web-at-Risk project as a guide. This has proven to be more difficult and more simple than I expected. Here's what I mean. Because the UARMS impetus for collecting websites is partially grounded in North Carolina Public Records Law, much of this information can be taken directly from the department's current scope used for paper records, and can be based on the department's established mission. On the other hand, areas dealing with digital object preservation, migration and workflow are more difficult to determine without any overarching digital object policies to which I can refer.

I know from past work experience how people abhor the word "policy." But I'm coming to see that policies can end up serving a unifying purpose and also save work for those down the road. The difficulty, I guess, is in having all interested parties agree on one policy. It's a lot of effort up front and probably seems to delay "real" work. Peddling around on your own can end up giving others something to work with, and I hope that will be the case here.

3. A Little Experiment: I've decided to try an experiment in order to determine exactly how many scheduled records would be pulled in from websites. Using a random sequence generator, I've selected 20% of the UNC departments with completed records schedules. I'm going through and listing the series that would normally be transferred to UARMS at some point in time. Then, as best I can, I'm assessing the departments' websites for those series. Although collecting records would not be the only reason we would harvest the sites, I'm interested in seeing exactly how often scheduled records are put online. Are departments really posting documents UARMS would normally collect in print?

I recognize that this is limited in certain ways. For instance, those who have completed individual records schedules may be more likely to publish their important documents online in the first place. Second, those who do not have completed records schedules are probably departments we would or should be more aggressive in harvesting. I may also fail to locate the records during my survey of the website. Still, I want to make sure I have a closer idea to exactly what a web harvest would accomplish from a public records compliance angle, and I think this little experiment will help.

4. Ongoing Efforts: I'm reading any new publications I come across on web archiving. I'm alsoruminating on my poster presentation coming up in April at DigCCurr 2009. With the delays I've had in getting some web archiving tools set up, I may need to take that poster in a different direction. There's still a bit of time, there. Whatever it ends up being, I hope someone will find it useful!

Tuesday, January 20, 2009

January 20, 2009

Through some listservs, I caught wind of a year-old initiative of which I was previously unaware: LiWA (Living Web Archive). In its own words:

"LiWA's aim is to develop new tools and approaches that can deal with these challenges and improve archive fidelity, coherence, and interpretability. The strategy is to frame them so that they are more resistant to future changes, by leveraging universal tools like the browser and by fostering community-based responses to new challenges."

In my words, this initiative seems to be tackling some of the pervasive issues with current web archiving technology, issues like capturing deep and hidden web content, and dealing with what they call "temporal coherence." This last concept, from what I can understand, encompasses the following type of problem: if it takes you 2 days to crawl a large site, and parts of that site change during that time period, have you captured the site faithfully?

Topical areas they're exploring include the aforementioned temporal coherence, as well as archive fidelity, spam cleansing, semantic evolution, the social web, and rich media. There are a number of presentations on the website explaining these concepts, and I'll be digging into them this week.

Wednesday, January 14, 2009

January 14, 2009

The semester begins, and it's time to revisit the goals. Here they are, from the beginning of last semester, and my thoughts on what's happened.

Goals:

  1. What is the current idea of a responsible web archiving strategy, including preservation, metadata, access?
  2. Work to refine scope - how much do we want to keep/is feasible to keep
  3. Identify what has been tried, is being tried
  4. Compare these with UNC's situation (resources/collections)
  5. Review current policy and procedure
A lot of progress has been made, and I feel much more aware of what is achievable during my fellowship.

Goal #1 - I have read up on quite a bit of "practice" but even more projection. Institutions are either treating websites just like other objects they would normally ingest, are keeping their distance from collecting websites, or are outsourcing to the Internet Archive. My main discovery is that UNC seems to be with the pack on most of the issues.
Goals #2 and #3 - How much and how often? These are the big questions for harvesting. For our purposes, I feel that selective and routine but infrequent crawls would work the best. Domain-wide harvests are typically done by national libraries, and this seems beyond our scope for now. We also want to make sure we're dovetailing with other institutions in our area, not duplicating work. It would be my hope that selective, routine harvests would fill in a lot of the public records sought by University Archives, fulfilling the university's legal obligations, and also round out current series in the Southern Historical collection.
Goal #4 - As I mentioned above, UNC appears to be milling in the pack when it comes to pursuing a web archiving plan. In order to move ahead, some simple infrastructure and policy commitments would need to be made.
Goal #5 - I feel that this is where I will be starting this semester. Reading on and developing a solid policy and procedure base for archiving websites is, I feel, the best preparation I can give should the UNC libraries decide to begin regular harvesting. This will also support my work with iRODS this semester, which needs a rule base built from policies.

Tuesday, November 25, 2008

November 25, 2008

The American Library Association Association for Library Collections and Technical Services recently held a 3 day email forum on web archiving. I was clued in to this forum by one of the archivists here, and I'm glad I got to lurk a bit. Although the conversation was a little disjointed, and dominated by several respondents, it was interesting and enlightening. I thought I'd put down some of my thoughts here.


I got the impression that many who spoke up on the list were focusing on government documents. This reflects what I've found in the literature - most published accounts of web archiving in the US also seem to target government sites. This makes sense, considering (1) those sites have, what is considered to be, vetted information (2) have information libraries often collect without heavy selection procedures (3) are perceived as more stable than others.


Of those who responded, the most often mentioned service or tool was the Internet Archive's Archive-it. Others include HTTrack, Context Miner, Fedora, OCLC's Web Archives Workbench, and del.icio.us. This forum was also where I learned about Zoetrope (see my November 20 post).

Manual vs. automated harvesting also seemed to be equally represented. Most still feel that curating the materials is the valuable service that librarians can provide over and above a broad approach like that of the Internet Archive.

Although many stated they weren't actually harvesting web content yet, some of those who were mentioned that they have looked for copyright statements on pages and also sought permission to archive. I think that, because so many expressed their newness to this topic, and because of the newness of web archiving in general, this topic hasn't yet gained a critical mass of experience or attention.


Another really interesting thing was the number of respondents who said they're creating MARC records for freely available website resources, without actually harvesting those resources. This really astonishes me. Most cited storage and technological limitations as barriers to harvesting. I understand that the librarians are interested in getting their users as much information as possible, and hope to do so through catalog searches, but with the ephemeral nature of web content, not to mention unstable URLs, I would imagine these links rot quickly and it would be incredibly demanding to maintain. I also wonder if a list on an archivists' listserv would garner different reactions and methods. More interesting was the claim that they only enter "stable" sites. Sort of an oxymoron. From reading between the lines, I gather that these are sites they feel have a lower possibility for changing URL, but I'm not sure. If there's one thing I've learned in my time at SILS (there's more than one thing, but I'll keep it to one for now), it's that digital objects are anything but stable and that it's best to never think of them as such.


On a related note, many also talked about collecting e-books and e-prints versus the entire website. This must be a difficult decision when you're looking at a document that you formerly collected in print, which is now available online. Many still seem to look for the online equivalent of books or pamphlets for pulling into their repositories. Drawing this distinction allows repositories to draw boundaries in the boundary-less web, but it also may mean they're missing valuable information that has been converted to a "non-book" format.

Again, these are just my thoughts and interpretations of the conversations from the forum and shouldn't be taken as a complete synopsis!

Thursday, November 20, 2008

November 20, 2008

Talk about a neat and shiny tool. Julien Masanes forwarded this link to a web archiving listserv I've subscribed to. It is definitely worth the time to watch the video:

http://uwnews.washington.edu/ni/article.asp?articleID=45255

Zoetrope looks like a fascinating way to actually access harvested web material. Developed by University of Washington and Adobe, it allows you to compare a page over time, multiple pages over time, or even sections of multiple pages (using "lenses") over time. It will even graph changes in a numerical entity (think product prices or temperatures). Although it harvests sites VERY often, I would hope it would still be useful for a site harvested only monthly or bimonthly. They mention eventually using the Internet Archive as a source for harvested sites, which may mean it can read and display .arc files. Even better - they plan to make it available for free.

Monday, November 10, 2008

November 10, 2008

I am not a system administrator.

This is a really hard admission for me to make. I've been struggling to try to wrap my head around installing some web archiving utilities, like Netarchive Suite and WERA. Thanks to the generous help of ibiblio.org folks, the first tool is now up and running (albeit in a trial version) on our test server and I've completed my first harvest. I'm really excited about that major hurdle. Now, I just need to get a viewer for the .arc files created by Netarchive Suite.

I've had to admit that, despite my open mind and eagerness to tackle these things myself, I simply do not have the technological background to strike out on my own. This hit home when I was sitting down with an ibiblio.org staff member, who deftly and quickly took care of an issue I had been struggling with for over a week. I've been trying to figure out why this bothers me so much. Perhaps its because I've never really met with something within the scope of a job that was so far beyond my skill level that I knew I couldn't just bootstrap myself up. Maybe it's because everyone seems to need IT support, in low supply, and this project will undoubtedly fall to the bottom of the stack. Or maybe it's knowing that, without installation of these tools, I'm stuck. I can't go tackle a different set of boxes.

My frustration in part comes from the fact that installation instructions for the full version of Netarchive Suite, or the Web Curator Tool, or even WERA, are all written for a system administrator. Even the user manuals often state "your system administrator will do X..." The set of skills that these documents assume is already beyond what I have. It's a matter of encountering one term, only to be met with multiple other terms that I'm unfamiliar with.

This has been an eye-opening experience. Despite the fact that I feel pretty comfortable and familiar with technology, there is an entire realm of technological depth that supports a lot of the open source tools many librarians and archivists are advocating. To use these tools, I cannot operate without system administrator support. Period.

Friday, October 24, 2008

October 24, 2008

While we're waiting for the test server to be set up, I've decided to start investigating policies, guidelines and workflows.

I've started two new wiki pages. The first is a place to aggregate mission statements, informing policies, and example web archiving policies. I've begun by collecting information from the University Library, University Archives and Records Management Service, the North Carolina Collection, North Carolina General Statutes (Public Records Law), and information from the UNC IT department on website policy. I've only found one bona fide web archiving collection policy thus far (thanks to the British Library), but I think more are out there. It's unfortunate that more libraries and archives don't put their collection policies online. I'm having to cobble together ideas based on articles I've read.

The other new page is a place to start formulating a workflow for web archiving. To help me make sure I'm accounting for all parts of the process, the workflow will be informed by the infamous OAIS reference model. Right now, what I've written is at a very high level, and I'll slowly add in more detail, especially once I can do some test harvests.