I'm using this last post to link to a couple of the documents I've created, all of which are living on the wiki.
The first is a Web Site Collection Plan. This draft suggests policies related to archiving web sites within UARMS.
The other is a Web Site Archiving Workflow and General Information document. This outlines a suggested general workflow, instructions on using HTTrack and the NetarchiveSuite Quick Start version, and information related to the crawls I've done toward the end of this semester.
I have no plans to take down the wiki or this blog anytime in the near future, so these documents should be available until further notice.
Tuesday, April 28, 2009
Thursday, April 23, 2009
April 23, 2009
My fellowship practicum ends next Monday, so this may be my last post.
The best news, perhaps of the semester, is that my ibiblio.org contact got Web Curator Tool (WCT) up and running for me. I've been testing different parts of the program, and there are some bugs (I haven't been able to actually run a harvest) but I've gotten a feel for the procedure the software imposes (harvest authorisation -> target creation -> harvest approval -> target instance -> harvest). The interface is definitely more approachable compared with NetarchiveSuite and HTTrack, down to the colorful grains of wheat next to the "Harvester Configuration" box.
It can still feel a little complex, clicking back and forth between harvest authorisation and target, but I think this is due to (1) my lack of experience with the program (2) some lingering bugs and (3) the fact that setting up the authorisations and targets would require more time investment in the beginning than after your staple harvests were configured.
One point about WCT that vexes me is the difficult install. If I was presented with these three tools, without any prior experience, and told to evaluate them based on interface alone, I'd choose WCT above the others in a heartbeat. But ease-of-use balanced against installation tribulations is a harder decision to make. Not being the installer doesn't really alleviate that tension for me, either.
Also, the harvest authorisation feature of WCT would not be an especially useful aspect of the program for archiving UNC sites. This can be done without gaining the creator's permission based on public records law. The HA feature then becomes a superfluous extra step. On the other side of that coin, in the future, if UARMS or another department wishes to expand the scope of their web site archive to other sites for which permission is needed, this feature would become very useful, because it helps keep the temptation to crawl on the sly at bay.
Another feature of WCT that could work to slow or facilitate the workflow is the harvest approval. If you're a lone harvester, getting approval from yourself is just an extra step. But if multiple people are wrangling and suggesting different harvests, this helps portion out the resources.
I can't comment on the harvesting feature, but will do so if we can make it work in the next few days. Testing it has held up the opinion I gave at the end of my poster: right now, HTTrack is a good option for UARMS, but WCT would be a very nice alternative in the (technologically-supported and web-site-archiving-program-developed) future.
The best news, perhaps of the semester, is that my ibiblio.org contact got Web Curator Tool (WCT) up and running for me. I've been testing different parts of the program, and there are some bugs (I haven't been able to actually run a harvest) but I've gotten a feel for the procedure the software imposes (harvest authorisation -> target creation -> harvest approval -> target instance -> harvest). The interface is definitely more approachable compared with NetarchiveSuite and HTTrack, down to the colorful grains of wheat next to the "Harvester Configuration" box.
It can still feel a little complex, clicking back and forth between harvest authorisation and target, but I think this is due to (1) my lack of experience with the program (2) some lingering bugs and (3) the fact that setting up the authorisations and targets would require more time investment in the beginning than after your staple harvests were configured.One point about WCT that vexes me is the difficult install. If I was presented with these three tools, without any prior experience, and told to evaluate them based on interface alone, I'd choose WCT above the others in a heartbeat. But ease-of-use balanced against installation tribulations is a harder decision to make. Not being the installer doesn't really alleviate that tension for me, either.
Also, the harvest authorisation feature of WCT would not be an especially useful aspect of the program for archiving UNC sites. This can be done without gaining the creator's permission based on public records law. The HA feature then becomes a superfluous extra step. On the other side of that coin, in the future, if UARMS or another department wishes to expand the scope of their web site archive to other sites for which permission is needed, this feature would become very useful, because it helps keep the temptation to crawl on the sly at bay.
Another feature of WCT that could work to slow or facilitate the workflow is the harvest approval. If you're a lone harvester, getting approval from yourself is just an extra step. But if multiple people are wrangling and suggesting different harvests, this helps portion out the resources.
I can't comment on the harvesting feature, but will do so if we can make it work in the next few days. Testing it has held up the opinion I gave at the end of my poster: right now, HTTrack is a good option for UARMS, but WCT would be a very nice alternative in the (technologically-supported and web-site-archiving-program-developed) future.
Monday, April 6, 2009
April 6, 2009
I spent last week volunteering for and attending DigCCurr2009 here in Chapel Hill, which was robustly attended by almost 300 people from around the world. It was an informative experience, and as always a nice chance to renew my enthusiasm by seeing that there are others out there working on similar problems and coming up with different paths forward.
On Thursday, I presented my poster. A lot of good questions came my way. As I've mentioned before, this poster describes my work here in UARMS evaluating three open source software tools used for archiving websites. Those tools are HTTrack, NetarchiveSuite, and Web Curator Tool (WCT).
All three tools would be good choices, and scored very high. HTTrack came out on top with a score of 29. WCT was close behind with 28 points. NetarchiveSuite was third, but still had 25 points.
My recommendation, right now, is for UARMS to go with HTTrack (with WCT as runner-up). This was surprising to me, since I had originally considered it to be the least enticing option. It definitely has the fewest features (no metadata assignment, preservation functions, permissions tracking), but it is the simplest to install and run. I suppose, the moral of this story is that your situation and personnel dictate your best software choice. The thoroughbred may not always be the right horse for you. HTTrack will be easiest for any user to pick up and operate without extensive time reading documentation or learning about archiving websites.
On Thursday, I presented my poster. A lot of good questions came my way. As I've mentioned before, this poster describes my work here in UARMS evaluating three open source software tools used for archiving websites. Those tools are HTTrack, NetarchiveSuite, and Web Curator Tool (WCT).
All three tools would be good choices, and scored very high. HTTrack came out on top with a score of 29. WCT was close behind with 28 points. NetarchiveSuite was third, but still had 25 points.
My recommendation, right now, is for UARMS to go with HTTrack (with WCT as runner-up). This was surprising to me, since I had originally considered it to be the least enticing option. It definitely has the fewest features (no metadata assignment, preservation functions, permissions tracking), but it is the simplest to install and run. I suppose, the moral of this story is that your situation and personnel dictate your best software choice. The thoroughbred may not always be the right horse for you. HTTrack will be easiest for any user to pick up and operate without extensive time reading documentation or learning about archiving websites.
Tuesday, March 24, 2009
March 24, 2009
What a lapse in posts! The good news is that I've been busy preparing for the DigCCurr conference and trying to wrap up installation of the last tool to try (Web Curator Tool). Installation and setup of WCT has proven to be the most onerous of all, and beyond my own technological skill. Two new(ish) computers later, though, and things seem closer. I'll keep you posted.
My poster is practically finished. I ended up doing some modification to the rubric I discussed in the last post. I decided to move some of the criteria on documentation and support to their own category, Documentation/Support, because of the importance of having good explanations for how to install and use these tools. In my experience, lack of documentation can be common with open source tools. But when those using the tools are new to the field or to that type of technology, it becomes all the more important.
I changed the order of criteria for Interface. After some more thought, it seemed to me that simply giving access to a GUI (graphical user interface) is less ideal that offering access to a GUI and to the command line. That category was revised to reflect as much.
I also added some additional features, including bandwidth allocation and the ability to make alterations to harvested materials.
I won't post the results here until after DigCCurr 2009, just to keep up the suspense. Being able to test and thoughtfully evaluate a range of tools is surely a luxury for most institutions. I'm glad I have been able to help UNC in this way.
My poster is practically finished. I ended up doing some modification to the rubric I discussed in the last post. I decided to move some of the criteria on documentation and support to their own category, Documentation/Support, because of the importance of having good explanations for how to install and use these tools. In my experience, lack of documentation can be common with open source tools. But when those using the tools are new to the field or to that type of technology, it becomes all the more important.
I changed the order of criteria for Interface. After some more thought, it seemed to me that simply giving access to a GUI (graphical user interface) is less ideal that offering access to a GUI and to the command line. That category was revised to reflect as much.
I also added some additional features, including bandwidth allocation and the ability to make alterations to harvested materials.
I won't post the results here until after DigCCurr 2009, just to keep up the suspense. Being able to test and thoughtfully evaluate a range of tools is surely a luxury for most institutions. I'm glad I have been able to help UNC in this way.
Tuesday, March 3, 2009
March 3, 2009
I've made more headway on my poster, and I thought I'd share in more depth the software evaluation rubric I've come up with. As I mentioned last time, I've come up with three categories of criteria: Ease of Installation/Setup, Ease of Use, and Features. I started doing some research into software evaluation, to try to see if I could find a vetted rubric to use.
In my literature search, I discovered that software evaluation can be an extremely intense process, punctuated by algorithms and equations. I think this type of evaluation actually centers on the code. I also found some information on ISONORM 9241/10, described here. This came closer to what I was looking for, but centered too much on ergonomics and usability of the software dialog. It didn't cover some of the more subjective aspects I wanted to measure. More relevant to my task were books and articles talking about how to perform a needs assessment and then devise a rubric to suit your own requirements. The down side is the subjective nature of the rubric. But in the end, I decided that for the purposes of my fellowship, where no replicable results are needed, this would work best.
So here's the rubric I've developed. (It's on the wiki, so you'll need access to that before it will come up.) The idea is that each tool will be assessed based on the criteria in each of the 3 categories. The numbers that correspond to the criteria for that tool would be added up to a total. The lower the number, the less conducive it would be to use in UARMS at this point in time. I've written out some explanations for some of the criteria below. For instance, here are my definitions of the criteria in the "Available Documentation for Installation" scale:
None: No documentation available
Sparse: Highly technical and brief
Adequate: Easy to read, mimics common install guides, may be out of date
Robust: Up-to-date, includes troubleshooting section
The "Available Documentation for Use" is similar:
None: No documentation available
Sparse: Some online documentation, may be interspersed with installation information
Adequate: Dedicated user's manual
Robust: User's manual plus built-in help
Finally, the "Knowledge Required" section deserves some clarification:
Program Administrator: Requires a program administrator
UNIX Savvy Librarian: Knows basics of a UNIX system
Technology Savvy User: Feels comfortable doing basic troubleshooting on Windows and Mac platforms
Out of the Box: Feels comfortable doing out of the box installations
Again, I know these are subjective. But I've tried to make them as explicit as possible, and also to select the characteristics I feel have the most impact on whether or not a tool might be adopted in the UARMS setting.
In my literature search, I discovered that software evaluation can be an extremely intense process, punctuated by algorithms and equations. I think this type of evaluation actually centers on the code. I also found some information on ISONORM 9241/10, described here. This came closer to what I was looking for, but centered too much on ergonomics and usability of the software dialog. It didn't cover some of the more subjective aspects I wanted to measure. More relevant to my task were books and articles talking about how to perform a needs assessment and then devise a rubric to suit your own requirements. The down side is the subjective nature of the rubric. But in the end, I decided that for the purposes of my fellowship, where no replicable results are needed, this would work best.
So here's the rubric I've developed. (It's on the wiki, so you'll need access to that before it will come up.) The idea is that each tool will be assessed based on the criteria in each of the 3 categories. The numbers that correspond to the criteria for that tool would be added up to a total. The lower the number, the less conducive it would be to use in UARMS at this point in time. I've written out some explanations for some of the criteria below. For instance, here are my definitions of the criteria in the "Available Documentation for Installation" scale:
None: No documentation available
Sparse: Highly technical and brief
Adequate: Easy to read, mimics common install guides, may be out of date
Robust: Up-to-date, includes troubleshooting section
The "Available Documentation for Use" is similar:
None: No documentation available
Sparse: Some online documentation, may be interspersed with installation information
Adequate: Dedicated user's manual
Robust: User's manual plus built-in help
Finally, the "Knowledge Required" section deserves some clarification:
Program Administrator: Requires a program administrator
UNIX Savvy Librarian: Knows basics of a UNIX system
Technology Savvy User: Feels comfortable doing basic troubleshooting on Windows and Mac platforms
Out of the Box: Feels comfortable doing out of the box installations
Again, I know these are subjective. But I've tried to make them as explicit as possible, and also to select the characteristics I feel have the most impact on whether or not a tool might be adopted in the UARMS setting.
Tuesday, February 24, 2009
February 24, 2009
It's past time for another update, and things have been moving along slowly but surely. The most major accomplishment I can report is that I've finished a draft Web Site Collection Plan. I last mentioned this at the end of January. It's based on my assessment of other web site collection policies as well as the Collection Plan template from the Web-at-Risk project. This draft is available here.
Some parts of this policy are incomplete, simply because they will depend on an overarching digital preservation program. Other parts depend on current metadata practices here in Wilson Library that I haven't been able to pin down yet. Of course they're all flexible recommendations. I think it would be a good place to start, and possibly even a document that can be expanded to other collecting areas.
There's also been some headway on the Web Curator Tool (WCT) installation, thanks to help from staff at ibiblio.org. I gather that it is installed, but I could not yet get it to run. I'm sure this is due to my own tenuous knowledge of how to run things in a Unix environment.
Finally, I've been working on my poster presentation for DigCCurr 2009, which is looming on the horizon. The title is "Getting the Tar Off of Our Heels: Moving Forward with Archiving University of North Carolina at Chapel Hill Websites." Since this poster hinges on the successful trial of WCT, I'm hoping that I can get that tool running in some capacity soon. I've decided to develop a rubric for measuring different aspects of the web archiving tools I've trialed. The three facets will be (1) ease of installation/setup (2) ease of use (3) features. I'm trying to find a vetted set of characteristics for evaluating software, but am thus far not having luck. Barring that, the characteristics I have come up with are highly subjective, but hopefully still meaningful.
Some parts of this policy are incomplete, simply because they will depend on an overarching digital preservation program. Other parts depend on current metadata practices here in Wilson Library that I haven't been able to pin down yet. Of course they're all flexible recommendations. I think it would be a good place to start, and possibly even a document that can be expanded to other collecting areas.
There's also been some headway on the Web Curator Tool (WCT) installation, thanks to help from staff at ibiblio.org. I gather that it is installed, but I could not yet get it to run. I'm sure this is due to my own tenuous knowledge of how to run things in a Unix environment.
Finally, I've been working on my poster presentation for DigCCurr 2009, which is looming on the horizon. The title is "Getting the Tar Off of Our Heels: Moving Forward with Archiving University of North Carolina at Chapel Hill Websites." Since this poster hinges on the successful trial of WCT, I'm hoping that I can get that tool running in some capacity soon. I've decided to develop a rubric for measuring different aspects of the web archiving tools I've trialed. The three facets will be (1) ease of installation/setup (2) ease of use (3) features. I'm trying to find a vetted set of characteristics for evaluating software, but am thus far not having luck. Barring that, the characteristics I have come up with are highly subjective, but hopefully still meaningful.
Tuesday, February 10, 2009
February 10, 2009
I've finished my little experiment, as mentioned in the last post. I have to say, it was pretty enlightening, and I'm glad I took the time to do it. (As a side note, when I was almost finished, I suffered a data loss - you would think that a student of digital curation would know better - and had to start over. So, let that be a lesson to me for backing up my own data.) So here's how it went.
K-12 International Outreach Office
Research & Evaluation (OIA)
Program in the Humanities and Human Values (CoAS)
City & Regional Planning
Dean's Office (School of Medicine)
Biochemistry & Biophysics (School of Medicine)
Medical Science Teaching Labs (School of Medicine)
Business & Finance (School of Dentistry)
Human Resources (School of Dentistry)
Student Services (Dept of Maternal & Child Health)
Center for Urban & Regional Studies
Manuscripts Department (Wilson Library)
Conservation (Wilson Library)
Grounds Services (Facilities)
Building Services (Facilities)
Housekeeping (Facilities)
I read each schedule, keeping track of which series were supposed to be transferred to University Archives (I'll call these "UARMS series" here). I then tried to locate each department's associated website, and was able to do so in all cases. Finally, I scoured these sites looking for any records that fall into the UARMS series.
Here's an example, Housekeeping (Facilities) has 6 series whose records are normally transferred to UARMS in paper format. Two of those six, photographs and policies/procedures, can be found on the Housekeeping website. This means that documents that fall into 33% of its UARMS series records types can be found to some degree in digital form on their website.
As anticipated, the website pages ranged widely in depth and type of content. Some were only one page of text, others were numerous pages deep with multiple file types present. Here are the general results of my survey:
- The percentage of UARMS series document types that could be found on the departments' websites ranged from 0% to 67%.
- The average number of those types found on the website across all sites was only 27%.
- The department with the most record-worthy online content was, surprisingly, Grounds Services.
- Many of these departments had scheduled records series in common. The most common were committee and council records (12 out of 16), correspondence (15 out of 16), photographs (12 out of 16), and polices and procedures (13 out of 16). Of these, only photographs were routinely found online (92% of the departments).
- Photographs, policies/procedures, and conference/workshop (conducted) records were the most common types found online.
- Only 10 of these 16 departments had "website records" on their record schedules, even though all had some sort of website.
Academic Affairs Administration (incl Int'l Programs)
CFNC-Pathways
NC in the World (NC Center for Int'l Understanding)
Licensure Office (Academic Affairs)
Research and Sponsored Programs (Academic Affairs)
To compare, here are some more findings:
- The percentage of UARMS series document types that could be found on the departments' websites ranged from 10% to 57%.
- The average number of those types found on the website across all sites was 39%.
- Much more prevalent for these departments were reports (including annual) and publications.
- I am surprised that more information isn't online, especially considering that UNC is a state university. I had expected a lot more online information in the form of events, strategic planning documents, statistics, and meeting records. I was gratified by the fact that the General Administration sites, on average, did have a lot more of this type of information.
- In general, I was also surprised at how little information some of these departments have on their websites, and how out of date some of that information is.
- I am curious about the number of intranets that exist on campus. I found one or two sites that had log in pages, and one that required a log in to view committee minutes, which was a UARMS series. I think/hope that much of the information I mentioned in #1 may in fact be on the intranet instead of the public web.
- Harvesting these websites will pull in a lot fewer documents that qualify as records than I had expected.
Again, my method has some faults. My sample is small compared to the gamut of UNC departments. Even though I chose randomly, many of the departments were from the same school/college. Also, those who have completed schedules probably are more records-aware in the first place, and may, as a result, have more content online. It also probably isn't fair to look for correspondence or some types of personnel records online, even though they are UARMS series. These aren't the types of things people would typically put on a website. Still, if nothing else, scanning through all of these websites in search of records has given me a better idea of what exactly this institution is putting out there and what, eventually, we would pull in.
Thursday, January 29, 2009
January 29, 2009
Here's an update on what I've been working with in the last few weeks. It's been difficult trying to figure out where to go this semester, especially while I'm waiting for some technology-related projects to move forward.
1. iRODS: I've been thrown head first into iRODS in an incredibly short but very interesting class. I've gotten the client installed and my group is now working on identifying policies we can use to create rules.
2. Web Archiving Policies: In continuation from last semester, I'm trying to draft some web archiving policies using the Collection Plan Template from the Web-at-Risk project as a guide. This has proven to be more difficult and more simple than I expected. Here's what I mean. Because the UARMS impetus for collecting websites is partially grounded in North Carolina Public Records Law, much of this information can be taken directly from the department's current scope used for paper records, and can be based on the department's established mission. On the other hand, areas dealing with digital object preservation, migration and workflow are more difficult to determine without any overarching digital object policies to which I can refer.
I know from past work experience how people abhor the word "policy." But I'm coming to see that policies can end up serving a unifying purpose and also save work for those down the road. The difficulty, I guess, is in having all interested parties agree on one policy. It's a lot of effort up front and probably seems to delay "real" work. Peddling around on your own can end up giving others something to work with, and I hope that will be the case here.
3. A Little Experiment: I've decided to try an experiment in order to determine exactly how many scheduled records would be pulled in from websites. Using a random sequence generator, I've selected 20% of the UNC departments with completed records schedules. I'm going through and listing the series that would normally be transferred to UARMS at some point in time. Then, as best I can, I'm assessing the departments' websites for those series. Although collecting records would not be the only reason we would harvest the sites, I'm interested in seeing exactly how often scheduled records are put online. Are departments really posting documents UARMS would normally collect in print?
I recognize that this is limited in certain ways. For instance, those who have completed individual records schedules may be more likely to publish their important documents online in the first place. Second, those who do not have completed records schedules are probably departments we would or should be more aggressive in harvesting. I may also fail to locate the records during my survey of the website. Still, I want to make sure I have a closer idea to exactly what a web harvest would accomplish from a public records compliance angle, and I think this little experiment will help.
4. Ongoing Efforts: I'm reading any new publications I come across on web archiving. I'm alsoruminating on my poster presentation coming up in April at DigCCurr 2009. With the delays I've had in getting some web archiving tools set up, I may need to take that poster in a different direction. There's still a bit of time, there. Whatever it ends up being, I hope someone will find it useful!
1. iRODS: I've been thrown head first into iRODS in an incredibly short but very interesting class. I've gotten the client installed and my group is now working on identifying policies we can use to create rules.
2. Web Archiving Policies: In continuation from last semester, I'm trying to draft some web archiving policies using the Collection Plan Template from the Web-at-Risk project as a guide. This has proven to be more difficult and more simple than I expected. Here's what I mean. Because the UARMS impetus for collecting websites is partially grounded in North Carolina Public Records Law, much of this information can be taken directly from the department's current scope used for paper records, and can be based on the department's established mission. On the other hand, areas dealing with digital object preservation, migration and workflow are more difficult to determine without any overarching digital object policies to which I can refer.
I know from past work experience how people abhor the word "policy." But I'm coming to see that policies can end up serving a unifying purpose and also save work for those down the road. The difficulty, I guess, is in having all interested parties agree on one policy. It's a lot of effort up front and probably seems to delay "real" work. Peddling around on your own can end up giving others something to work with, and I hope that will be the case here.
3. A Little Experiment: I've decided to try an experiment in order to determine exactly how many scheduled records would be pulled in from websites. Using a random sequence generator, I've selected 20% of the UNC departments with completed records schedules. I'm going through and listing the series that would normally be transferred to UARMS at some point in time. Then, as best I can, I'm assessing the departments' websites for those series. Although collecting records would not be the only reason we would harvest the sites, I'm interested in seeing exactly how often scheduled records are put online. Are departments really posting documents UARMS would normally collect in print?
I recognize that this is limited in certain ways. For instance, those who have completed individual records schedules may be more likely to publish their important documents online in the first place. Second, those who do not have completed records schedules are probably departments we would or should be more aggressive in harvesting. I may also fail to locate the records during my survey of the website. Still, I want to make sure I have a closer idea to exactly what a web harvest would accomplish from a public records compliance angle, and I think this little experiment will help.
4. Ongoing Efforts: I'm reading any new publications I come across on web archiving. I'm alsoruminating on my poster presentation coming up in April at DigCCurr 2009. With the delays I've had in getting some web archiving tools set up, I may need to take that poster in a different direction. There's still a bit of time, there. Whatever it ends up being, I hope someone will find it useful!
Tuesday, January 20, 2009
January 20, 2009
Through some listservs, I caught wind of a year-old initiative of which I was previously unaware: LiWA (Living Web Archive). In its own words:
"LiWA's aim is to develop new tools and approaches that can deal with these challenges and improve archive fidelity, coherence, and interpretability. The strategy is to frame them so that they are more resistant to future changes, by leveraging universal tools like the browser and by fostering community-based responses to new challenges."
In my words, this initiative seems to be tackling some of the pervasive issues with current web archiving technology, issues like capturing deep and hidden web content, and dealing with what they call "temporal coherence." This last concept, from what I can understand, encompasses the following type of problem: if it takes you 2 days to crawl a large site, and parts of that site change during that time period, have you captured the site faithfully?
Topical areas they're exploring include the aforementioned temporal coherence, as well as archive fidelity, spam cleansing, semantic evolution, the social web, and rich media. There are a number of presentations on the website explaining these concepts, and I'll be digging into them this week.
"LiWA's aim is to develop new tools and approaches that can deal with these challenges and improve archive fidelity, coherence, and interpretability. The strategy is to frame them so that they are more resistant to future changes, by leveraging universal tools like the browser and by fostering community-based responses to new challenges."
In my words, this initiative seems to be tackling some of the pervasive issues with current web archiving technology, issues like capturing deep and hidden web content, and dealing with what they call "temporal coherence." This last concept, from what I can understand, encompasses the following type of problem: if it takes you 2 days to crawl a large site, and parts of that site change during that time period, have you captured the site faithfully?
Topical areas they're exploring include the aforementioned temporal coherence, as well as archive fidelity, spam cleansing, semantic evolution, the social web, and rich media. There are a number of presentations on the website explaining these concepts, and I'll be digging into them this week.
Wednesday, January 14, 2009
January 14, 2009
The semester begins, and it's time to revisit the goals. Here they are, from the beginning of last semester, and my thoughts on what's happened.
Goal #1 - I have read up on quite a bit of "practice" but even more projection. Institutions are either treating websites just like other objects they would normally ingest, are keeping their distance from collecting websites, or are outsourcing to the Internet Archive. My main discovery is that UNC seems to be with the pack on most of the issues.
Goals #2 and #3 - How much and how often? These are the big questions for harvesting. For our purposes, I feel that selective and routine but infrequent crawls would work the best. Domain-wide harvests are typically done by national libraries, and this seems beyond our scope for now. We also want to make sure we're dovetailing with other institutions in our area, not duplicating work. It would be my hope that selective, routine harvests would fill in a lot of the public records sought by University Archives, fulfilling the university's legal obligations, and also round out current series in the Southern Historical collection.
Goal #4 - As I mentioned above, UNC appears to be milling in the pack when it comes to pursuing a web archiving plan. In order to move ahead, some simple infrastructure and policy commitments would need to be made.
Goal #5 - I feel that this is where I will be starting this semester. Reading on and developing a solid policy and procedure base for archiving websites is, I feel, the best preparation I can give should the UNC libraries decide to begin regular harvesting. This will also support my work with iRODS this semester, which needs a rule base built from policies.
Goals:
- What is the current idea of a responsible web archiving strategy, including preservation, metadata, access?
- Work to refine scope - how much do we want to keep/is feasible to keep
- Identify what has been tried, is being tried
- Compare these with UNC's situation (resources/collections)
- Review current policy and procedure
Goal #1 - I have read up on quite a bit of "practice" but even more projection. Institutions are either treating websites just like other objects they would normally ingest, are keeping their distance from collecting websites, or are outsourcing to the Internet Archive. My main discovery is that UNC seems to be with the pack on most of the issues.
Goals #2 and #3 - How much and how often? These are the big questions for harvesting. For our purposes, I feel that selective and routine but infrequent crawls would work the best. Domain-wide harvests are typically done by national libraries, and this seems beyond our scope for now. We also want to make sure we're dovetailing with other institutions in our area, not duplicating work. It would be my hope that selective, routine harvests would fill in a lot of the public records sought by University Archives, fulfilling the university's legal obligations, and also round out current series in the Southern Historical collection.
Goal #4 - As I mentioned above, UNC appears to be milling in the pack when it comes to pursuing a web archiving plan. In order to move ahead, some simple infrastructure and policy commitments would need to be made.
Goal #5 - I feel that this is where I will be starting this semester. Reading on and developing a solid policy and procedure base for archiving websites is, I feel, the best preparation I can give should the UNC libraries decide to begin regular harvesting. This will also support my work with iRODS this semester, which needs a rule base built from policies.
Subscribe to:
Posts (Atom)