A few weeks ago I was honored to attend the National Digital Stewardship Alliance meeting. The NDSA was planned by the Library of Congress as part of their NDIIP project.  There were more people there than I expected and it was a humbling experience to hear some of the brightest and most creative brains in Digital Preservation speak.

The high volume of information was overwhelming.  I spent more than six hours at the end of the conference compiling a PowerPoint of the important research highlights.   A small sample of this information is included below.

There were many wonderful presentations giving case studies on how institutions used their own creativity to try and enhance the longevity or migratability (new word?) of their digital files.  The amazing work often was done on a shoestring which though unfortunate, also forced a certain level of imagination and invention.

A few examples are:

Jack Brighton, of campus radio station WILL, gave a wonderful presentation on what a small station is doing to make their civil rights collection more accessible.

Kickstarter.com did a great presentation on how they are helping arts projects get funded and we hope that as they branch into community work that digital preservation might fit into that.

The UK Web Archiving project covered some of the complexities and true effort that it takes to try and tackle capturing the online history of its nation.   http://www.webarchive.org.uk/ukwa/

- As of December 2010 – 9 million sites with .uk, probably 1M more

- 10,027 websites archived

- Need skills in Linux, Java, Hadoop, and SOL

5 keys processes to web archiving

- Selection

- Harvesting

- Storage

- Preservation

- Access

http://www.webarchive.org.uk/ukwa/ngram/

So after taking in all this good information, what is it that I have left the conference with?

People just like us are doing some wonderful problem solving out there.  There is some potential being unlocked, but there is so much to do.

As I see it the Action Items are:

1)    Greater broadcasting of the successful case studies for migration and open solutions.

2)    Training classes in how to boil this down for each type of format/issue.  The NDSA Outreach group held a session called “Digital Preservation in a Box”.  This is the beginning of standardizing the tools that we need.

3)    Overarching education to information and production professionals, as well as, the general public about the dangers of digital fragility and the need for migration (at the least).

I have mentioned to my classes for years that future anthropologists, sociologists and historians will have little to sift through from the late 20^th century.

Some of it is being worked on by archivists now but much is gone.  Let’s keep making progress so that the future of our current history is not lost, like the way of silent films.

More informational tidbits from NDSA:

Other Great Projects

—       ThatCamp.org

—       http://www.scola.org/scola/sampledigitalarchive.aspx

NYPL Labs

—     http://search.creativecommons.org/?q=nypl+map+rectifier&sourceid=Mozilla-search

—     http://menus.nypl.org/

Archiving Facebook

Grad student designed Firefox add-on for individual archiving of Fb.

www.Bit.ly/archivefb

Preserving Virtual Worlds

—      www.ideals.illinois.edu/handle/2142/17097

Cool Tools

—  http://www.google.com/landing/historypin/

—  http://blogs.yu.edu/cpa/2011/02/23/open-source-video-platforms-kaltura-vs-entermedia/

Great Quotes

—    JackBrighton”(DAM) is more like an appliance than an Ecosystem.”

—    Michael Nelson “We need to raise the level of user expectations.”

—    Michael Nelson “In all good computer science functions you solve the problem through indirection.”

—    Wheatley and Frieze “The world does not change one person at a time.  It changes as networks of relationships form among people        who discover they share a common cause and vision of what’s possible.”

—    Tim O’Reilly(?) “Teach preservation as a mindset.  Bake this into the tools.”

New Phrases

—    Social Curation

—    Metadata Ecologists

There has been much written recently about digital fragility. Researchers and archivists have heard dueling longevity and futuristic projections. In trying to push this dire need without appearing like Chicken Little, I have embarked on serious primary research to expose the sheer volume of the problem. The in-depth article will be coming out in a professional journal within the year. Until then, I felt that I needed to speak up a bit about the need for our activism.

Between my teaching digital archiving courses and my work with clients, this issue has been prevalent in each work day. In fact, while re-processing an archives for a client, a case of 5 ¼” floppy discs were found. No one in-house knew exactly if these were of value, what was on them or even if they were created by the organization. When we offered to open them on a computer with a floppy drive, we were told just to throw them out. This is the fear that archivists are living with. Each time an archivist approaches this obsolete media, the questions come: How many others are out there? How many are being thrown out because it is easier? How many are left? How long do I save them? If I am able to find a player/drive/ etc. will I be open the software that the data is formatted in? Will it even be playable? Are we missing decades of human knowledge? How long will this continue? How can archivists slow down the moving train of media change? Can archivists increase re-formatting awareness? Is reformatting my only option? Where does emulation stand? Who do I call? Who do I write? How do I make a difference in this loss that flies in the face of everything my profession holds dear?

A colleague, Tom Featherstone, once told a class “Archivists get paid for throwing things out.” After the horrific silence, he explained that we cannot save everything. Archivists place value on what is received, appraisal occurs and the kernels of importance are retained.

In today’s Digital Age, few are seeing the kernels, the wheat, and perhaps even the farm!

A recent IDC/UC San Diego study estimated that the average American is taking in 34 gigabytes of information per day. As an archivist, let’s think about the volume of data that is being created and disseminated. If even half of one percent were historically significant, we still have a large preservation problem.

The loss of corporate, academic and personal data from the late 1980s to the current time is tragic for future generations of historians, technologists, anthropologists and sociologists. We are nearing twenty-five years with little implementation of preservation processes. This is not to say that Archivists have not offered plans. This is to say that they are not being followed. In pure sales terms, we have NOT “sold” the crisis to the people. This is not to imply that the issue is not real, it means that the dry facts were not enough to convince people of the crisis. More facts had to be gathered. Now a true implementation plan with typical business practices needs to be created.

Here are the options:

1) Do nothing and continue to sweep up after mass dumps of data. Process what passively comes to us. Complain a little (or a lot) and do the best that we can with the little that we receive

2) Be moderately proactive to educate the general public on the loss of human knowledge. Start education workshops at local archives, issue press releases individually and work at the grassroots level to educate your donors and users.

3) Be passionately proactive and begin a coordinated media campaign aimed at the public and the computer industry to work with archivists, historians, sociologists and anthropologists to stop the destruction of electronic records on all media. Work this campaign hand in hand into a reformatting program that is easy. Much like the environmentalists needed to educate consumers (e.g. “Reduce Reuse Recycle”). Catchy phrases work.

One of the biggest complaints leveled against our largest member associations is that they do not get involved in the issues that are most impactful for our day to day work. There is NOTHING bigger to archives than this, right now. Member associations are built on exactly that, their members. We can choose to have a voice.

How each professional decides to act on this data is an individual choice, but a large percentage of archivists and other professionals impacted by this severe and irreparable data loss would be a dominant force in the media, to donors and to the computer industry.

For twenty five years, archivists have been that little chick crying about disaster. It is time we grew up and became the rooster at the farm, crowing for the populace to wake up.

In an age where music is so easily copied and accessed, it’s hard to imagine that any valuable recordings could ever be lost. But a new study predicts a grim future for millions of recordings across America.

The National Recording Registry was established ten years ago, following the passing of a congressional bill. The purpose of the NRR is to “maintain and preserve sound recordings and collections of sound recordings that are culturally, historically, and aesthetically significant, and for other purposes” (Public Law 104-474; H.R. 4846). Recently, the NRR released a 181-page report, The State of Recorded Sound Preservation in the United States: a National Legacy at Risk in the Digital Age. This report was the first “comprehensive, national-level study of the state of sound recording preservation ever conducted in the U.S.” 130 years since the invention of the phonograph, it’s about time the subject was addressed.

The report found that an estimated forty-six million recordings are held in American libraries, archives, and other institutions. Efforts made nationally in collecting and creating recordings are not matched by efforts to preserve them and this puts recordings of all types at stake. Furthermore, inconsistent local, state, and national laws have caused a “lack of national coordination… in addressing the challenges of preservation, professional education and public access.” The report states that not enough is being done to prevent a “permanent loss of irreplaceable sound recordings in all genres.”

Why does all of this matter? While most commercial recordings are not at stake at being lost, millions of recordings of performances, interviews, and broadcasts are at risk. So while the average Joe doesn’t have to worry about his precious Eagles and Zeppelin records ever being irreplaceably lost or damaged, scholars and historians face losing incredibly valuable resources. Someone conducting a study on race relations, for example, might use recordings of Minstrel shows as to demonstrate the willful acceptance and ignorance of racism in the late 19th century. A researcher for a film, be it a documentary or historical drama, might need to find old radio broadcasts for a project. The need for access to obscure recordings is far-reaching.

A huge part of the sound recording preservation problem is the “restrictive and anachronistic” copyright laws currently in place. Ironically, the copyright laws designed to protect recordings are essentially destroying them, by heavily restricting preservation methods. The report explains, “All US recordings, both commercially released and unpublished, created before February 15, 1972, are protected by a complex network of [copyright] laws.” As a result, any recording made before 1972 will not enter the public domain until 2067—95 years after the bill was passed. So, for example, a recording made in 1900 will not be able to legally be copied without permission until 2067, or 167 years after its creation. There’s no question that a considerable—if not irreparable—amount of decay will happen in that timeframe.

The good news is that copyright laws are not widely enforced, and library and archive facilities have some leeway in preservation and digitization. However, without legal rights to sound recordings, many institutions cannot procure the funding needed to restore and maintain their own collections.

Initiating a nation-wide study on the subject of sound preservation was a good first step on tackling the issue. What should be done next? Change the copyright laws, obviously. And while it would take years or decades to make copyright laws relevant to the digital age, librarians and archivists can take action in the meantime. The study states, “an individual representing one institution has noted that, unless or until instructed to cease and desist certain practices, his organization was compelled to ‘fly under the radar’ to support its mission.”

Awareness of the issue can also be a huge step in fighting loss of sound recordings. Educating record collectors and owners of original recordings on how to maintain and store their collections could go a long way in ensuring a better outlook for sound preservation.

In teaching multimedia archive,s I think about the future of our content constantly. Like a new mother, I fret for its security, growth and health. What is THE answer for our degrading media, emulsions, for our software obsolescence and our equipment falling down around our ears?

Recent discussions on the AMIA listserv brought new energy to this discussion and I wanted to put my spin on this. The subject line was “What’s Not Cool About Cold?” and it solicited some serious discussion about whether we have made a horrible mistake for a generation of archivists and content.

Jim Lindner argues that the imminent demise of tape players is more important in an archivist’s preservation decision than our focus on the imminent degradation of the media itself. The latter being our big decision to place much of our media in cool or cold storage. The group discussion mentioned the fact that many of our players are no longer supported by their manufacturers and the simple math that the lack of machines and the existing wear on their parts will not even cover the playback of the volume of archival tapes awaiting…migration? This hits a deep reality. Have we lost hundreds of thousands of hours of archival motion under our watch? Maybe even millions or billions of hours?

With the current demand from patrons increasing and the desire to use motion outtakes, news, etc. for historic documentation this is a disappointing state. Motion is powerful, emotional and immediate. It puts patrons in the experience in a way that to the visual human being is dramatic.

This important discussion on the listserv was a strong exchange which took place over several days. Other important points included the argument that the research on degradation and proper handling has made the tape degradation less of an issue and now the issue is playback for migration and access.

Other points included:

1) The need to get these gems a proper venue to be seen (whether due to money or rights issues)

2) The natural evolution of gathering obsolete machines from production facilities and creating some centralized site for archival institutions to use.

3) How to custom make parts so that we can maintain the machines so that we can transfer the content.

4) Wait for a miracle technological innovation to happen in the next 20-30 years so that you can migrate. (Keep them cool until then).

We come back again and again to the complexities of managing archival multimedia. The next generation of archivists will have changed their paradigm and released themselves from the desire to preserve the artifact. I am not condoning refusing to preserve all originals but our focus as far as video tape and digital files will be on content migration.

Other formats still have certain inherent value and that is a different blog posting!

Future archivists will have a much better long-term understanding of what is an artifact and what is intellectual content that needs to be migrated. They will be more adept to the rapid pace of format change and will HAVE to adapt quickly.

Not that I am faulting us. We are the transitory archival generation, the one that bridges the 100 year film format and the thumb drive.

I agree that the great research done in the field has helped us to minimize that problem, but it does seem that we took the eye off the ball a bit on migration. We cite lack of funds, lack of understanding of the urgency by the non-archival world, the frustration with equipment manufacturers, etc. It sounds like we need a development guru to raise funds, awareness and help to join all the key players for a collaboration. Many mention that the creators often do not prioritize preservation as they should. That is true, but our role as professionals is to educate.

Hand-wringing is not allowed! The energy that we have spent on that could have been put forth to an international collaboration with a real potential for migration, managing equipment, and innovation. One manufacturer can not do this alone, everyone needs to get on board and when I say “everyone” I mean:

Archivists
Creators (producers, studios, channels, directors, talent, writers, etc.)
Media
Equipment and Media Manufacturers
Technologists
Professional Associations
Funders

I think all of the above agree that the loss is imminent. The question is can we use our collective economy of scale to work together? Or maybe it is will we?

After teaching so many archival and technology classes, I began to realize the incredible depth and breadth of our loss of data. Over the last three decades billions of discs have been created and sold and presumably used. What has happened to these discs? To the data? If even 5% was worth saving for historical purposes, that is still about one and a half million discs to save and migrate. Has that been done?

We all know that the answer is “no.” So that means that we need to look at what is important and what level of effort is necessary to save it. I know that we can not save everything and I know that we would not want to. As Nik Cubrilovic mentioned in a recent Washington Post article entitled “Letting Data Die a Natural Death”: “Not only is a lot of this data not important, but do we really want to keep it? I certainly would not want a full account of everything I did in my youth sitting on a server somewhere. I am also certain that we do not want the record of our as a society time being documented and discovered by future civilizations based on Twitter messages.”

Yet, how much is lost that we have accepted and are wistful for? Some of my own graduate papers written on 5 1/4” floppies were not salvageable. That is of course, a personal loss that matters to NO ONE else but me.

What about the loss of communication histories in Presidential administrations? It took NARA six years just to process the Clinton presidential emails. (I am not sure how many people they had working on it, but that certainly sounds overwhelming!) Assuming that each succeeding administration will have geometrically more, what effort will it take to separate the diamonds from the coal? Add into that the constant evolution of tools like Instant Messaging, Texting, etc. and where will historians be able to turn to determine how national decisions were really made? Or have we EVER had that, even in a paper driven world of the past? Doesn’t everyone edit trails that reflect badly on them?

In the article “The E-Memory Revolution,” Jim Gemmell and Gordon Bell talk about the new “digital person” that has a “total recall” to their life as it is all in “e-memory.” They talk about patrons asking librarians about helping them to build new connections for them to their content. As an archivist, I ask, “In what format will their 20, 30, 40, or 50 year old history be?” How many of us can access our data from a phone that we had one year ago? These digital tools are wonderful BUT they are transitory.

They are primarily to transmit current records. Whether an email joke to all your friends or a tweet to meet someone at a concert. The problem is that we also have history making decisions ONLY in email form and new marketing ideas only documented on Twitter. So for future researchers, how will they take your 1000 new weekly emails and get the funding to sort through them?

Is it the same way that the 4 billion floppies manufactured by 2003 (according to one site which I can no longer find, how ironic!) and the 200 million Zip discs manufactured in 1999 alone (according to that same site) were handled? My 15 years of experience tells me that only a fraction of a percent might have been migrated. Is that a loss? Maybe we will never know? Because without the data we don’t know what we don’t know. Is everyone comfortable with that?

Fast, easy and cheap. This was the title of a recipe book my mother created for me when I left home and entered the “real world,” but it can also describe the ideal technology for dealing with digital obsolescence. Who wouldn’t want technology that is fast, easy and cheap? It’s straightforward (“I’d like to order the archival program that is fast, easy and cheap, please”) and it certainly rolls off the tongue nicely. But what exactly does fast, easy and cheap mean?

Let’s tackle each element on its own:
Fast – You can quickly enter information.
Easy – You know exactly what information to enter and where to find it.
Cheap – You can do all this within your ridiculously slashed budget.

Ideal technology solution – DONE! – just like Mom’s enchilada casserole. What’s that, you ask? It’s one of Mom’s fast, easy and cheap recipes: every ingredient comes from a can, my husband hates it and there’s nothing remotely close to authentic about it (unless you think Hormel chili is Mexican food).

Hmm… maybe fast, easy and cheap isn’t the ideal technology after all… Maybe the solution to digital obsolescence is more complex.

For starters, what is digital obsolescence? Neither the SAA glossary nor Merriam-Webster offers a definition, but the latter defines obsolete as “no longer in use.” The Jem Report (http://www.thejemreport.com/content/view/33/74) provides a bit more meaning: “Obsolete means that the equipment in question is no longer valid or able to be used in the modern context.” (The Report also declares obsolete “the most misused word in the history of computing.”)

So if digitally obsolete files can no longer be used in the modern context, than the ideal technology is one that can read and display these files. Well that was easy! (Not to mention fast, and since you’re reading this for free… you get the idea.)

Not so fast. Even if a program reads and displays the files, it may not render the files the same way. So what? Maureen Pennock (http://digitalarchiving.wordpress.com/2009/04/16/authenticity-digital-obsolescence) provides several excellent examples as to why this may concern archivists and it all comes down to authenticity. Programs that simply read and display may alter document formatting, change fonts, remove headers, alter formulas and delete hidden characters; all of which may be valuable to future researchers.

Even the simple act of preserving content by printing it onto paper may cause loss of data, such as mathematical equations embedded in Excel files; loss of image quality, such as an original piece of digital art created at 600 dpi and printed from a desktop printer that can only output 150 dpi; or loss of original point of view, such as a PowerPoint presentation meant to be viewed on a computer screen.

Suddenly the simple issue of reading obsolete files becomes a more complex issue of preserving digital authenticity.

Remember how there’s nothing authentic in enchilada casserole? Sure it’s fast, easy and cheap; but it sacrifices authenticity. Is the same true for technology? In the ideal world we want fast, easy and cheap, but in the real world, we may have to pick one.

Technology relative to the field of Archives is a much-needed friend as well as an enemy of sorts. The merits of technology are easy to note – probably most importantly the quick access it provides to Archives both internally and externally. Items are much easier to keep track of and to locate internally. When items are made available to the public online, research is easier, time is saved, and so are many other resources. There’s no doubt that technology has opened Archives up to the world.

The speed with which information can be uploaded and made available is astounding – whether from a digital camera, digital video recorder, or general file type (Word, Excel, etc.). Gone are the days of waiting days, if not weeks, to update your files and information. Time to clear the library of that old hand-written card catalog. Or is it?

The enemy moniker comes into play in many ways, though, making technology difficult for many Archives and Archivists to embrace. Cost is one of the major issues. Small and large, corporate and not-for-profit Archives struggle to make initial hardware, software, and programming purchases – never mind upgrades and constant format changes and improvements. Not only is the hard cost of upgrades an issue, but also the soft costs of redoing, reformatting, and migrating file types.

Nevermind power failures, unreadable files or disks, or fried hard drives. Not that Mother Nature can’t do plenty of damage to our trusty old hard copies and originals of everything – we can’t make it sound like technology is our only worry as Archivists!

Technology isn’t just software and hardware either. A whole new world has been opened up to Archives with various social media options available to inform others about activities within your Archives as well as promoting and marketing your Archives or services. This blog is one of them! Others include Facebook, Twitter, LinkedIn, MySpace, etc. The drawback to these relatively new technologies is the time it takes to keep them fresh with regularly updated materials. Another consideration is to decide what you hope to gain – new business, cheap PR, disseminating information as a gesture of goodwill?

While Technology offers plenty of concerns to the Archives community, you can’t really beat all of the advantages it offers us. So, if you’ll excuse me, I’m going to run and do some online research for a larger scanner for my next project!

Archivists have special needs when it comes to choosing a file format for storage. We have all heard of or endured the nightmares of file format obsolescence. There are no guarantees when it comes to file format longevity, but here are some tips to help you choose a file format that stands a chance of still being readable in a few years.

Consider your needs. Are there legal or other requirements that make it necessary to preserve the document in an unedited version? Is it important to retain the formatting or structure of a document or is the preservation of the content the primary concern? If your requirements dictate that a format must retain the original presentation or associated metadata, you may not be able to use an open source format as a replacement.

Don’t be an early adopter. Wait for a file format to become widely adopted before converting all of your files. When a file format becomes commonly used by a variety of software manufacturers, it stands a better chance of long-term survival.

Investigate your potential new file format. Check user groups & forums for their comments about the format. Talk to colleagues that have taken the leap to see if they have encountered any difficulties.

Check the reputation of the format’s originator. Is the format proprietary or open source? If proprietary, is the company stable and reliable for support questions that may occur? Is the format compatible with previous versions? Will it continue to be compatible with future software versions? If the vendor has a history of discontinuing support for previous software versions or formats, can you afford to continue updating to new versions in the future?

Take time to consider your options carefully. What looks like a stable format today can easily be made obsolete by advances in technology. Keep track of new trends in technology to stay aware of your future migration needs.

Sources:
National Archives: Digital Preservation
Northeast Document Conservation Center