A few weeks ago I was honored to attend the National Digital Stewardship Alliance meeting. The NDSA was planned by the Library of Congress as part of their NDIIP project.  There were more people there than I expected and it was a humbling experience to hear some of the brightest and most creative brains in Digital Preservation speak.

The high volume of information was overwhelming.  I spent more than six hours at the end of the conference compiling a PowerPoint of the important research highlights.   A small sample of this information is included below.

There were many wonderful presentations giving case studies on how institutions used their own creativity to try and enhance the longevity or migratability (new word?) of their digital files.  The amazing work often was done on a shoestring which though unfortunate, also forced a certain level of imagination and invention.

A few examples are:

Jack Brighton, of campus radio station WILL, gave a wonderful presentation on what a small station is doing to make their civil rights collection more accessible.

Kickstarter.com did a great presentation on how they are helping arts projects get funded and we hope that as they branch into community work that digital preservation might fit into that.

The UK Web Archiving project covered some of the complexities and true effort that it takes to try and tackle capturing the online history of its nation.   http://www.webarchive.org.uk/ukwa/

- As of December 2010 – 9 million sites with .uk, probably 1M more

- 10,027 websites archived

- Need skills in Linux, Java, Hadoop, and SOL

5 keys processes to web archiving

- Selection

- Harvesting

- Storage

- Preservation

- Access

http://www.webarchive.org.uk/ukwa/ngram/

So after taking in all this good information, what is it that I have left the conference with?

People just like us are doing some wonderful problem solving out there.  There is some potential being unlocked, but there is so much to do.

As I see it the Action Items are:

1)    Greater broadcasting of the successful case studies for migration and open solutions.

2)    Training classes in how to boil this down for each type of format/issue.  The NDSA Outreach group held a session called “Digital Preservation in a Box”.  This is the beginning of standardizing the tools that we need.

3)    Overarching education to information and production professionals, as well as, the general public about the dangers of digital fragility and the need for migration (at the least).

I have mentioned to my classes for years that future anthropologists, sociologists and historians will have little to sift through from the late 20^th century.

Some of it is being worked on by archivists now but much is gone.  Let’s keep making progress so that the future of our current history is not lost, like the way of silent films.

More informational tidbits from NDSA:

Other Great Projects

—       ThatCamp.org

—       http://www.scola.org/scola/sampledigitalarchive.aspx

NYPL Labs

—     http://search.creativecommons.org/?q=nypl+map+rectifier&sourceid=Mozilla-search

—     http://menus.nypl.org/

Archiving Facebook

Grad student designed Firefox add-on for individual archiving of Fb.

www.Bit.ly/archivefb

Preserving Virtual Worlds

—      www.ideals.illinois.edu/handle/2142/17097

Cool Tools

—  http://www.google.com/landing/historypin/

—  http://blogs.yu.edu/cpa/2011/02/23/open-source-video-platforms-kaltura-vs-entermedia/

Great Quotes

—    JackBrighton”(DAM) is more like an appliance than an Ecosystem.”

—    Michael Nelson “We need to raise the level of user expectations.”

—    Michael Nelson “In all good computer science functions you solve the problem through indirection.”

—    Wheatley and Frieze “The world does not change one person at a time.  It changes as networks of relationships form among people        who discover they share a common cause and vision of what’s possible.”

—    Tim O’Reilly(?) “Teach preservation as a mindset.  Bake this into the tools.”

New Phrases

—    Social Curation

—    Metadata Ecologists

There has been much written recently about digital fragility. Researchers and archivists have heard dueling longevity and futuristic projections. In trying to push this dire need without appearing like Chicken Little, I have embarked on serious primary research to expose the sheer volume of the problem. The in-depth article will be coming out in a professional journal within the year. Until then, I felt that I needed to speak up a bit about the need for our activism.

Between my teaching digital archiving courses and my work with clients, this issue has been prevalent in each work day. In fact, while re-processing an archives for a client, a case of 5 ¼” floppy discs were found. No one in-house knew exactly if these were of value, what was on them or even if they were created by the organization. When we offered to open them on a computer with a floppy drive, we were told just to throw them out. This is the fear that archivists are living with. Each time an archivist approaches this obsolete media, the questions come: How many others are out there? How many are being thrown out because it is easier? How many are left? How long do I save them? If I am able to find a player/drive/ etc. will I be open the software that the data is formatted in? Will it even be playable? Are we missing decades of human knowledge? How long will this continue? How can archivists slow down the moving train of media change? Can archivists increase re-formatting awareness? Is reformatting my only option? Where does emulation stand? Who do I call? Who do I write? How do I make a difference in this loss that flies in the face of everything my profession holds dear?

A colleague, Tom Featherstone, once told a class “Archivists get paid for throwing things out.” After the horrific silence, he explained that we cannot save everything. Archivists place value on what is received, appraisal occurs and the kernels of importance are retained.

In today’s Digital Age, few are seeing the kernels, the wheat, and perhaps even the farm!

A recent IDC/UC San Diego study estimated that the average American is taking in 34 gigabytes of information per day. As an archivist, let’s think about the volume of data that is being created and disseminated. If even half of one percent were historically significant, we still have a large preservation problem.

The loss of corporate, academic and personal data from the late 1980s to the current time is tragic for future generations of historians, technologists, anthropologists and sociologists. We are nearing twenty-five years with little implementation of preservation processes. This is not to say that Archivists have not offered plans. This is to say that they are not being followed. In pure sales terms, we have NOT “sold” the crisis to the people. This is not to imply that the issue is not real, it means that the dry facts were not enough to convince people of the crisis. More facts had to be gathered. Now a true implementation plan with typical business practices needs to be created.

Here are the options:

1) Do nothing and continue to sweep up after mass dumps of data. Process what passively comes to us. Complain a little (or a lot) and do the best that we can with the little that we receive

2) Be moderately proactive to educate the general public on the loss of human knowledge. Start education workshops at local archives, issue press releases individually and work at the grassroots level to educate your donors and users.

3) Be passionately proactive and begin a coordinated media campaign aimed at the public and the computer industry to work with archivists, historians, sociologists and anthropologists to stop the destruction of electronic records on all media. Work this campaign hand in hand into a reformatting program that is easy. Much like the environmentalists needed to educate consumers (e.g. “Reduce Reuse Recycle”). Catchy phrases work.

One of the biggest complaints leveled against our largest member associations is that they do not get involved in the issues that are most impactful for our day to day work. There is NOTHING bigger to archives than this, right now. Member associations are built on exactly that, their members. We can choose to have a voice.

How each professional decides to act on this data is an individual choice, but a large percentage of archivists and other professionals impacted by this severe and irreparable data loss would be a dominant force in the media, to donors and to the computer industry.

For twenty five years, archivists have been that little chick crying about disaster. It is time we grew up and became the rooster at the farm, crowing for the populace to wake up.

In an age where music is so easily copied and accessed, it’s hard to imagine that any valuable recordings could ever be lost. But a new study predicts a grim future for millions of recordings across America.

The National Recording Registry was established ten years ago, following the passing of a congressional bill. The purpose of the NRR is to “maintain and preserve sound recordings and collections of sound recordings that are culturally, historically, and aesthetically significant, and for other purposes” (Public Law 104-474; H.R. 4846). Recently, the NRR released a 181-page report, The State of Recorded Sound Preservation in the United States: a National Legacy at Risk in the Digital Age. This report was the first “comprehensive, national-level study of the state of sound recording preservation ever conducted in the U.S.” 130 years since the invention of the phonograph, it’s about time the subject was addressed.

The report found that an estimated forty-six million recordings are held in American libraries, archives, and other institutions. Efforts made nationally in collecting and creating recordings are not matched by efforts to preserve them and this puts recordings of all types at stake. Furthermore, inconsistent local, state, and national laws have caused a “lack of national coordination… in addressing the challenges of preservation, professional education and public access.” The report states that not enough is being done to prevent a “permanent loss of irreplaceable sound recordings in all genres.”

Why does all of this matter? While most commercial recordings are not at stake at being lost, millions of recordings of performances, interviews, and broadcasts are at risk. So while the average Joe doesn’t have to worry about his precious Eagles and Zeppelin records ever being irreplaceably lost or damaged, scholars and historians face losing incredibly valuable resources. Someone conducting a study on race relations, for example, might use recordings of Minstrel shows as to demonstrate the willful acceptance and ignorance of racism in the late 19th century. A researcher for a film, be it a documentary or historical drama, might need to find old radio broadcasts for a project. The need for access to obscure recordings is far-reaching.

A huge part of the sound recording preservation problem is the “restrictive and anachronistic” copyright laws currently in place. Ironically, the copyright laws designed to protect recordings are essentially destroying them, by heavily restricting preservation methods. The report explains, “All US recordings, both commercially released and unpublished, created before February 15, 1972, are protected by a complex network of [copyright] laws.” As a result, any recording made before 1972 will not enter the public domain until 2067—95 years after the bill was passed. So, for example, a recording made in 1900 will not be able to legally be copied without permission until 2067, or 167 years after its creation. There’s no question that a considerable—if not irreparable—amount of decay will happen in that timeframe.

The good news is that copyright laws are not widely enforced, and library and archive facilities have some leeway in preservation and digitization. However, without legal rights to sound recordings, many institutions cannot procure the funding needed to restore and maintain their own collections.

Initiating a nation-wide study on the subject of sound preservation was a good first step on tackling the issue. What should be done next? Change the copyright laws, obviously. And while it would take years or decades to make copyright laws relevant to the digital age, librarians and archivists can take action in the meantime. The study states, “an individual representing one institution has noted that, unless or until instructed to cease and desist certain practices, his organization was compelled to ‘fly under the radar’ to support its mission.”

Awareness of the issue can also be a huge step in fighting loss of sound recordings. Educating record collectors and owners of original recordings on how to maintain and store their collections could go a long way in ensuring a better outlook for sound preservation.

Earlier this month, the Library of Congress announced that it would house every “tweet” ever posted on Twitter. Every 140-character-or-less blurb on the site is now part of the vast LoC archives. This got me thinking: what are the issues at hand in archiving social networking sites? And why is it important?

Recently, while cleaning out my apartment, I found a relic of primitive social networking—a printed-out Facebook message from 2005. Nostalgia instantly struck. Five years ago, Facebook was [thefacebook], with a much simpler interface. A toolbar on the left listed the humble features of the relatively new site: My Profile, My Groups, My Friends, My Away Messages. Clearly, Facebook was trying to emulate MySpace —which was then by far the preferred means of social networking.

The past five years of my Facebook existence flashed before my eyes. I remembered when the site was only accessible to students at selected universities, when users could upload only one photo, when “the wall” was merely a text box that anyone could edit. A Facebook before FarmVille, “like” buttons, and mini-feeds.

With hundreds of millions of users, there are many reasons to archive social networking sites. First, for historical documentation—millions of photos are uploaded to the site every week, capturing the trends of the present era. Additionally, no matter how worthless each status update might seem, taken as a whole they reflect our reactions to modern events. It would be worth keeping a log of Facebook updates from the night of the 2008 election, or as the news broke about the earthquake in Haiti, or even as the Tiger Woods scandal unfolded.

Second, following the history of social networking sites offers insight into the development and perceptions of Web 2.0. What was it that made Facebook become so much more popular than MySpace ? How has Facebook increased the average person’s awareness of internet safety and privacy? Studying the evolution of such sites, web developers can predict future trends in online technology.

Third, social networking sites are paramount to the way we communicate in the 21st century. How do we choose to represent ourselves online? Everyone has that one friend who reveals too much personal information (Matt Maclean: too much vodka + ice cream = the great pukefest of 2010). There’s the perpetual Facebook drama, including my favorite, the Passive Aggressive Status Update (Alexis Cooper thinks some people need to get over themselves!!!) And how has Facebook changed our interpersonal relationships? At some point in their cyber-existence, most people have undoubtedly become angry or hurt or jealous over something said on Facebook. Relationships that may end in real life passively continue online—the psychological implications of which are new to our generation.

But how should sites like Facebook be archived, and to what extent?

The Internet Archive has preserved 150 billion web pages, dating back to 1996. Using web crawlers to archive websites, the IA serves as a publicly-accessible digital library, allowing users to revisit older versions of their favorite sites. Documenting and preserving static websites such as aol.com is relatively easy because they are open to the public. Public profiles on MySpace and Twitter are also easily accessed. But how do web crawlers get inside websites, like Facebook that require login information to view its content?

It’s not easy. The Internet Archive has developed a software application called Archive-It, which allows organizations to make digital copies of their own websites. As more and more universities, corporations, and libraries create their own Facebook pages, they can now archive them. While this is great news for individual organizations, it doesn’t include the interactions of ordinary Facebook users—the cyber chatter that documents our day-to-day lives.

Recently, the White House announced plans to preserve its social networking content. Media capturing is done using software applications and daily screen shots, storing text, graphics, audio and video in the context in which they were originally presented. Once this project gets off the ground, it can serve as a model for other organizations.

But what about the interactions of everyday Americans—how will their voices be recorded? Perhaps the LoC or the Smithsonian could begin a digital initiative to capture people’s online identities. An open call for submissions, asking Facebook users to send in screenshots of their profiles, walls, photos, or feeds, would be a simple and effective start. Similar projects, including the September 11 Digital Archive and the Hurricane Digital Memory Bank, were created to obtain Americans’ reactions to history-making events.

It’s difficult to say if a comprehensive range of Facebook users would be willing to submit their personal information to an archive. However, thousands are already sending in their screenshots to blogs, including Failbook and Lamebook, which capture people’s regrettable and embarrassing Facebook moments.

No matter how trivial they may seem, social networking sites are a defining part of our generation—and that’s something worth preserving.

Further reading:

Back when dinosaurs roamed the earth, my father brought home our first computer. It was a Mac. I can’t recall which model it was, but it was an all-in-one box with a screen the size of a small Kleenex box (I only wish I was joking!). Initially I was suspicious of this computer: having been raised on a steady diet of science fiction and comic books, I knew what computers were capable of. But my father convinced me that computers are only as smart as the person who programs them, so I gave in, turned it on, mastered the mouse and became addicted to computer gaming.

There was one game in particular that I liked. I can’t remember what it was called, but I created a group of witches, elves and trolls and we went on adventures: slayed dragons, defeated evil overlords, rescued princesses – that sort of thing. It was like a single-player, kiddie computer version of Dungeons & Dragons. I loved it, but I was also very bad at it. The computer won every time.

Then one day I had a brilliant idea: I would save my game, build up amazing gaming skills, go back to the game and beat the computer! It made perfect sense to my 12-year old mind. Pesky questions such as how I would obtain these awesome skills and how long it would take never crossed my mind. So I saved the game and did other things on the computer. Around that same time a relative gave me a book on calligraphy and I took that up, eventually becoming so consumed by it that I forgot all about my saved game on the computer.

Years passed by, Dad brought home a new computer and the computer with my saved game was relegated to the basement. Eventually I went to college, got a job and moved away. But before I did I asked Dad if I could take computer with me. He said sure (and probably though, “Yes! That useless thing is finally out of my house!”). Now it sits in my basement, with a game saved on it since 1986.

What do I do with it? How do I access it? Why should anyone care? I admit, that last one is a great question. But what if you were a researcher of late 20th century pastimes? What if you wanted to study how children spent their free time or how they experienced the first personal computers? Suddenly the box collecting dust in my basement is more interesting.

But what are the chances that the library or archive will have the technology to access, preserve and make available the computer’s data? We all know technology has changed rapidly since the mid-1980s (5-1/4 inch floppies anyone?), but there are also different standards. My first computer was a Mac. So not only must a library or archive be able to handle old data, it must be able to handle data in a variety of formats.

Nor is the issue likely to resolve itself any time soon. Although for many business applications the PC is the only option, many in the arts still use a Mac. Libraries and archives that cannot access Mac data will severely limit the research that can be conducted on late 20th and early 21st century digital art – and that’s just one example.

Yet I doubt any library or archive will “have it all.” There’s simply not enough funding to plan for all the different storage mediums and files. One option may be libraries and archives developing expertise in niche areas – for example, a library may have the technological ability to provide access to IBM AS/400 files from the 1990s but not for digital arts files from the 2000s – but this is unlikely as it is a direct contradiction to current collection development policies. Libraries and archives collect on specific subject matter or people, but most people don’t make technology decisions based on which files will be easier for archivists to preserve and make accessible in the future.

To meet these various accessibility needs, we may see the emergence of a new type of vendor: one that specializes in access and preserving old data in a variety of formats (for a fee, of course). In a way, this is already happening. Small libraries participate in union catalogs, and the host often has digitization capabilities. DALNET is a great example: the Library & Archives of the Detroit Institute of Arts had DALNET convert 45 rpm records to digital format that can now be accessed via the library’s online catalog.

In the meantime, I plan to keep my old computer in the basement. Who knows? Maybe some day I will develop amazing gaming skills.

After teaching so many archival and technology classes, I began to realize the incredible depth and breadth of our loss of data. Over the last three decades billions of discs have been created and sold and presumably used. What has happened to these discs? To the data? If even 5% was worth saving for historical purposes, that is still about one and a half million discs to save and migrate. Has that been done?

We all know that the answer is “no.” So that means that we need to look at what is important and what level of effort is necessary to save it. I know that we can not save everything and I know that we would not want to. As Nik Cubrilovic mentioned in a recent Washington Post article entitled “Letting Data Die a Natural Death”: “Not only is a lot of this data not important, but do we really want to keep it? I certainly would not want a full account of everything I did in my youth sitting on a server somewhere. I am also certain that we do not want the record of our as a society time being documented and discovered by future civilizations based on Twitter messages.”

Yet, how much is lost that we have accepted and are wistful for? Some of my own graduate papers written on 5 1/4” floppies were not salvageable. That is of course, a personal loss that matters to NO ONE else but me.

What about the loss of communication histories in Presidential administrations? It took NARA six years just to process the Clinton presidential emails. (I am not sure how many people they had working on it, but that certainly sounds overwhelming!) Assuming that each succeeding administration will have geometrically more, what effort will it take to separate the diamonds from the coal? Add into that the constant evolution of tools like Instant Messaging, Texting, etc. and where will historians be able to turn to determine how national decisions were really made? Or have we EVER had that, even in a paper driven world of the past? Doesn’t everyone edit trails that reflect badly on them?

In the article “The E-Memory Revolution,” Jim Gemmell and Gordon Bell talk about the new “digital person” that has a “total recall” to their life as it is all in “e-memory.” They talk about patrons asking librarians about helping them to build new connections for them to their content. As an archivist, I ask, “In what format will their 20, 30, 40, or 50 year old history be?” How many of us can access our data from a phone that we had one year ago? These digital tools are wonderful BUT they are transitory.

They are primarily to transmit current records. Whether an email joke to all your friends or a tweet to meet someone at a concert. The problem is that we also have history making decisions ONLY in email form and new marketing ideas only documented on Twitter. So for future researchers, how will they take your 1000 new weekly emails and get the funding to sort through them?

Is it the same way that the 4 billion floppies manufactured by 2003 (according to one site which I can no longer find, how ironic!) and the 200 million Zip discs manufactured in 1999 alone (according to that same site) were handled? My 15 years of experience tells me that only a fraction of a percent might have been migrated. Is that a loss? Maybe we will never know? Because without the data we don’t know what we don’t know. Is everyone comfortable with that?

Archivists have special needs when it comes to choosing a file format for storage. We have all heard of or endured the nightmares of file format obsolescence. There are no guarantees when it comes to file format longevity, but here are some tips to help you choose a file format that stands a chance of still being readable in a few years.

Consider your needs. Are there legal or other requirements that make it necessary to preserve the document in an unedited version? Is it important to retain the formatting or structure of a document or is the preservation of the content the primary concern? If your requirements dictate that a format must retain the original presentation or associated metadata, you may not be able to use an open source format as a replacement.

Don’t be an early adopter. Wait for a file format to become widely adopted before converting all of your files. When a file format becomes commonly used by a variety of software manufacturers, it stands a better chance of long-term survival.

Investigate your potential new file format. Check user groups & forums for their comments about the format. Talk to colleagues that have taken the leap to see if they have encountered any difficulties.

Check the reputation of the format’s originator. Is the format proprietary or open source? If proprietary, is the company stable and reliable for support questions that may occur? Is the format compatible with previous versions? Will it continue to be compatible with future software versions? If the vendor has a history of discontinuing support for previous software versions or formats, can you afford to continue updating to new versions in the future?

Take time to consider your options carefully. What looks like a stable format today can easily be made obsolete by advances in technology. Keep track of new trends in technology to stay aware of your future migration needs.

Sources:
National Archives: Digital Preservation
Northeast Document Conservation Center