There has been much written recently about digital fragility. Researchers and archivists have heard dueling longevity and futuristic projections. In trying to push this dire need without appearing like Chicken Little, I have embarked on serious primary research to expose the sheer volume of the problem. The in-depth article will be coming out in a professional journal within the year. Until then, I felt that I needed to speak up a bit about the need for our activism.

Between my teaching digital archiving courses and my work with clients, this issue has been prevalent in each work day. In fact, while re-processing an archives for a client, a case of 5 ¼” floppy discs were found. No one in-house knew exactly if these were of value, what was on them or even if they were created by the organization. When we offered to open them on a computer with a floppy drive, we were told just to throw them out. This is the fear that archivists are living with. Each time an archivist approaches this obsolete media, the questions come: How many others are out there? How many are being thrown out because it is easier? How many are left? How long do I save them? If I am able to find a player/drive/ etc. will I be open the software that the data is formatted in? Will it even be playable? Are we missing decades of human knowledge? How long will this continue? How can archivists slow down the moving train of media change? Can archivists increase re-formatting awareness? Is reformatting my only option? Where does emulation stand? Who do I call? Who do I write? How do I make a difference in this loss that flies in the face of everything my profession holds dear?

A colleague, Tom Featherstone, once told a class “Archivists get paid for throwing things out.” After the horrific silence, he explained that we cannot save everything. Archivists place value on what is received, appraisal occurs and the kernels of importance are retained.

In today’s Digital Age, few are seeing the kernels, the wheat, and perhaps even the farm!

A recent IDC/UC San Diego study estimated that the average American is taking in 34 gigabytes of information per day. As an archivist, let’s think about the volume of data that is being created and disseminated. If even half of one percent were historically significant, we still have a large preservation problem.

The loss of corporate, academic and personal data from the late 1980s to the current time is tragic for future generations of historians, technologists, anthropologists and sociologists. We are nearing twenty-five years with little implementation of preservation processes. This is not to say that Archivists have not offered plans. This is to say that they are not being followed. In pure sales terms, we have NOT “sold” the crisis to the people. This is not to imply that the issue is not real, it means that the dry facts were not enough to convince people of the crisis. More facts had to be gathered. Now a true implementation plan with typical business practices needs to be created.

Here are the options:

1) Do nothing and continue to sweep up after mass dumps of data. Process what passively comes to us. Complain a little (or a lot) and do the best that we can with the little that we receive

2) Be moderately proactive to educate the general public on the loss of human knowledge. Start education workshops at local archives, issue press releases individually and work at the grassroots level to educate your donors and users.

3) Be passionately proactive and begin a coordinated media campaign aimed at the public and the computer industry to work with archivists, historians, sociologists and anthropologists to stop the destruction of electronic records on all media. Work this campaign hand in hand into a reformatting program that is easy. Much like the environmentalists needed to educate consumers (e.g. “Reduce Reuse Recycle”). Catchy phrases work.

One of the biggest complaints leveled against our largest member associations is that they do not get involved in the issues that are most impactful for our day to day work. There is NOTHING bigger to archives than this, right now. Member associations are built on exactly that, their members. We can choose to have a voice.

How each professional decides to act on this data is an individual choice, but a large percentage of archivists and other professionals impacted by this severe and irreparable data loss would be a dominant force in the media, to donors and to the computer industry.

For twenty five years, archivists have been that little chick crying about disaster. It is time we grew up and became the rooster at the farm, crowing for the populace to wake up.

In years of teaching visual indexing and being called in to create metadata schemas, I have seen some crazy attempts at description.

Sometimes we have been involved from the beginning developing thesauri of specialized terms for a collection, more often we are called in to fix existing records.

As I roll up my sleeves to tackle either project, I often wonder why organizations do not know more about what they want.

I come down to the same answer that permeates our profession as a whole. The majority of people do not understand the work that goes into providing quality. In our current era of fast and cheap; people have lost the quality aspect almost completely. When they can not successfully execute an accurate search in their database, then they call us to fix it. I am absolutely happy to do so, but make no mistake, I wish for that collection to have done it right the first time; rather than to have called us after hundreds of hours of wasted work. Quality becomes a feature of importance often only after a failure rather than as a preventative measure.

As I tell my classes, let’s talk about why doing it right is rarely done:

1) Illusion – “Everyone is digitizing” is akin to what your mother taught you as a child. “If everyone jumped off a bridge would you do it too?” Many Asset Management Companies sell short the highest cost of digitization which is (dun, dun, dun) linking the metadata to the record. The metadata needs to mean something.

I once saw a vendor selling his “automatic indexing” system. I stopped to chat with him. His product, he told me, will negate needing a human to index. As this is one of our services, I thought that I had better pay attention. He proudly told me that the video clips that he was showing me worked off of closed captioning. I was glad to know that I was not out of business. If you have ever viewed closed captioning, it is a fantastic service to those hearing impaired but it is far from error free. Aside from the many spelling errors within this due to the pressure of typing the words as a show airs (for live shows), there is no intellectual analysis of what is being said and how it relates to the visual.

If an actor said about a child, “Her temperature is 105 degrees!” Assuming the spelling was correct, that is all that the search tool would allow for. A professional indexer could include “Fevers, Childhood Illnesses, Sickness, etc”. This extra analysis would allow a successful search. Most user’s would not find that video clip by looking for “temperature” and they might not know it was a child if that is what they wanted. They would have to pull up the clips and view them. If your collection is going to stay very small, maybe this kind of quality will not matter to you.

For some, I worry that when management, tax payers or a municipality sees bulky systems with little relevance in results, they will certainly shudder at writing more checks for the system or archive.

2) Internal Pressure – “Everything needs to be digitized”
We see this pressure to digitize everything without a clear plan for prioritization. A serious needs assessment is required to be done to understand what needs to be digitized, why and what needs to be researched and described.

I have often told my students that I would rather misfile a photo negative in a physical drawer than have misinformation on a digital record. I am more likely to find it again in the drawer than in a large database.

3) Money – “Scanners are cheap, how much could it cost?”
Money is tight and people are even more apt to cut corners now. It is always cheaper and more accurate to plan something out and do it right rather than to try and fix it afterwards.

Building a business case for the step by step process of tracking assets, designing metadata, the costs of hardware/software/maintenance, training, etc. is often looked at as daunting or impossible. It is not. You have to think like a cost accountant to spell out the savings and efficiency gained. There is also often a publicity component to having an organized and highly accessible collection. This is something that needs to be built in to the value.

4) Ignorance of Computational Linguistics/Human Computer Interfaces/ Usability Studies/Search Strategies/Term Linking/(More) – “Just throw some keywords on it.”

Many times I have seen upper management wave their hands in the air as if with a magic wand and say “Just get it done.” Unfortunately, I am too old to believe in the magic wand and hard work is the only way to create a successful search tool. When I say “hard work”, I actually mean really, really hard work. Research, focus groups, linguistic analysis, understanding search tool limitations, etc. all play a part in quality design.

Along these lines, there is an interesting project that was all over the news. IBM has a team from their labs that have designed a computer to compete on Jeopardy. PBS – Nova had a documentary on it and many of the things that the lead researcher, Dr. Ferrucci, mentioned in that documentary is relevant to our field. The primary comment that caught my interest was when he discussed the fact that a computer competing on Jeopardy can be fed thousands of background documents but they have to work very hard to understand the actual question being asked. So they have the answer, they just do not know the question.

The human brain “gets” the context of place and language. Computers have not yet mastered this. Those brilliant connections of slang, historical context, cultural cues, body language, etc. are a tremendous gift that humans have.

I have stated for years that I wished our culture valued the human brain as much as technology. Watson’s project is interesting and what it proved on Jeopardy is just a portion of what it will prove going forward.

Until then, we who aim to direct searchers into exactly the video clip, manuscript or image that they desire need to value our brains and find better ways to sell our skills.

My indexers know that “Picket Fences” have a certain lifestyle context. Automation or even off-shore indexers do not know that and we can do so much better than cutting corners on core concepts.

Let’s use the gifts that technology gives to us. The ability to link, create synonyms, cross-reference records, stream clips, etc. are all exciting tools and work best in conjunction with a well thought out plan designed by a human brain.

Good luck Dr. Ferrucci but I am not sure that it is Watson that is on trial but your brain.

Part I: Introduction

A conversation I seem to have a lot these days is discussing the use and instruction of social media, specifically for archival and library institutions. One particular topic that I keep coming back to over and over again in these conversations is that there is a huge push for institutions to use social media, with this push intensified by conferences and professional organizations (to name a few outlets). These outlets heavily advertise posters, panels and classes (to name a few methods) that teach professionals the hows of social media and networking with specific illustration of the more popular social media tools without really explaining the whys.

This in and of itself is not a bad thing. Last winter, Alexis Braun Marks, Kim Schroeder and I presented at AMIA‘s yearly conference on this very subject. Our topic, “When Are New Technologies For You?” was an attempt to give a general overview of what social media is and why it should be used while illustrating a few of the big players in the social networking world. Our audience poll at the beginning of our presentation only enforced what we knew from our research: Most institutions are desperate to get on the social media bandwagon and know that they should, but they have no idea WHY they should or how to go about doing it. Then what happens is that many institutions end up doing one of two things: they join every social network under the sun and then forget about it or they just ignore the siren call of social media in the first place, artificially secure that they don’t need it in the first place.

Therein, I believe, lies the problem: we have the knowledge of the hows but not necessarily the whys. In the the last year, since our presentation at AMIA, I’ve paid heavy attention to professional organizations and communities and noticed the rise in the offerings in classes on social media methodologies (good) but no real explanation as to they whys (bad). It is becoming widely accepted that these tools are to be and are being integrated into professional job descriptions and daily use, but no one seems to be clearly explaining why these seems to be so important.

This series is going to be a musing of and attempt to list and explain reasons of the WHY libraries and archives need to be using social media while hopefully doing away with marketing buzzwords and jargon. But before we start, let me offer up at least one simple reason as to why libraries and archives need to use social media:

It’s fun!

Next week: Part II: Social Media Simply Explained

In teaching multimedia archive,s I think about the future of our content constantly. Like a new mother, I fret for its security, growth and health. What is THE answer for our degrading media, emulsions, for our software obsolescence and our equipment falling down around our ears?

Recent discussions on the AMIA listserv brought new energy to this discussion and I wanted to put my spin on this. The subject line was “What’s Not Cool About Cold?” and it solicited some serious discussion about whether we have made a horrible mistake for a generation of archivists and content.

Jim Lindner argues that the imminent demise of tape players is more important in an archivist’s preservation decision than our focus on the imminent degradation of the media itself. The latter being our big decision to place much of our media in cool or cold storage. The group discussion mentioned the fact that many of our players are no longer supported by their manufacturers and the simple math that the lack of machines and the existing wear on their parts will not even cover the playback of the volume of archival tapes awaiting…migration? This hits a deep reality. Have we lost hundreds of thousands of hours of archival motion under our watch? Maybe even millions or billions of hours?

With the current demand from patrons increasing and the desire to use motion outtakes, news, etc. for historic documentation this is a disappointing state. Motion is powerful, emotional and immediate. It puts patrons in the experience in a way that to the visual human being is dramatic.

This important discussion on the listserv was a strong exchange which took place over several days. Other important points included the argument that the research on degradation and proper handling has made the tape degradation less of an issue and now the issue is playback for migration and access.

Other points included:

1) The need to get these gems a proper venue to be seen (whether due to money or rights issues)

2) The natural evolution of gathering obsolete machines from production facilities and creating some centralized site for archival institutions to use.

3) How to custom make parts so that we can maintain the machines so that we can transfer the content.

4) Wait for a miracle technological innovation to happen in the next 20-30 years so that you can migrate. (Keep them cool until then).

We come back again and again to the complexities of managing archival multimedia. The next generation of archivists will have changed their paradigm and released themselves from the desire to preserve the artifact. I am not condoning refusing to preserve all originals but our focus as far as video tape and digital files will be on content migration.

Other formats still have certain inherent value and that is a different blog posting!

Future archivists will have a much better long-term understanding of what is an artifact and what is intellectual content that needs to be migrated. They will be more adept to the rapid pace of format change and will HAVE to adapt quickly.

Not that I am faulting us. We are the transitory archival generation, the one that bridges the 100 year film format and the thumb drive.

I agree that the great research done in the field has helped us to minimize that problem, but it does seem that we took the eye off the ball a bit on migration. We cite lack of funds, lack of understanding of the urgency by the non-archival world, the frustration with equipment manufacturers, etc. It sounds like we need a development guru to raise funds, awareness and help to join all the key players for a collaboration. Many mention that the creators often do not prioritize preservation as they should. That is true, but our role as professionals is to educate.

Hand-wringing is not allowed! The energy that we have spent on that could have been put forth to an international collaboration with a real potential for migration, managing equipment, and innovation. One manufacturer can not do this alone, everyone needs to get on board and when I say “everyone” I mean:

Archivists
Creators (producers, studios, channels, directors, talent, writers, etc.)
Media
Equipment and Media Manufacturers
Technologists
Professional Associations
Funders

I think all of the above agree that the loss is imminent. The question is can we use our collective economy of scale to work together? Or maybe it is will we?

Oh, I get it. “IT” – of course, the acronym for Information Technology. The name of the computer departments where I used to work. The place where all the “computer guys,” as I fondly referred to them, were busy working their techie magic.

However, when it comes to this particular blog format, a resource for archivists and librarians, “IT” takes on different connotations. What occurs to me is this concept, the quest for high-tech answers to make all our jobs, nay, our lives, easier and cooler, perhaps that is it. Having been teased about how uncool librarians and archivists are (by those not in the profession, natch), it is nice to be able to talk knowledgeably about computer use and social networking applications. It almost proves we are cool.

Certainly technology has made accessibility of library and archival material easier, and that is good. But all this computerization and the need of “computer guys” to create the next big thing is a little overwhelming. What are they trying to prove? Aren’t the existing tools enough? Many of them do similar tasks like helping people reconnect or share ideas. But how helpful are they? Sometimes I think these technology developers just want to come up with catchy names. Delicious, del.icio.us? Really?

Moreover, what is really daunting about the search for “IT” is that it probably will never be found, at least in its entirety. All of this reminds me of my obsession with the news and reading the paper. Many years ago a few people began discussing some current events, and I didn’t know what they were talking about. So I made it a mission to read the newspaper every day, as well as listen to the news on the radio, so I would always know what is going on. But the news is always going on; it will never end. Trying to know all the news is both productive and futile.

So it seems that the quest for “IT,” too, is both ends of that spectrum. Facebook, MySpace, Twitter, LinkedIn, ad nauseum have been interesting and useful tech tools. However, new technology will continue; some applications may be created that cause current tools to become obsolete. Will any of this ever be enough? Like chasing the wind, perhaps it is unattainable. But the quest certainly makes the work more interesting. And all these incarnations of technology should be documented for future researchers to study. Perhaps this will create a few jobs for archivists who have the foresight to collect and preserve it.

Database searching, the Internet, websites, email, blogs, social networking here there and everywhere.

What is it that we are seeking?

The leap that we had thought that we took into information technology is just a step. No giant leap, no crevasse to reach, no earth shattering change yet. The leap was a baby step to the next baby step to the next.

We read the literature and jump each year, each quarter into a new technology that will help us in our quest to reach users, to organize information, to migrate data. The next technology that will ease our burden, manage our info glut, make us smarter.

The core question and the reason that we started this blog is…HOW are we managing these technologies? The change? These baby steps?

For many institutions it seems to either be a “jump” or a “leap”. By this I define a “jump” as running to a technology without the benefit of a needs assessment or proper research. Questions such as the below come to mind:

How will this benefit my users?

What will be the net (and defined) gain to my institution?

Do I have the in-house expertise to assess this, implement it and maintain it?

How will I promote it?

Sometimes this occurs because management does not realize the planning steps needed for successful implementation. Educating them sometimes helps.

A “leap” is defined as realizing that your institution is behind and trying to leap up to be on par with other institutions.

Sometimes peer pressure affects this decision and the realization that you are the only institution without wireless forces a “leap” into new technologies. Planning questions that come to mind here include:

Do you have the staff training for this?

Are we able to pull along all other associated technologies, services and users to successfully integrate this “leap”?

I speak to students and working professionals every day that embrace technology but worry about whether we are choosing the best technology for the best service or just buckling under to the marketing of consumer products.

Does your institution need to Twitter? Why? To seem hip or to reach Generation C? Is Generation C your user base? Do you have a real Twitter plan for keeping it up to date and understanding how to optimize it?

It is our hope that this will be the venue in which you can do that.

Archivists have special needs when it comes to choosing a file format for storage. We have all heard of or endured the nightmares of file format obsolescence. There are no guarantees when it comes to file format longevity, but here are some tips to help you choose a file format that stands a chance of still being readable in a few years.

Consider your needs. Are there legal or other requirements that make it necessary to preserve the document in an unedited version? Is it important to retain the formatting or structure of a document or is the preservation of the content the primary concern? If your requirements dictate that a format must retain the original presentation or associated metadata, you may not be able to use an open source format as a replacement.

Don’t be an early adopter. Wait for a file format to become widely adopted before converting all of your files. When a file format becomes commonly used by a variety of software manufacturers, it stands a better chance of long-term survival.

Investigate your potential new file format. Check user groups & forums for their comments about the format. Talk to colleagues that have taken the leap to see if they have encountered any difficulties.

Check the reputation of the format’s originator. Is the format proprietary or open source? If proprietary, is the company stable and reliable for support questions that may occur? Is the format compatible with previous versions? Will it continue to be compatible with future software versions? If the vendor has a history of discontinuing support for previous software versions or formats, can you afford to continue updating to new versions in the future?

Take time to consider your options carefully. What looks like a stable format today can easily be made obsolete by advances in technology. Keep track of new trends in technology to stay aware of your future migration needs.

Sources:
National Archives: Digital Preservation
Northeast Document Conservation Center