How did one institution attract 50,000-plus volunteers to help with an archiving project?

The National Library of Finland is in the process of digitizing its archives so that they are fully searchable on the Internet. Scanning the centuries-old newspapers, journals, and documents isn’t so much the problem as is accurately transcribing the text. OCR (Optical Character Reading) software can only do so much. Standard fonts are easy enough for a computer to identify, but aging print in fancy scripts are more difficult. Obscure language, proper names, and decaying paper also interfere with OCR’s text recognition. In order for the materials to be accurately digitized, every document must then be double-checked by human eyes.

To help with the process, The National Library of Finland teamed up with Finnish technology company Microtask to come up with an innovative solution: make a game of it. Granted, it’s hard to imagine how anything like checking manuscript text against a computer’s digital interpretation could really be fun. But Microtask saw things differently—instead of pages of repetitive work, they broke down each individual word-check into what they (appropriately) call microtasks.

Taking each microtask as a tiny action to perform, two online flash games were created.

In Mole Bridge is a fast-paced typing challenge. An image of a digitized word from the original manuscript appears at the top of the screen. The player must type the word, as best he or she can read it, as quickly as possible. The typed word helps build a “bridge” that ensures the safety of some adorably cute, but evidently suicidal moles, who tirelessly try to cross the void. The quicker the words are entered, the more moles make it to safety. Words that are indecipherable—or too difficult to type (American keyboards lack the å, ö, and ä symbols common in Finnish) — can be skipped without penalty.

In Mole Hunt, cute critters hold up signs with individual manuscript words and the computer’s interpretation of each word below. Players identify whether or not the computer’s recognition of the word is accurate.

In terms of fun, the games are more on par with Mavis Beacon than Super Mario. But like many other online distractions, there’s something strangely addicting about them. Knowing that the games actually contribute to a worthwhile cause —preserving the cultural heritage of Finland — is evidently enough to keep people playing. Since the Mole Bridge and Mole Hunt were launched in February, they’ve logged over 50,000 players, hailing mostly from Finland, the UK, and the US. At last count, the grand total of work contributed to these games is over 3,500 hours, with 4.5 million microtasks achieved.

The real brilliance of these games is that not only do they get the work done, they get it done right. With thousands of people playing, gamers effectively cross-check one another’s work. The software developers at Microtask have even created a defense against the few people who might have malicious intent (apparently, even Finnish libraries occasionally encounter mean-spirited Internet trolls).

When the game starts, the computer shows words that have already been accurately identified. It can then determine if someone is making deliberate mistakes or is just too poor of a typist to benefit the project, and then disregards every further answer submitted by the player. Microtask reported that volunteers’ transcriptions were 99% accurate.

While the games delegate hours of work to the hands of others, they are in no way a substitute for trained professionals. No matter how many microtasks are achieved, the games only cover a fragment of the digitization process. Archivists, librarians, and other specialists are still necessary to develop and implement every step of digitization projects. Accurately digitizing a nation’s archives is an enormous task. For starters, it would require:

• Different classification schemes for different types of materials
• Categorization by subject matter
• Indexing every document
• Entering metadata on every file
• Secure storage space with prevention against digital decay
• A user-friendly interface that allows for effective browsing and searching

And that’s in addition to a plethora of other issues as well. How well does centuries-old newsprint fare in a scanner? What do you do with materials still under copyright? How do you anticipate the needs of both serious researchers and more casual users, such as genealogists?

Implementing gaming systems is not a quick solution to digitization projects, but many institutions could benefit from The Finland National Library‘s model. Introducing similar programs would not only help the workload behind large-scale digitization projects, it would (perhaps most importantly) establish a connection between the public and their libraries. It would raise awareness to libraries’ missions and likely cause more people to support such institutions. Preserving cultural heritage should not just be the work of archivists, and now, it doesn’t have to be.

Play Mole Bridge and Mole Hunt at http://www.digitalkoot.fi/en

Further reading:
Gamification time: What if everything were just a game?
The secrets of Digitalkoot: Lessons learned crowdsourcing data entry to 50,000 people (for free)
Digitalkoot e-programme breaks 25,000 participant mark

Please note that the images are used with permission from Microtask. Several attempts were made to receive written permission from the National Library of Finland, but there was no response. We assume that in the interest of education, this is fair use.

A few weeks ago I was honored to attend the National Digital Stewardship Alliance meeting. The NDSA was planned by the Library of Congress as part of their NDIIP project.  There were more people there than I expected and it was a humbling experience to hear some of the brightest and most creative brains in Digital Preservation speak.

The high volume of information was overwhelming.  I spent more than six hours at the end of the conference compiling a PowerPoint of the important research highlights.   A small sample of this information is included below.

There were many wonderful presentations giving case studies on how institutions used their own creativity to try and enhance the longevity or migratability (new word?) of their digital files.  The amazing work often was done on a shoestring which though unfortunate, also forced a certain level of imagination and invention.

A few examples are:

Jack Brighton, of campus radio station WILL, gave a wonderful presentation on what a small station is doing to make their civil rights collection more accessible.

Kickstarter.com did a great presentation on how they are helping arts projects get funded and we hope that as they branch into community work that digital preservation might fit into that.

The UK Web Archiving project covered some of the complexities and true effort that it takes to try and tackle capturing the online history of its nation.   http://www.webarchive.org.uk/ukwa/

- As of December 2010 – 9 million sites with .uk, probably 1M more

- 10,027 websites archived

- Need skills in Linux, Java, Hadoop, and SOL

5 keys processes to web archiving

- Selection

- Harvesting

- Storage

- Preservation

- Access

http://www.webarchive.org.uk/ukwa/ngram/

So after taking in all this good information, what is it that I have left the conference with?

People just like us are doing some wonderful problem solving out there.  There is some potential being unlocked, but there is so much to do.

As I see it the Action Items are:

1)    Greater broadcasting of the successful case studies for migration and open solutions.

2)    Training classes in how to boil this down for each type of format/issue.  The NDSA Outreach group held a session called “Digital Preservation in a Box”.  This is the beginning of standardizing the tools that we need.

3)    Overarching education to information and production professionals, as well as, the general public about the dangers of digital fragility and the need for migration (at the least).

I have mentioned to my classes for years that future anthropologists, sociologists and historians will have little to sift through from the late 20^th century.

Some of it is being worked on by archivists now but much is gone.  Let’s keep making progress so that the future of our current history is not lost, like the way of silent films.

More informational tidbits from NDSA:

Other Great Projects

—       ThatCamp.org

—       http://www.scola.org/scola/sampledigitalarchive.aspx

NYPL Labs

—     http://search.creativecommons.org/?q=nypl+map+rectifier&sourceid=Mozilla-search

—     http://menus.nypl.org/

Archiving Facebook

Grad student designed Firefox add-on for individual archiving of Fb.

www.Bit.ly/archivefb

Preserving Virtual Worlds

—      www.ideals.illinois.edu/handle/2142/17097

Cool Tools

—  http://www.google.com/landing/historypin/

—  http://blogs.yu.edu/cpa/2011/02/23/open-source-video-platforms-kaltura-vs-entermedia/

Great Quotes

—    JackBrighton”(DAM) is more like an appliance than an Ecosystem.”

—    Michael Nelson “We need to raise the level of user expectations.”

—    Michael Nelson “In all good computer science functions you solve the problem through indirection.”

—    Wheatley and Frieze “The world does not change one person at a time.  It changes as networks of relationships form among people        who discover they share a common cause and vision of what’s possible.”

—    Tim O’Reilly(?) “Teach preservation as a mindset.  Bake this into the tools.”

New Phrases

—    Social Curation

—    Metadata Ecologists

After reading some marketing information from a DAM vendor, and working in the field for nearly 20 years, I just needed to vent about how some present their product.

Some DAM system vendors often tout their automated systems as replacements for what they claim is “costly manual tagging”. Yet, after implementing one of these expensive systems, their customers often turn to information professionals for metadata development help, because their end users are unable to find the assets they need in a timely manner. There is an obvious disconnect between full automation versus high-end manual service.

Using “smoke and mirrors” has unfortunately been an approach for some DAM system vendors. Like stage magicians, they use misdirection to steer potential customers away from the failings of their systems, sometimes by using confusing jargon or misleading customers about the true costs of accurate, efficient metadata development. They argue that automated indexing is superior and less expensive than human catalogers, yet the customers who purchase these systems are disappointed to find that after spending hundreds of thousands, if not millions, of dollars on an automated system, they still need to hire information professionals to make their assets truly useful. At this point, like the professional mechanic trying to repair the shade-tree mechanic’s work, the issue can become more costly than it would have been if the information professionals were consulted in the first place. Cleansing metadata is always more difficult than designing it well in the beginning.

So, despite the “sexy librarian” stereotypes that abound, our profession is often “out-sexed” by the glitz and glamour of technology and the deliberately misleading sales pitches of some DAM vendors. Some corporate officers are enamored of technology and reluctant to expend budget on people. It is only when they realize that the assets that are vital to their core mission cannot be easily found and reused that they turn to information professionals to “fix” the metadata problems. Keep in mind that reuse generates income to help pay for the database system, and in the most successful cases, to more than pay for the existence of the archives & its staff.

Each industry has a specific terminology that distinguishes itself from other industries and from competitors within their own industry. This complexity of language requires mediation by individuals who understand not only the language, but the way in which users apply the language in searching. While computers & software can do a great job of capturing content, they are not as successful at interpreting that content’s meaning to the organization which generated it, or to the public at large. Interaction with the public to increase brand recognition is vital to every company’s mission, especially in light of the increased use of social networking.

Building brand recognition depends on how well one’s marketing efforts are communicated to the public. The textual content of advertising, even with the use of closed captioning and scripts, does not always convey the concepts that a brand manager wants to stress to the public. Often, advertising is so subtle that it does not even contain dialogue. How successful is machine generated indexing to this type of material? A human indexer can capture concepts such as value, convenience, refreshment, fun, excitement, vitality, friendship, family, customer satisfaction, and many others, that may only be contained in the visual component of the material.

The communications department of each company may also have specific requirements for reusing historical imagery. It may be necessary to know that an image of a delivery truck represents the first use of innovative technology, or that a product package image is a limited edition, advertising the anniversary of the company’s participation in a sporting event. This information may only be available from visual clues within the image itself, or from non-documented “corporate memory” which exists only in the knowledge of the people who created the images. It is also essential for a company to know whether or not the talent that appears in an advertisement has been bought out for a limited period of time, in order to avoid unauthorized usage. This requires not only that the talent be identified, but that documentation of the rights contract is available in a form that can easily be accessed. Advertising agencies have not historically kept this documentation readily accessible. Researching this information can be time consuming and difficult to navigate, as the records may only be organized by date, job number or brand, or by the name of the union with which the rights had to be negotiated. Working titles for individual spots may change multiple times, ending with a completely different final title. Accurate identification is essential for successful reuse of corporate assets. A skilled researcher can quickly cut through the layers of obfuscation to get to the heart of talent rights.

Isn’t it about time that we, as a profession, sell our abilities as well as the DAM system vendors sell their systems? After all, we’ve been proven vastly superior, despite the claims to the contrary. Human beings can make the connections between content and context that automated systems are incapable of making. While there have been improvements in face recognition, speech-to-text and other automated indexing methods, nothing beats a professional indexing team for improving accuracy and adding value to machine-generated metadata. For accuracy of retrieval, there is no replacement for a well-tailored controlled vocabulary used in combination with a well-designed indexing policy. This is one of the many places where information professionals excel.

Fortunately, there does seem to be a movement among smarter corporate entities to incorporate librarians, archivists and other information professionals into the teams making the decisions regarding DAM system purchases & implementation. In addition, many employ specialists to create metadata dictionaries, crosswalks to vendor databases, and to manage the quality of metadata entry. These companies have the best track records in developing systems that are the most useful & successful, even winning awards for their efforts. The success of these projects should lead other organizations involved in digital asset management projects to emulate their efforts and consult the experts in metadata development.

The best DAM vendors know that their tool is just part of the solution for asset management and a great team of people needs to set guidelines, determine field requirements and design searchability together. This is what any institution looking for an answer to Asset Management should look for.

If you’ve heard of Second Life but wasn’t really sure what it was, or perhaps you’re feeling hesitant about learning “another” new thing, take heart. I’m here to provide information and guidance to this new social media tool. Check out this 3 minute YouTube video for a quick demo on Second Life and how it’s being used to teach university classes.

In a nutshell, Second Life is a software program that looks like you’ve entered a three dimensional (3D) world on your computer. It’s fantastic for demonstrating processes, displaying artifacts and information, interacting with other people, collaborating, building in accountability to distance education courses, teaching complex concepts, simulations and more. As if it couldn’t get any better, creating an account in Second Life and using their software is free!

Ok, there can be some costs involved. If you decide to use Second Life to create a museum display, hold classes online, run a simulation, etc., you’ll most likely need to rent some server space from the Linden Lab company. But to the public (your customers, clients or students), the use of the system is free. In future posts I’ll go into more depth about how to do a cost analysis and project plan for getting your business or institution set up in Second Life. If you’re already familiar with Second Life and need a little help getting your institution started now, shoot me an email at sjkmartin@gmail.com.

The key concept behind this 3D environment is that it’s entirely user created. You get to decide what you want to do with it. Several years ago I opted to create an interactive museum exhibit called the Black Abolitionist Digital Archive. Second Life visitors walk into a period court house, see portraits of black abolitionist speakers, link to the University of Detroit Mercy’sabolitionist digital research collection, watch informational slide shows and examine 3D artifacts. The most powerful aspect of the exhibit is the ability to reenact the actual speeches and have panel discussions with experts in the field. Communication is done through text chat or via microphone similar to a conference call. All this is done in the comfort of your chair at your home or office. Where else can you get access to that kind of information? If you already have the Second Life software downloaded, click on this slurl and it will open your software and take you right to the Black Abolitionist Digital Archive.

If you haven’t experienced Second Life yet, think seriously about giving it a try. My next posts will deal with interesting places and examples of how institutions and individuals are using the technology. Be prepared for some surprises!

In March, Rachael Clark discussed the necessity of completing a needs assessment as one of the important skills used in developing a digital collection. I would definitely agree that it is a vital skill for any information professional. Cliché as it is, one of my go-to phrases is, “If you fail to plan, you plan to fail,” and performing a needs assessment is an essential part of the planning phase for any project. This includes diving into the world of social networking.

The first decision to consider with your team is what kind of social media outlet will be most beneficial. There’s more to the world of social networking than microblogging on Twitter. You can get your professional persona out there with LinkedIn, share attention-grabbing images of your collection using Flickr or post a webinar on YouTube. Consider your needs, speak with people who are fully immersed into the world of social networks and learn from them. There are many options from which to choose. Do a bit of research into what will work best for your organization before you spend too many “hit and miss” hours. Time is money, folks!

Once you’ve decided on the tools of choice, you probably think that adding social networking to your organization seems like such a simple thing. Go to your social networking site of choice, register and start posting, right? But before you dive in, stop and consider what kind of benefit you’re expecting from these popular tools. Do you want to reach out to patrons? Are you interested in generating buzz about your company? Do you hope to gain new clients? Perhaps you just want to keep in touch with colleagues around the world. No matter what your purpose, you’ll be doing a little writing and though this is writing on a small scale, the importance of writing for your audience cannot be underestimated.

More things to consider when creating a needs assessment are, how often and how much of information is being shared? How often are you going to use this tool? You will need to think about quality and quantity if you want to reach people and make the most of your social networking tool of choice. For example, if you’re using Twitter or Facebook, you will want to build an audience of people who read and repost or retweet your updates. If you want to attract loyal readers,  frequent, appropriate and relevant posts and tweets are vital.

Decisions will also have to be made about who will be the social media guru of your organization. You may find staff resistant to using these tools or you may have staff members who are ready to make it their full-time job. The right person will have enthusiasm for this new way to communicate, but it’s important that he or she will be literate in the language of social networking. They need to know that it’s “tweeting” NOT “twittering,” and they need to have the ability to tell the difference between useless information and great things to know. Many social networking sites have users who are spewing a lot more garbage than gems. The person doing the majority of work with social networking tools will have to do quite a bit of filtering.

In this day and age it seems as if everyone and their grandmothers have at least dipped their toes into the giant pond that is social networking. If you take your time to examine your needs, then plan and execute a process to get the most out of the social networking tool of your choice, you will add volumes to the intellectual value of your organization.

I have been using for the above phrase for many years. I say it with conviction in my voice while making sure to maintain eye contact. I believe it deep in my bones.

Why is the history business such an important issue for me and thousands of archivists across the country? Part of it is the growth in demand over the last 15 years by cable networks to fill their channel with documentary programming. Some of it is the keen interest I personally have in learning about the human condition and learning from those events. Mix that in with years of licensing negotiation and seeing how amazed producers are with what archivists can provide and I know that this is big business.

History as a hobby is having a surge in popularity as well. TV shows that put you in the historic moment or researching your ancestry are feeding from one of America’s top pastimes: genealogy. According to a Maritz marketing research poll back in 2000, 60% of Americans are interested in their ancestors. In 2002 the Internet genealogy market was estimated to be $200 million and as recently as March of 2010 the estimate of a $1 billion market was cited. When doing research about the history industry I found a title “Who knew the genealogy business would be so cut throat”. Articles and blogs are full of stats and predictions of this marketplace. The sheer volume of the discussion inherently points to the perceived value.

Couple the genealogy market with the thousands of documentaries about historic topics as well as original network programming about finding your ancestors and we have a distribution explosion! This great opportunity is open to advance the exposure of our collections and of history itself.

The caveat is that we have realized that licensing from these opportunities will not be the heaven sent funding that we hoped in the late 1990s. Licensing is more competitive now and budgets are tighter. It will provide funding but not at the Getty level.

What these historic documentaries and stockhouses also provide us is access to the public in ways that we would never have dreamed of 20 years ago.

Yes, the Getty licensed roughly 22 million images last year. Why not your archive?

The simple answer is money makes money. Most archives are not in a position of having excess cash, so the best advice is to work on collaborative venues for licensing, usage and copying your content. Many archives are working together instead of fighting for grant money. Cross-promotion and use of free and open source tools all provide things that only multi-million dollar organizations could afford 10 years ago. If marketing your collection allows generation of revenue for on-going preservation or digitization then this is a treasure you need to promote.

Many archives are even going into co-productions and signing contracts to put their imagery up with the big commercial venues. This leads to a bigger shot at wide distribution without the investment risk. As long as the contract provides for certain protections, this may be a viable option.

You have some real avenues for revenue generation these days, but you do need a niche, IT support, a marketing plan and most importantly legal advice!

More to come on this topic!

Sources:
Grow Your Family Tree in Salt Lake City
Genealogy: A $1B Market? Maybe
ITV, Mormons and Family Trees

Earlier this month, the Library of Congress announced that it would house every “tweet” ever posted on Twitter. Every 140-character-or-less blurb on the site is now part of the vast LoC archives. This got me thinking: what are the issues at hand in archiving social networking sites? And why is it important?

Recently, while cleaning out my apartment, I found a relic of primitive social networking—a printed-out Facebook message from 2005. Nostalgia instantly struck. Five years ago, Facebook was [thefacebook], with a much simpler interface. A toolbar on the left listed the humble features of the relatively new site: My Profile, My Groups, My Friends, My Away Messages. Clearly, Facebook was trying to emulate MySpace —which was then by far the preferred means of social networking.

The past five years of my Facebook existence flashed before my eyes. I remembered when the site was only accessible to students at selected universities, when users could upload only one photo, when “the wall” was merely a text box that anyone could edit. A Facebook before FarmVille, “like” buttons, and mini-feeds.

With hundreds of millions of users, there are many reasons to archive social networking sites. First, for historical documentation—millions of photos are uploaded to the site every week, capturing the trends of the present era. Additionally, no matter how worthless each status update might seem, taken as a whole they reflect our reactions to modern events. It would be worth keeping a log of Facebook updates from the night of the 2008 election, or as the news broke about the earthquake in Haiti, or even as the Tiger Woods scandal unfolded.

Second, following the history of social networking sites offers insight into the development and perceptions of Web 2.0. What was it that made Facebook become so much more popular than MySpace ? How has Facebook increased the average person’s awareness of internet safety and privacy? Studying the evolution of such sites, web developers can predict future trends in online technology.

Third, social networking sites are paramount to the way we communicate in the 21st century. How do we choose to represent ourselves online? Everyone has that one friend who reveals too much personal information (Matt Maclean: too much vodka + ice cream = the great pukefest of 2010). There’s the perpetual Facebook drama, including my favorite, the Passive Aggressive Status Update (Alexis Cooper thinks some people need to get over themselves!!!) And how has Facebook changed our interpersonal relationships? At some point in their cyber-existence, most people have undoubtedly become angry or hurt or jealous over something said on Facebook. Relationships that may end in real life passively continue online—the psychological implications of which are new to our generation.

But how should sites like Facebook be archived, and to what extent?

The Internet Archive has preserved 150 billion web pages, dating back to 1996. Using web crawlers to archive websites, the IA serves as a publicly-accessible digital library, allowing users to revisit older versions of their favorite sites. Documenting and preserving static websites such as aol.com is relatively easy because they are open to the public. Public profiles on MySpace and Twitter are also easily accessed. But how do web crawlers get inside websites, like Facebook that require login information to view its content?

It’s not easy. The Internet Archive has developed a software application called Archive-It, which allows organizations to make digital copies of their own websites. As more and more universities, corporations, and libraries create their own Facebook pages, they can now archive them. While this is great news for individual organizations, it doesn’t include the interactions of ordinary Facebook users—the cyber chatter that documents our day-to-day lives.

Recently, the White House announced plans to preserve its social networking content. Media capturing is done using software applications and daily screen shots, storing text, graphics, audio and video in the context in which they were originally presented. Once this project gets off the ground, it can serve as a model for other organizations.

But what about the interactions of everyday Americans—how will their voices be recorded? Perhaps the LoC or the Smithsonian could begin a digital initiative to capture people’s online identities. An open call for submissions, asking Facebook users to send in screenshots of their profiles, walls, photos, or feeds, would be a simple and effective start. Similar projects, including the September 11 Digital Archive and the Hurricane Digital Memory Bank, were created to obtain Americans’ reactions to history-making events.

It’s difficult to say if a comprehensive range of Facebook users would be willing to submit their personal information to an archive. However, thousands are already sending in their screenshots to blogs, including Failbook and Lamebook, which capture people’s regrettable and embarrassing Facebook moments.

No matter how trivial they may seem, social networking sites are a defining part of our generation—and that’s something worth preserving.

Further reading:

In the last month, I have had to replace a one year old refrigerator, a 30 day old phone, a two week old portable drive and a one-day old server. Technology is not always our friend!

When I first got involved in digitization (15 years ago!), I was sorely disappointed with the inefficiencies and struggles to get output as promised. I teach my students today that the information world is a difference place. We finally have tools that talk to each other, tools that can be modified through menus as well as hard-coding. This is a wonderful world of possibilities.

Tonight as I sighed and sat down thinking about all the time I have wasted on new products failing, I thought…”Am I right to be telling students that finally the tools are working for us instead of the other way around?”

In getting this blog up and running, we chose a simple template. We wanted it to be clean and easy for users and for us. So far, according to our change document we have encountered five unexpected problems that took a fair amount of time to fix (even with experts on staff). These problems all centered around incompatibilities to get information where, when and how we wanted. As information managers these seemed like simple moves and formats. We thought that we were not doing anything fancy.

I also tell my students to take as many technology classes as they can “stomach”, because the more that you know the smarter your decisions will be on which technology to use.

We are a pretty technical bunch but since each project is so different, and each institution is managing different types of information from any country and any time period in history, how can our information tools keep up to accommodate all human knowledge?

Open source is a start to integrating these tools but I still believe deep down that more librarians and archivists as programmers would make our tools much more common sense based.

I have seen many tools that sell well but their implementations are so short-sighted as to make the product unusable.

Here are some for instances:
1) All fields migrate from one database to another but only a portion are searchable!
2) The new Speech to Text feature gets every single word wrong and yet it is still for sale.
3) The new template for your site will not allow you to migrate data from your old template.
4) You can’t migrate your contacts from one phone to a new one
5) You can’t change your default directory when scanning.

These examples are counter-intuitive to good information management. Are there better products out there? In some cases, yes. This is why we need to be careful consumers. In some cases though, there are so many unknowns until you install your data that it is a painful “learning on the job” situation. In many more instances, your institution is already married to a technology and you just have to work with it.

Will things get better? Or is the question HOW will they get better?
I firmly believe that we (librarians and archivists) need to become more technologically astute and create our own products. The more we know will also allow us to push developers to better understand what we need.

We also as a group, have to really praise developers that are listening to us and that DO create products that are user and administratively friendly.

So, my question to you, is how best to do that?