This week in digitization: The good, the buggy, and the curious

This will be old news to many, but regardless: two big projects related to specimen digitization and biodiversity informatics launched in the past couple weeks.   Quick impressions on both below, focusing on the good, the buggy and a few items of curiosity.  Both projects are great, but — how will they fit into the broader landscape of existing resources, and into what niches?

1) Notes From Nature — a new Zooniverse project for the transcription of natural history collection ledgers.  This has been a long time in the making (more details here) and as of this writing, the two available collections (Herbarium specimens from SERNEC and insects from CALBUG) are already 26% and 21% transcribed, respectively.

The Good: As always, clean and intuitive interfaces from the Zooniverse team make transcription fast and easy.  Data entry screens are customized to each type of collection (e.g. plant labels often contain more detailed locality descriptions than insects, whereas insect labels often contain data about what host-organism they were found on).  Awesomely, all the code is available on Github (https://github.com/zooniverse/notesFromNature) in case other Museums want to set up their own transcription engines locally.  There is also an intriguing teaser buried at the very bottom of the Notes from Nature “About” page: “Interested in publishing your collection? Contact us.”

The Buggy: Maaaan, I’ve transcribed around 40 labels and my total isn’t showing up under my user profile. This bothers me more than I care to admit, though it’s primarily out of worry that my transcriptions aren’t being saved.

The Curious:  It would be great to learn more about how these data get back to the collections databases, and how exactly that handoff happens.  What do the transcribed files look like?  How is accuracy checked?  Do the museums have plans to make these records publicly available, or harvestable by aggregators like GBIF?

2) The patriotically-named Biodiversity Information Serving Our Nation (AKA BISON) biodiversity data portal out of USGS –  I know less about this project, other than what I’ve learned at various conference talks — however I’ve heard it referred to as the “federal version of iDigBio.”

The Good: On first look, really nice integration of specimen occurrence data with USGS map layers, and as Hilmar Lapp pointed out, there’s an API, which is great.

The Buggy:  There are no identifiers on these specimens — not even their local catalog numbers.  Per Stinger Guala in the G+ thread linked above, the data is there — it’s just not yet visible (though will be soon).  Perhaps there are reasons (a need for better formatting? a need for cleaner data?  a need for more server space?) that they’re not yet making this data visible yet — but it struck me as a pretty glaring omission.  While I realize that many researchers don’t spend a lot of time looking at  catalog numbers, I imagine that they’d be absolutely critical if one was integrating BISON data with that from other sources (say, something from another portal like GBIF). Also, how could any of these records ever be linked back to the source data or any other data out there?  Provenance = important, no?

The Curious: BISON is apparently the US node of GBIF — which I had assumed meant they would be providing GBIF with US data  – however, the data in BISON appears to invert that model and is a US-centric mirror of GBIF.  I hope that BISON becomes a platform through which US, federally owned and managed biocollections can be made publically discoverable, and would be interested to hear from BISON reps if there are any plans in place to do this.

What gets linked to global unique identifiers (GUIDs) in natural history collections digitization?

Co-written with David Bloom.

For as long as explorers have been collecting specimens and bringing them back to museums, collection managers and museum staff have been assigning unique (well, more or less unique, with a margin for human error) numbers to them. Collections management isn’t just about conservation of physical objects – it’s also about care and support of an information retrieval system.  And while new technologies come and go, the maintenance of this system of locally unique identifiers remains at the core of collections management.

Recently, that aforementioned information retrieval system has been growing increasingly dispersed, and the ways in which information about specimens is disseminated, accessed, and manipulated have been changing rapidly.  The ability to place digital representations of specimens and their data en masse onto the World Wide Web is fundamentally changing collections-based scholarship.  Consequently, locating the right giant clam specimen in the University of Colorado Museum of Natural History (CU Museum) invertebrate collection is a very different task than locating a digital image of the same specimen on the Internet.  Tracking and connecting all of the digital representations (e.g., images, metadata records digitized from labels, tissue samples derived from the specimen, sound and video files) derived from that single specimen is even more difficult.  While local identifiers suffice to connect data and specimens within the CU Museum, global identifiers are needed to maintain these connections when content is released into the wilds of the Internet.  The challenge before us is how to best set up a system of globally unique identifiers (GUIDs) that work at Internet-scale.

iDigBio, an NSF-funded project tasked with coordinating the collections community’s digitization efforts, just released a GUID guide for data providers.  The document clarified the importance of GUIDs and recommended that iDigBio data providers adopt universally unique identifiers (UUIDs) as GUIDs.  It went further, however.  In the document, a call-out box (on page 3) states that “It has been agreed by the iDigBio community that the identifier represents the digital record (database record) of the specimen not the specimen itself. Unlike the barcode that would be on the physical specimen, for instance, the GUID uniquely represents the digital record only.”

In response, Rod Page wrote an (as always, entertaining and illuminating) iPhylo blog post “iDigBio: You are putting identifiers on the wrong thing ” in which he makes a strong case that a GUID must refer to the physical object.  “Surely, “ writes Rod, “the key to integrating specimens with other biodiversity data is to have globally unique identifiers for the specimens.”

The disagreement above underscores our community’s need to be very clear about what GUIDs reference and how they resolve^[1].  This seems simple, but it has been one of the most contentious issues within our community.  So who is right – should GUIDs point to digital records, or physical objects?

iDigBio has a clear mission to support the digitization of natural history specimens, and thus, deals exclusively with digital objects, which, as any database manager knows, need to have identifiers.  So, it does make sense that they would be concerned with identifiers for digital objects.  Those identifiers, however, absolutely must be as closely associated with the physical specimens as possible.  In particular, they need to be assigned and linked to the local identifiers stored in local databases managed by on-site collections staff.  If iDigBio is saying that only digitized objects that get passed to iDigBio from their data providers need GUIDs, not the original digital catalogs, we can’t agree with that.

On the other side, we’re not sure we agree with Rod either.  If Rod is suggesting that GUIDs replace, or serve as additions to the catalog numbers literally, physically attached to specimens or jars, we think that is simply impractical.  What is the incentive for putting a GUID on every single specimen in a collection, especially wet collections, from the point of view of a Collections Manager?  Does it help with loans?  Who is going to go into a collection and assign yet another number to all the objects in that collection (and how many institutions have the resources to make that happen)?

What we think is feasible and useful (and likely what Rod meant) is to assign GUIDs to digital specimen records stored in local museum databases and linked to the local identifiers.  When these data get published online, the associated GUID can be pushed downstream as well. Assigning GUIDs to the local, authoritative, electronic specimen records as they are digitized should be a mandatory step in the digitization process — a process that iDigBio is uniquely poised to support. This is the only way that GUIDs will be consistently propagated downstream to other data aggregators like VertNet, GBIF, and whatever else comes along fifty years from now (and fifty years after funding runs out on some existing projects).  Again, we want to point out: it’s important to remember that natural history collections management has always entailed the management of identifiers; the adoption of global identifiers will only increase the need for local identifier management

Now, we can imagine one case in which GUIDs could conceivably serve as the originating catalog number: during field collection.  As biologists generate more and more digital content in the field (such as images and DNA collected in the field), minting GUIDs at the moment of collection (or during the review of daily collection events) and assigning them to samples and specimens directly could be quite useful.  As these physical objects make their way into collections, we anticipate that collections folks will still assign local identifiers.  Both have their uses and are made stronger and more useful when linked.

In summary: we are less worried about what exactly a GUID will point to (digital record vs physical object) as long as the content referenced is valuable to the biodiversity collections and science community.  However, we are more worried that we’re not explicitly identifying what we’re assigning identifiers to, and not discussing who and how these identifiers will be managed and integrated.  Our focus should be on developing trusted and well-understood GUID services that provide content resolution for the long (50-100 years) term.

---

[1] To resolve a GUID, you dump it into a resolution service maintained by a naming authority that originally created that GUID, such as CrossRef or DataCite . That service then returns to you links and other information that point you to other content attached to the same identifier.  A great example of a resolvable GUID is a Digital Object Identifier (DOI).  In the case of journal articles, resolution of a DOI will usually direct you to the paper itself via hyperlink, but it could also be a web page with information about the resource, a sound file, or any other representation of the object associated with the GUID.

Post-Henderson Post

So You Think You Can Digitize had a bit of an unplanned hiatus; turns out that maintaining a blog while its authors take a something like 15+ trips, attend to work/school responsibilities, and write gobs of papers is a bit trickier than anticipated.  One of those papers was born directly out of this blog: a paper submitted to Zookeys about our work on the Henderson Field Notes Project.  Ironically, we spent so much time on the Henderson manuscript that we were forced to spend less time uploading Henderson’s notebooks for additional annotation.  That should be remedied soon!   With help from our some still anonymous friends, we’ve made a lot of progress, including the annotation and extraction of over a thousand species observations made by Henderson between 1905-1909, which we then packaged as a Darwin Core archive for publication along with the paper — but there are still 9 more notebooks to go.

We’ve written a lot about field notebooks, especially older notebooks, penned by pioneers, describing the Old West, but we do worry that this reverence for the past has the unintended consequence of diminishing the importance of field and lab data recorded in the exact same way in the present day.  Digitizing old things, showing off the labels written in India ink and cursive: the romance of this might reinforce unfortunate stereotypes about the “dustiness” and antiquity of museum collections.  In the here and now, field notes are still handwritten, and specimens are still collected and catalogued, often at higher rates than ever before.  What do we do with the analog present?

This very issue came up in a post on Andy Farke’s blog, The Open Source Paleontologist.  Dr. Farke is a paleontologist who recently published a paper describing a new species not found in the field, but rather, in Yale’s paleontology collection.  Kudos to him, because he decided that publishing his findings in PLoS ONE was not enough; he also published the lab notes on which his paper was based.  In his words,

“There isn’t really anything earthshaking in there… but in any case now other folks can use them. The sketches of real bone vs. reconstruction should be particularly useful.”

Earthshaking or no, we agree that there’s a huge need to openly archive this sort of documentation, both to support the reproducibility and replicability of scientific results, and to better describe the soon-to-be-historical use of the specimens in question.

Because repositories like Dryad aren’t intended to accept field or lab notes,  Dr. Farke turned to Figshare as a place to deposit them.  While Figshare is great for its ease of use and flexibility in licensing options, it’s nevertheless not ideal for text; pdfs are not easily navigable, nor is there any support for transcription and annotation of notes.  This made us wonder: could WikiSource be a place to deposit these notes?

After Gaurav conferred with the Wiki-community, we found that Wikisource could indeed be used to provision more recent field or lab notes in addition to historical documents, provided they meet certain criteria (and the Wikipedians don’t eventually find this to be a violation of existing policies).  Dr. Farke’s notes on the Torosaurus are now available here, and in need of transcription too.

Why put lab notes on Wikisource?  Why not just leave them on Figshare?  Well:

• Just as we were able to annotate the Henderson field notes for taxa, it’s easy to imagine notes like Dr. Farke’s being annotated with specimen catalog numbers, and even linked to other records describing the specimen in question.
• Lots of Copies Keeps Stuff Safe — If either of these sites goes belly up, then a copy of the notes would still be available, with the same CC-BY license that Dr. Farke requires.
• Publishing notebooks to a platform like Wikisource bridges gaps between formal and informal publication, not to mention institutional archiving and self-archiving (which more often than not is simply left-bottom-desk-drawer-archiving).  Yes, though anyone can edit or post anything to the various Wikimedia sites, there are nevertheless quality and notability requirements that must be met for an article to be considered Wiki-worthy; e.g. Dr Farke’s notes qualify because he (or his papers) meets that notability requirement, but generally speaking, Wikisource is not a place to “just put notes.”
• Something worth noting and maybe exploring further: Deposition of notebooks post-publication is not quite the same thing as maintaining an Open Notebook, although clearly related.  Wikisource/Wikimedia aren’t intended to be means of making science transparent, and may balk at that level of repurposing of the platform — only time (and continued experimentation…) will tell.

We are curious to see what others think of the idea of using Wikisource as a repository not just for historical notes, but more recent notes as well (and we also wonder if there are any eager paleo people out there looking to help transcribe Dr. Farke’s notes).

Post-Henderson, and post-Wikisource: we do want to turn our attention back to digitization, natural history collections, and what is going on right now.  A lot has changed since we started this blog back in March 2011; a number of projects that were merely in the planning stages have not only been funded, but have actually started digitizing collections.  Some of that work putting project plans into practice has been happening right on Rob’s doorstep.  A few weeks ago, the Herbarium at the University of Colorado had  a visit from New York Botanical Gardens traveling digitization set-up gurus Melissa Tulig and Kim Watson, who were here to set up an imaging station for use in the Tri-Trophic Thematic Collections Network — we hope to talk about that next post.

And hey — we’ll also be at SPNHC again this year, presenting on the Henderson project in the Archives and Special Collections session, so if you’re in New Haven this June, and wanna talk digitization, Wikisource, or anything else, do say hello!

JHFNP, Post 4.5

A quick mini-post here, to tell of some interesting things:

1) Notebook 1 is DONE.  Fully annotated, and all within 3 days of our last post.  This represents many hours of work and the creation of hundreds of annotations:

{{place|…}} annotations: 218
{{dated|…}} annotations: 64
{{taxon|…}} annotations: 347

So total annotations = 629!

2) Furthermore, Notebook 1 was annotated almost entirely by 1 person… who we can’t thank by name because they didn’t make a WikiSource account.  So, thank you, IP address 173.69.207.29!  And also to: 207.145.38.73.  If you would like a “thank you” coffee mug please let us know in the comments or via email.

3)  We have set up a Wikisource Project page.  We still need to write detailed, step by step descriptions for some pages (e.g. transcription and annotation), but we do have an initial help page on upload available, and more will be posted there soon.

4)  We’re putting new notebooks and transcriptions onto Wikisource as fast as we can get transcriptions and scans associated with one another; transcriptions of Notebook 2 are already up.  This notebook features more paleontology and geology than the last, and describes several trips to Northern Colorado: “Instead of attending commencement and taking my B.A. degree I started north on foot, up the Lykins lateral valley NW of Colorado Sanitarium.”  

We are excited about, and appreciative of, the annotation help, and we really do want to acknowledge as many folks as possible in the upcoming Zookeys paper.  So once again, if you want to be credited by name, make sure to create (and login to) a WikiSource account before annotating (though if you prefer public anonymity, just email us directly to say you are helping); we probably can’t acknowledge IP addresses, and we’re not sure if we can acknowledge aliases in a published paper (though how great would it be to say, “special thanks to Paul Flemons, Laura Russell and SquirrelFan23 for her or his help”).

Next, more substantial update: annotations and occurrences, as promised last time.

Field Notes Challenge Part 4: Help, ‘Cause We Need Somebod(y/ies)

Co-written once again with Gaurav Vaidya.

Over the last week, Gaurav has continued to pull templates out of his hat (leaving rabbit pulling to Rob and his bunnies) and we now have templates for locations and dates.  The syntax for these templates are very similar to the “taxon” template we discussed in the last post (and touch on below).

Let’s start with date.  The general syntax is:

{{dated|<date in YYYY-MM-DD Format>|<date as transcribed>}}

So this example from Page 12 of Notebook 1 becomes, “{{dated|1905-08-04|Aug. 4}}.”

The Location template looks similar –

{{place|Location Name|Location name as transcribed}}

– but does something particularly nifty: it creates a linkout to OpenStreetMaps which immediately resolves the place name on a map along with links to Wikipedia and Wikimedia Commons.  So this example on Page 6 of Notebook 1 is annotated like this:
“{{place|Florissant Lake|Florissant Lake basin}},” which creates a box like this: The little circled map in the image above is the link out.  YAY MAPS.

As always, the million dollar question is: what next?!?  The next step is to continue using these templates to fully annotate Henderson’s first notebook — which is where you guys come in.  All these experiments are well and good, but until the rubber hits the road (or the fingers hit the keyboard), its more theory than practice. We need your help.  If you have time and inclination, please jump in and annotate.  We want to make it clear that you can’t hinder our progress, only help, and that this is really easy.  It’s a wiki, so hit “edit” on one of the pages (for example this one, which mentions pikas (!)), and just try out a taxon or location annotation.  For example, on Page 31, you could hit edit and replace the word “chickadee” with this:  {{taxon|Paridae|chickadee}}, or even {{taxon|chickadee}} and VOILA — annotation!  Go back to the main index page (the up arrow next to the ‘Page | Discussion | Image” links) and you can see the changes have also shown up in the main contents page.

We have a particular need to get this done quickly, because we have been asked to assemble our experiments into a peer-reviewed and hopefully published paper in a special issue of Zookeys.  A manuscript is due in mid-March and, yeah, that isn’t very far away.  So we could really use your help with annotating this text. The rewards — apart from a general sense of well-being and the satisfaction of contributing to the furthering of knowledge about our planet — will be a direct mention of your help in the acknowledgements section of our paper.  If you are interested in making a more substantive contribution in terms of work and writing, we’d be pleased to chat more and possibly include you as a co-author.

We have been on the fence previously about the utility of prizes, and whether these are effective incentives, or a titch gimmicky.  In the past we’ve given (very small) Amazon gift cards as prizes, as a way to say thank you to those that took the time to comment on post, but this time around we’re thinking of something different: Rob is happy to make a small donation from (very limited) personal funds to make a Junius Henderson coffee mug and then give those away as prizes to people.  But we wanna know — do prizes help motivate you to get involved in annotating?  Or are they eye roll inducing?

COME ON who doesn’t want a coffee mug?

Next post: text mining, annotations and occurrences!

Field Notes Challenge Part 3: New Year’s Digital Resolutions

“Resolution” is a word with many meanings. It can refer to the granularity of a digital image, or the solution to a problem, or a firm decision to do or not do something.  The Carefree Cogitation Coalition here at So You Think You Can Digitize has been thinking all about resolutions as we enter the new year. We left 2011 with some exciting developments and new challenges related to making digitized and transcribed field notes openly available.  We have resolved some of the issues mentioned in our last post, and are now resolved to tackle perhaps the biggest challenge yet: to find a flexible way to annotate these notes.

First, to recap BRIEFLY:
1) We decided to use Wikisource as a platform for providing scans of CU Museum founder Junius Henderson’s field notes along with transcriptions.
2) The first notebook scans and transcriptions are now available.  Wikisource’s navigation can be less than intuitive, so here’s the index page, which lists all the pages of the Field Notebook along with metadata.  Next, click the “Notebook 1” link on the upper right hand side of the screen to get to the contents page for that notebook.  You’ll see a Table of Contents here and, as you scroll down, the full transcription, with page numbers listed along the left border of the page.   Click on any of these page numbers (for example, Page 5) and you will go to a page displaying the scanned image of the Henderson notebook and transcription.  This page is editable according to Wiki rules.
3) We’d originally resolved to only spend 5 hours a piece TOTAL on this project.  Yeah, consider that resolution broken.  Maybe 5 hours a piece… per week?

Figure 1.  Page 5 of Henderson’s field notebook as shown on Wikisource

So far, most of our work has been focused on figuring out how to get Wikisource to represent these notes in a way that’s consistent with existing Wikisource standards and policy, while also serving our needs as field-note data-miners.  We think we’ve done this pretty well; Gaurav Vaidya has put a ton of work into developing templates for taxon annotations.  You see those little items in boxes up there in Figure 1? Gaurav’s template (which has it’s own Wikisource page) automagically creates both a direct hyperlink to the species page as well as the floating boxes that link out to Wikispecies and the Wikimedia Commons.  The mark up itself looks like this:

{{taxon|<taxon-name>|<text-to-appear on transcription>}}.

So “Lark buntings” would become “{{taxon|Calamospiza melanocorys|Lark buntings}}.”

Pretty simple!  Our next steps are similarly simple.  We (read: Gaurav) will create annotation templates for “Dates” and “Locations,” and then start marking them up in the text.  Andrea has been linking together resources (uBio’s FindIt, Europeana’s Geoparser,  and her own rudimentary code) to automate this markup, so that the future notebooks we upload will be pre-loaded with links (we’ll talk about this more in our next post).  Locations, at least, will also be inter-wiki linked so interested readers can learn about the places Henderson visited during his journeys.  As soon as we have the templates for location and date done, we’ll post the syntax here and you can just jump in and try.  We’d love the help!

So now we have annotated field notes online, readily and freely available to everybody!  Exciting!  But here is the really exciting part: we think we can push these annotations out of the World of Wikipedia and into the larger semantic web.  Our plan is to unlock those annotations from Wikisource and try to represent them as separate Darwin Core observation records; more on what those records look like here.  We… aren’t entirely sure how we’re going to do this yet, but we’ll keep you posted.  We also have some interesting ideas about what to do with these:  http://commons.wikimedia.org/wiki/Category:Junius_Henderson.  Maybe you do too.  If you do, please comment.  We live for comments.  Be resolved to let us know what you think, and Happy New Year.

Field Note Challenge Part 2: Veni, Vidi, Wiki

SYTYCD would like to welcome guest blog co-author Gaurav Vaidya.

A week ago, we told you about our cunning plan to play around with annotating and publishing one  transcribed notebook of Junius Henderson’s field notes. We’ve had two big successes in the last seven days, which is not bad for soul-crushing finals and project deadline week the holiday season.

Success #1:  YOU GUYS, the internet is amazing.  Within half an hour of posting our last post, we were contacted by Dena Smith and Kathy Hollis, who alerted us to the existence of scans of Henderson’s notebooks — remember, we only had transcribed text files when we started!  This started a chain of events that put us in touch with two folks from the National Snow and Ice Data Center (NSIDC): Allaina Wallace, librarian and analog data archivist, and Ruth Duerr, Manager of Data Stewardship.  Less than 24 hours later, Rob and Gaurav had a productive meeting at NSIDC offices in east Boulder, and a DVD containing all the scans.  This DVD included three notebooks we hadn’t known about, two of which cover Henderson’s travels between 1927 and 1936 — adding another decade to his life on the road — AND were accompanied by more of Peter Robinson’s transcriptions.

From the field notes of Junius Henderson, Notebook 1

Success #2:  Having the scans made a huge impact on what we were able to do with the text.  In particular, Gaurav has made headway in using WikiSource as a platform for maximal use and re-use of Henderson notes. WikiSource is “an online digital library of free content textual sources on a wiki, operated by the Wikimedia Foundation” (i.e. Wikipedia). Uploading the scan of Henderson’s first notebook to the Wikimedia Commons was easy: these are now available as PDF or DjVu files. Once the scans were in the Commons, Gaurav created an Index page (following instructions on the Beginner’s Guide to Index: files and the Introduction to Proofreading on WikiSource). The Index maps pages from the scanned DjVu file to pages on WikiSource. Click on a yellow-coloured page number to proofread or edit an existing page (for example, page 3), or on a red-coloured page to transcribe it.   Transcription itself is dead-easy: the page image is displayed on the right, and a textbox (which accepts all MediaWiki syntax) is displayed on the left.  In our case, since we have the transcriptions “done”, it was mostly cutting and pasting sections of Peter’s transcribed text so that it aligned with Henderson’s scrawl on the scanned pages.

So yay, successes!!  The fruits of a week’s worth of work are available on the “Notebook 1″ page on WikiSource, where — using WikiSource’s <pages /> command — Gaurav mapped pages from the scanned DjVu file to pages on WikiSource.  Numbers along the left margin of the main page link back to the corresponding page from the Index, making it easy to verify or fix transcription errors.  Also, Gaurav compiled pages from the Index into sections representing field trips (just as Henderson did in his notes), and listed them in a “Contents” box at the top of the page.

Henderson’s field notes continue to be, first and foremost, a good read. “Notebook 1” features details from Henderson’s week-long trips to Florissant, Colorado (August 1905) and Silver Lake Arapahoe (September 1905). He keeps record of everything from the stamina of his comrades:

“The party showed fatigue in the following order: Sievert least, I next, then Watts, Then Markman, then Frank.” (August 30, 1905)

to train delays and opportunities for rumination:

“Train again so late as to afford ample opportunity for philosophic meditation upon the motives which inspire railroad people to advertise time which they do not expect to make except under rare circumstances.” (September 3, 1905)

What next?  Our sense of what we want to do and what is possible is rapidly evolving.  Simply having the scanned field notebook pages completely changed our game plan.  Before Wednesday of this week, we just had transcriptions.  Now we have the whole enchilada.  What we currently want is a no-cost, minimal effort system that will make scans AND transcriptions AND annotations available, and that can facilitate text mining of the transcriptions.  Do we have that in WikiSource?  We will see.  More on annotations to follow in our next post but some father to a sister of some thoughts are already percolating and we have even implemented some rudimentary examples.

We’d like to encourage you to try your hand at transcribing or annotating this notebook along with us, and to let us know what you think about the process (reminder: Henderson’s first field notebook is still available as plain text or as a Word document).  As on Wikipedia, all edits are saved, so you can’t really mess up – be bold, jump in (!) and tell us what you think.