The Galen Palimpsest and the Modest Ambitions of the Digital Data Set

ABSTRACT

The digital Syriac Galen Palimpsest (SGP) data set is an archive built on the model of the digital Archimedes Palimpsest. As with Archimedes, the SGP data set is meant to promote the long-term preservation of and access to the digitized palimpsest. The SGP data set follows archiving best practices and uses the Archimedes Palimpsest Metadata Standard for spectral imaging metadata. The data is released under a Creative Commons Attribution 3.0 Unported license (CC BY 3.0). The SGP project used custom software to manage its data and metadata from the time of capture to final data set publication. In the years since initial publication, newly discovered leaves of the manuscript have been discovered, imaged, and added to the on-line archive. Since the publication of the SGP data set, subsequent projects have built on and refined the methods established by the SGP team by moving away from content-based file naming, establishing formal quality assurance practices, increasing automation in the creation and management of data and metadata, and including full bit-depth capture images in the digital product.

The digital Syriac Galen Palimpsest (SGP) data set is an archive built on the model of the digital Archimedes Palimpsest. As with Archimedes, the SGP data set is meant to promote the long-term preservation of and access to the digitized palimpsests. The work was accomplished through careful design of the archive following recommended best practices and by adopting a liberal open license to ensure the easy use of the archive and its contents. It is designed to be a self-contained, self-documenting, verifiable data set, amenable to both human and machine access. The data is released under a Creative Commons Attribution 3.0 Unported license (CC BY 3.0), which allows anyone to use the data in any form for any purpose—even commercial use—provided the user gives attribution to the owners of the documents therein. The design of the archive and the openness of its licensing contribute equally to the digital information's ease of use and sustainability, both of which are vital to the data's preservation.

The SGP project was the first time the Archimedes data-capture methods and data model were adapted to a new project. The SGP established a pattern of practice used by the project team for subsequent spectral imaging projects. The effort was focused on efficient and low-cost production of spectral imaging products. This article will look at the purpose of the data set and its content, the process of its creation, its life since initial release in 2010, and how the team's practice of building spectral data sets has changed in the years since.

Preservation and Access

Books, especially parchment books, can last for centuries. They are not, relatively speaking, fragile or volatile. Digital information—bits stored by altering the charge of microscopic dots on electromagnetic platters that spin at thousands of revolutions per minute—is changeable by design. While some books do degrade and sometimes the text is forcibly removed from their pages, to a large extent, short of completely destroying pages, that information is hard to eradicate completely, as the results of spectral imaging have shown. Digital files become corrupt. Hard drives fail. The hardware needed to read certain media become difficult to find. File formats become outdated and sometimes unreadable when the programs that created them cannot be found or, when found, run on existing hardware.

Codex books are readily comprehensible. The conceptual order and structure of a manuscript's content are an inherent part of its physical order and structure. The physical mechanics of a book and the process of its manufacture help to ensure order and sense. Not so with digital data. A computer hard drive does not ensure readily comprehensible order. A computer's presentation of folders and files that humans use to understand its content is a kind of projection of otherwise incomprehensible and discontinuous clusters of ones and zeros distributed on a hard drive's platters, stored on solid-state chips, or taking up transient space in computer memory. What's more, the structure of folders and files is entirely up to the creator of the content. Computers do not enforce order and structure in the way the codex format does.

The vulnerability of the parchment manuscript as storage medium is not its durability, but rather its relative scarcity. Books that are not reproduced in great numbers are especially at risk. The vulnerabilities of the digital disk as storage medium are its volatility and the fast-changing technologies that employ it. In the digital disk's favor are its low cost and the fact that it can be duplicated perfectly and rapidly an indefinite number of times. When we seek to preserve a digitized manuscript, especially when that digital information is complex and expensive to come by, as with spectral imaging and processing, we must work to overcome the vulnerabilities of digital storage and exploit its advantages.

The undertext of the Syriac Galen Palimpsest, like that of the more famous Archimedes Palimpsest, has been preserved for centuries on parchment leaves.¹ The modest goal of the project was to create a data set of spectral images that we hope will last for decades. To do this, the pattern established with Archimedes was followed. The Archimedes data set's design was informed by the Open Archival Information System (OAIS) reference model.² OAIS describes a model for the ingest, maintenance, and preservation of archival information. Key to the model are the collection and inclusion of data needed to preserve and access the information now and in the future. This includes technical and descriptive metadata—what file formats are present and what their content is—as well as information needed to verify the completeness and integrity of the data, and to migrate that data to new file formats and media as the old ones become obsolete. As such, the digital SGP is a self-documenting, verifiable set of archival-quality data and metadata.

The risk to that data and metadata is minimized by employing international standards and commonly used well-known file and metadata formats. Such practice serves two purposes. First, common, standard formats decrease the difficulty of working with the files in the present and migrating them to new formats in the future. Second, the likelihood of data use, and thus data duplication and proliferation, is increased by the use of common, popular file types. For core data, the digital SGP uses TIFF for archival images and UTF-8 encoded Unicode text documents for metadata.³ Metadata is also stored in each TIFF file's ImageDescription tag. Finally, open licensing is used for the image and metadata content. All content in the SGP archive may be used without prior permission of the archives that hold the manuscripts. This last point is of critical importance. The open licensing of the material, via the Creative Commons Attribution 3. 0 Unported license (CC BY 3. 0, https://creativecommons.org/licenses/by/3.0/),⁴ is as important as the structure of the archive and the use of standards. Without open licensing, the free exploitation of the data that is vital to its longevity is impeded.

The Structure of the Digital Palimpsest

The structure of the data set, the names of the files, and the included metadata all serve their part in the digital SGP. While it is available on the web at http://digitalgalen.net, and currently hosted by the University of Pennsylvania Libraries, the digital SGP is not a website, nor is it a database. It is a single set of files arranged under a single directory structure. All its files are core data or in some way support the understanding and use of that data, by humans and computers. The top-level directory, the root, of the digital SGP is shown in figure 1. As you can see, the root directory has four files and five subdirectories. The files are an introductory Read Me file and a list of files in the archive. Both are in text and HTML formats. The directories are:

• • Data—the core image data files and their supporting information

• • Documents—documentation, project-specific documentation in an "Internal" subdirectory, and documentation of the standards and file types used in the project (such as TIFF, XMP, and MD5) in an "External" subdirectory

• • ResearchContrib—important and useful image files that in one way or another do not conform to the standards for images in the core Data directory; for example, details or experimental images of the palimpsest leaves

• • Supplemental—alternate presentations of source material used to generate text and other content included with the core data; currently empty⁵ • Support—functional files used by the archive or the data set; currently the Cascading Stylesheet (CSS) file used for the appearance and style of the archive's HTML documents

The focus of the archive is the Data directory. It contains the image files and the metadata that directly supports them. Figure 2 shows the first few subdirectories of the Data directory. Each of these directories contains capture and processed images from a single shot sequence of a bifolium of the disbound SGP. A portion of the contents of the directory for bifolium 42v+43r is shown in figure 3. This directory contains twenty-seven archival TIFF images. Of this number, twenty-three are "pack8" files. These are 8-bit versions of the grayscale 16-bit capture images that have been computationally adjusted for visibility. The remaining four are processed images. These were computationally generated from select capture images to enhance the visibility of the undertext.

Figure 1.

SGP home directory.

Each TIFF image file is accompanied by three other supporting files. These are a JPEG thumbnail image, an XMP "sidecar" metadata file, and an MD5 file containing the TIFF file's checksum digest. In more detail:

• • The JPEG thumbnail file is one-tenth the size of the TIFF image of the same name. It is intended to provide a quick web preview of the TIFF's content. Example: 042v-043r_A_0365_pack8. jpg.⁶

• • The XMP file contains the TIFF image's metadata following the Archimedes Palimpsest metadata standard.⁷ It provides direct access to the metadata embedded in the TIFF's ImageDescription tag. Example: 042v-043r_A_0365_pack8. xmp.⁸

  

  Figure 2.

  Portion of the Data directory.

  

  Figure 3.

  Image directory for SGP bifolium 42v+43r.

• • The MD5 file contains the MD5 checksum digest of the TIFF file⁹ and is to be used to verify file integrity. Example: 042v-043r_A_0365_pack8. tif.md¹⁰

Digital Galen Metadata

The digital SGP uses the Archimedes Palimpsest Metadata Standard (APMS) for images.¹¹ The APMS defines six types of information:

1. 1. Identification information—identification of the individual image and its lineage (field prefix: ID_)

2. 2. Spatial data reference information—relation and orientation of the image to the physical object (field prefix: SPTL_)

3. 3. Imaging and spectral data reference information—parameters of the image's capture (such as aperture and exposure) and spectral characteristics (such as illumination wavelength and filter) (field prefix: IMG_).

4. 4. Data type information—file format information and technical provenance (such as processing techniques) (field prefix: DAT_).

5. 5. Data content information—content descriptive information, such as keywords and foliation information (field prefix: CONT_).

6. 6. Metadata reference information—identification of the metadata standard used and its version (field prefix: MET_).

Several of the descriptive elements in the APMS are taken from the Dublin Core Metadata Element Set. The Dublin Core Metadata Element Set provides a set of fifteen terms for the description of resources, typically resources found on the web.¹² Dublin Core terms are commonly used on the web to provide basic, machine-readable descriptive metadata, like Title, Description, Subject, and Date. In the APMS, Dublin Core terms have the expected names, like 'identifier', 'date', and 'subject'. All other elements have prefixes that indicate the element's information type. These six information types can be thought of as axes of information that intersect uniquely in each image. For example, all twenty-seven images of folio 42v+43r have the same data content information, but only one of those images has the spectral information associated with the 365-nanometer capture.

The APMS information types are expressed in over one hundred separate elements. Which elements are used depends on the image. Capture images have imaging and illumination information associated with them. Processed images, on the other hand, which are generated from multiple capture images, lack most imaging and illumination information, but are rich in data content information that describes the tools and techniques that were used to generate it. For an example, see the Appendix, which has the full text of an XMP sidecar file for an 8-bit version of a capture image and shows all the APMS metadata elements used.

Figure 4.

ImageDescription metadata excerpt.

The full metadata for each image is also stored in the TIFF's Image Description tag.¹³ Figure 4 shows a portion of the metadata extracted from 001r-004v_C_0365_pack8. tif. Note the Dublin Core elements 'identifier', 'date', 'creator', and others, as well as the APMS specific-fields with prefixes, like 'ID_File_Name' and 'SPTL_Grid_Coordinate_System'.

Collection of the Capture Data

Building on the experience of building the Archimedes Palimpsest data set, the SGP project was the first of a number of iterations in the development of an integrated system for the collection of spectral imaging data and metadata. It was known from the beginning exactly what metadata would be collected and what form the data set would take. The goal was a rapid and low-cost turnaround from the time of capture to publication. The team designed an integrated workflow that included both human and software processes for the collection of data and metadata from capture through processing. This system included software to manage the entire life cycle of the metadata, worksheets for collecting system geometry, and spreadsheets to collect structured information about image processing methods. The software was a Ruby on Rails application called the Multi-Spectral Imaging Manager or MSIMM. All metadata collected by the project was eventually entered into MSIMM.

The one-hundred-plus data elements in our implementation of the APMS represent multiple strands of data and come from multiple sources. Data for content information came from scholars and manuscript experts. Some information, like exposure time and camera settings, was collected by the camera software and embedded in each image's header. Detailed information about the illuminants and filters was provided by scientists on the project, as was detailed information about techniques used to generate processed images. Likewise, information about the camera system's geometry—the relative positions and angles of camera and lights to the imaged object—had to be collected by hand. The identities of project participants for acknowledging contribution to the project also had to be collected.

The first link in the chain of data collection came from MSIMM, which generated base file name components. The file name was the lynchpin of data management for the SGP project. Data managers, scientists, and soft-ware that worked with the data relied on the file name to identify image content and circumstances of image creation. The file name conveys each image's unique identity.

Before a bifolium was imaged, MSIMM was used to create the first segments of the filename. Take, for example, the filename 016v-021r_A_073 dng. The initial two segments say that this is an image of bifolium 16v+21r, and it is from the first sequence of images captured of that bifolium, designated by the letter A. If a bifolium was imaged again, sequence letters B, C, D, and so on would be used. When a user entered the folio or bifolium to be imaged, MSIMM would generate a series of possible file name bases.

• 016v-

• 021r_A 016v-

• 021r_B 016v-

• 021r_C 016v-

• 021r_D 016v-

• 021r_E 016v-

• 021r_F

The camera operator would enter the first of these base names, '<EXT_CODE>016v-021r_A</EXT_CODE>', into the camera software, MegaVision PhotoShoot, which used it to create the file names of captured images. Any important information about a set of shots was logged in MSIMM for future use. This log was exported to Excel and distributed to project members. Figure 5 is a portion of the log that shows how the notes were useful later, especially for understanding why there may be multiple shot sets for a bifolium and which of them to use.

PhotoShoot provided the next link in the data management chain in the form of a shot designation code. PhotoShoot was configured to manage the spectral light system (built by project scientist William A. Christens-Barry), as well a filter wheel that would move color filters in front of the camera lens for certain exposures. Each exposure in the capture sequence was configured in PhotoShoot's n-shot table with its own exposure time, light sources, and optional acquisition filter. These configurations were represented by twenty-three four-character codes, shown in figure 6. The file base-name component and exposure code constitute the unique identity of each image. These codes and the corresponding configuration information were added to MSIMM's database, as were all other relevant metadata values, like spatial reference information and camera setup geometry. To construct a capture image's metadata, MSIMM would parse the image's file name, collect the relevant metadata based on bifolium, shot sequence letter, and capture code, and combine that with metadata extracted from the image's header. This metadata was output in two formats: XMP serialized as XML and text file of a list of name-value pairs for insertion into the image's Image-Description tag.

A similar method was used with processed images. Separate codes were created by project scientists for each of their processing methods.

Figure 5.

MSIMM log data.

Figure 6.

Capture codes.

Figure 7.

Processing worksheet.

The details of these methods were provided in Excel spreadsheets filled in by the project scientists and loaded into MSIMM. An example is shown in figure 7

From the data management perspective, an application like MSIMM is a source and hub for project data and metadata. The identifying file name components originated with MSIMM. Images with those file name components were captured, collected, and then duplicated multiple times for distribution to image scientists for processing. Once processed, new images, bearing the same identifying components, were returned to the author, the project's data manager, for assembly and publication. The list of all those files along with exported metadata from them was loaded into MSIMM. Once all required metadata from all sources was entered, MSIMM was used to create final metadata. Custom scripts were used to build the published data set from the processed images and MSIMM-generated metadata.

Syriac Galen Data Since 2010

As reported in the New York Times, Grigory Kessel has uncovered six more leaves of the SGP, at Harvard's Houghton Library, the Bibliothèque nationale de France, the Vatican, and St. Catherine's Monastery in the Sinai.¹⁴ Images of these leaves are online with the rest of the digital palimpsest.¹⁵

The later images were captured by separate projects, with different equipment and methods. As such, their metadata is not as full as that from the 2010 session. They were, however, captured or processed by all or part of the same team that worked on the first session. The format of the file names and thus the appearance of the data is similar to that from 2010.

The later images were added quickly in 2015 and 2016 to ensure their availability and the completeness of the archive. As of this writing, the Read Me file, file list, and documentation have not been updated to reflect the new data. When those documents have been updated and corrected, the SGP data set will form a complete digital record of the known Syriac Galen Palimpsest.

Spectral Data Sets Since 2010

Since 2010, all or part of the team that worked on the SGP project has participated in other spectral imaging projects. The author has worked on two significant projects of this kind, the National Endowment for the Humanities–funded David Livingstone 1871 Field Diary project¹⁶ and the Arcadia-funded Sinai Palimpsests Project.¹⁷

Unlike Archimedes and Galen, the Livingstone and Sinai projects required working with multiple documents. These more complicated projects presented new challenges for the collection, management, and presentation of data. It is important to understand that in all these spectral imaging projects, a data manager assembles hard drives of captured data and sends them to multiple imaging scientists for processing. The scientists send the resulting processed images to the data manager, who must collect and collate them, and prepare them for publication with complete APMS metadata for each file. This requires not only tracking hard drives and outgoing and incoming files, but ensuring that the images can be associated with the one-hundred-plus metadata elements that uniquely describe each file. On a relatively small project like the SGP, there are several thousand files. At the other extreme, the five-year Sinai project dealt with over 60 documents and fragments, over 6,500 folio sides, and approximately 600,000 image files.

A hard lesson learned on the Livingstone project was the risk of using content-identifying information—namely, shelf marks and page numbers—in file names. Hundreds of files from one of the Livingstone documents were misnamed when, during imaging, two pages were accidentally turned instead of one, throwing all page numbers off from that point forward. This complication affected both the files and the database that had associated capture metadata for those images. Not only did files need to be renamed, but a change in the naming format affected both the files and the database records that depended on those names and required the manual correction of the associated records in the version of MSIMM that was used for that project. Many hours were lost addressing this problem.

Also complicating the Galen and Livingstone projects was the need to collect required, detailed metadata about processing parameters from the imaging scientists. Very often the information was collected well after processing had been completed, making information-gathering time-consuming and inefficient. The association of processing metadata with the correct files relied on parsing the file names in order to link images with the correct metadata, a method susceptible to error.

The scale of the Sinai project required different approaches for file naming, quality control, and the handling and processing of metadata.

First, the reliance on content information for file names was done away with. KatIkon, a new cataloging and data management application created for the project, assigned arbitrary numbers to each sequence of images. Most folios were shot as a single sequence, but if a folio was imaged twice or more, each time it would be done using a different shot sequence number. The relationship between this number and content metadata was maintained in KatIkon's database until an image was ready for publication. Then each file was renamed with the subject's shelf mark and folio number. Before that time, if an error was discovered in the association between the folio and the shot sequence number, it could be corrected in one place in the database without the need to change file names. Because arbitrary sequence numbers make for somewhat cryptic file names, project systems and practices were developed to help team members identify the content of images before publication. These included the distribution of contents lists and the addition, to each directory of images, of a text file named with the sequence's shelf mark and folio number.

Second, the practice of manually entering file names in the camera software was eliminated, and a system of post-capture independent verification and validation (IV&V) was used to ensure quality and accuracy. For file names, KatIkon was used to generate an ordered shoot list of folios to be imaged with their identifying shot sequence numbers. For the project, Megavision added a feature to the PhotoShoot camera software to manage imaging and name files based on the computer-generated shoot list. The operator would select the correct shot sequence from the shoot list, and PhotoShoot handled the rest, using the shot sequence number for the beginning of each file name. Immediately after a sequence was completed and all files were written to disk, the IV&V operator inspected the images to verify accuracy, rotation, and image quality.

To manage processed image metadata, a system was established for validating and packaging processed images called "spindle."¹⁸ To be valid, processed images had to have required metadata in their headers and conform to the project's requirements for valid file names. Imaging scientists used a spindle script called "deliver" to validate and then package a directory of images for delivery to the project system administrator. Upon receipt, the system administrator would validate a package of images by running the "receive" script. Other scripts in the spindle suite were used by the system administrator to add capture and processed images to the project's working repository and to extract metadata from them to be added to KatIkon's online database. This data, along with cataloging and other information, is used by KatIkon to construct zip archives of metadata and packaging instructions used by custom scripts to assemble packages of data for delivery to scholars and for archiving.

A final innovation of Sinai project data delivery was the inclusion of 16-bit monochrome capture TIFF images with the 8-bit monochrome and 24-bit color processed images. The SGP does not include the 16-bit capture images from the project. This decision was made following the Archimedes project. That data set is one terabyte in size, which was quite large in 2008 when the data was published. The overall size owes in part to the very large size of the individual files, each of which is a 256 MB, 24-bit color TIFF. For Archimedes, the 48-bit color capture images were excluded for two reasons. For general access, the 24-bit images had to be included because most commonly available image software supports only 8-and 24-bit files. Second, the full-depth 48-bit images were each 512 MB. Their inclusion would have tripled the size of the archive to an unmanageable three tera-bytes. The selection of the 8-and 24-bit images for the Galen project was a holdover from Archimedes, even though the 8-bit monochrome files were 37 MB, meaning that the capture images were approximately 75 MB each, and would likely add only another 500 GB to the 300 GB of the present archive. In the intervening seven years, storage costs have come down considerably. Given the archival goals of the data set, the decision to exclude these best-quality capture images is worth reconsidering.

Conclusion

The Syriac Galen Palimpsest project established a standard method for collecting spectral imaging data and assembling a data set. Archimedes offered the template for the project's output, but Galen was the first such project that began with a clear idea of a final product, with a clear roadmap of how to proceed. The project was not without its complications and areas that were later improved upon. Nevertheless, later projects have been exactly that: improvements and refinements of the method established with Galen.