Vera Research Progress Part 4 - Audio

For those of us most concerned with Standards and Best Practices for Audio Preservation, here is the sum-total of my research organized and collated for your viewing pleasure. For your information, anytime I interject in the course of this post, I will use italics.

I. Introduction - The Deliverables

According to Shannon Roach these will be:

1. Assess how much content there is, and how quickly the collection is likely to grow
2. Figure out the best storage for audio archives (naming conventions, file format, where to store them)
3. Figure out how to make these audio archives searchable
4. Document all of the above & explain it to Vera staff

II. Present Arrangement of Vera Audio Materials - A Conversation with Jeff McNulty

Since for whatever reason I decided to take this rather charming picture before Jeff gave me the audio tour, I thought I'd include it here just for fun. For those of you who do not know, Jeff is on the left and Nelson, one of the Vera audio staff, is on the right.

File formats used: WAV, AIF, mp3, ptf (protools file), and digital audio tapes (DAT)

A. The Hard Drives:

1.) 300GB HD - Live Shows & Mixdowns

Main folder: Live Show Mixdowns

>Sub folder: Name of band, date, initials of who recorded it

• File name: whole show, date
• File name: individual song,

>Sub folder: Rough mixes (day of show)
>Sub Folder: Live shows mp3s
>Sub Folder: 2007 live shows

2.) 500GB HD – Two partitions: mostly show back-ups

Main Folder: Vera Backup A-M
>Sub Folder: name of band, date, initials

Main Folder: Vera Backup N-Z
>Sub Folder: name of band, date, initials

3.) Vera 1, 2, 3 recording drives
Protools session files organized like so:

-One folder for a whole show named by headlining band, date, initials of who recorded it
This folder includes files for the opening bands as well

-Eventually all bands will have their own separate folder

-The recording drives employ a system of highlighting

• Green: indicates a folder whose contents have been mixed
• Orange: indicates a folder that is ready to be backed up
• Red: indicates a folder that has been backed up

B. Basic Workflow:

1. For live shows and recording sessions, they record to a recording drive
2. These files are then mixed
3. Then they are backed up onto a DVD (the protools files)
4. Then the protools files are transfered from the recording drive to a Back-Up Drive

They like to keep the recording drives as empty as possible - nothing stays in there permanently

C. What Jeff wants to see in the future:

- Record all DATs to their backup hard drives

- 3-4 backups

1. DVD backup for all protools files,
2. One or two terabyte hard drives connected to the server
3. I think he may have said he also wanted drives that are not connected to the server
4. Hard Drive + CDs for audio backup this would be for mp3s, podcasting etc...

- All dvd and cd backups organized and cataloged
- Copies of records and CDs for bands who recorded at Vera

D. Current Issues:

1. Toast: the standard for CD and DVD burning in the Audio engineering community

• Jeff is concerned with future compatibility because there is a new version of Toast and it has recently changed the way one can access a DVD or CD Older CDs and DVDs burned using the old Version of toast cannot be viewed with the new version of Toast.
• I also think he said the new version of toast requires that you have toast to view the files I could be wrong.
• Jeff wants something for DVD burning that will be more future compatible Jeff also mentioned an issue with toast wanting to automatically convert wav files to aif and how there is a cross platform (PC vs. Mac) issue with this.

2. Issue of cross platform compatibility for tagging and metadata

• The audio folks use a mac. Mac has a different layout when you right click a file and check its properties. On a mac there is this thing called "Spotlight comments" where you can type in anything you wish about a file and it is then easily searchable.
• My personal experience with this is that it is not cross compatible. I have an external hard drive that i use to store all my internship files and personal files. I own a mac and use the spotlight comments especially for my audio recordings. When connecting to a PC and right clicking the file and choosing "properties," the comments do not show up. Not only that - and if i remember correctly - when I returned to my Mac and viewed the file, the spotlight comments had been erased. This is a major problem if we go the windows explorer tagging and metadata route.

III. Individual Issues in Audio Preservation

A. Storage

1. File naming conventions:

According to the Colorado Digital Program (CDP):

"Systematic file naming is important for system compatibility, interoperability, and to demonstrate ownership of the digital asset. File naming conventions specific to an institution may be used. These might include protocols that include institutional acronyms, collection identifiers, or part designators, among others. In general it is recommended that the characters in the file names be alpha-numeric, lowercase, and not utilize spaces, tabs, commas, periods or any other characters reserved for computer system use, such as slashes, asterisks, or question marks."

Seems pretty obvious enough to me.

2. Hardware:

Hard Drives and particularly RAIDs (Redundant Array of Independent Disks) seem to be the general consensus for Audio storage.

The CDP defines a RAID as:

"A self-contained collection of disk drives that act as a single large hard drive storage system. These configurations are designed to enable a system to operate when an individual drive within the array fails, thus minimizing and ideally eliminating the potential for the loss of data. There are ten types of RAID configurations. In general, the better choices are RAID 5, 6 or 10."

Gary Louie, University of WA Music School archivist has the following to say about RAIDs:

• "Hard drives are the only practical method now to store the size and amount of files we have. Hard drives are also poor for reliability, so any HD storage has to be RAID of some sort, and must be duplicated in at least one other physical location."
• "Some things we are looking at for local, small scale archive "starters" are small RAID hard drive towers with removable sleds, duplicated onto similar arrays via gigabit ethernet in another building."
• "You can certainly consider a really small scale HD storage method that will still be helpful to you on all counts at low cost. An external HD holder (like the Newertech Voyager) and a handful of bare HDs, with constant duplicates and stored at different locations. Could be in conjunction with a small server system."

ARSC Technical Committee on RAIDs:

"RAID arrays remove some of the disadvantages of individual hard drives while they provide larger capacities than individual drives. They are extremely easy to use, especially when supplied as network-attached storage, but still should be considered mechanical devices that may fail at some point in time. A single RAID array is not adequate for long-term storage."

For a more in-depth discussion of various RAID configurations go here.

Beyond the consensus on RAIDs as the best way to store digital audio, there is also a consensus that audio be stored in multiple locations and not on a single array.

The ARSC Technical Committee suggests:

"Generate local backup copies of all files as soon as possible after creation
Store backups on a separate device or separate media from the original.
Make at least two, or preferably three sets of archival master digital files of preserved content and store them in different locations, and possibly on different types of media."

The CDP suggests the same:

"It is highly recommended that organizations utilize a strategy employing multiple redundant copies of digital files in separate locations, as a fail-safe strategy for the failure or destruction of the digital media. Acceptable media may be optical disks, hard drives, or magnetic data tapes – all of which have particular strengths and weaknesses."

Depending on the needs and resources of an organization offsite storage is a potential option. This is where I believe our connection to UW could come in very handy.

The CDP says:

"Projects considering outsourcing should consider the costs of ingestion and ongoing maintenance fees; metadata requirements for outsourced files; workflow requirements; and accessibility requirements."

Vera's consulting "Technical Generalist" Darren White says:

"In terms of hard disk storage, VERA should be looking at a NAS/SAN on the VERA internal network, preferably with offsite backup. That way, if disaster strikes..."

Other options such as DVDs and CDs are not recommended.

The ARSC Technical Committee has the following to say about CDs and DVDs:

• "Optical discs [are an] unsafe preservation storage media. Using this format requires costly test equipment to check the quality of both blank media and digital recordings. Even if this equipment is already available to an archive, the cost of storage media may be higher than the other formats, except in very small volume – individual CDs or DVDs are not expensive, but the cost-per-megabyte is substantially greater than other media types."
• "The format cannot hold files at higher bit depth and sample rate than 16 bit, 44.1 kHz, which is far less than the de facto 24 bit, 96 kHz standard. In addition, migrating an archive stored on optical discs is expensive, because it requires much human intervention."

What the CDP says about CDs:

• "Storage and retrieval costs will escalate as CD collections become larger and more challenging to manage."
• "Limited physical life span (3 to 20 years) and the files stored on them are vulnerable due to physical deterioration, mishandling, improper storage and obsolescence."
• "Obsolete equipment for reading the media poses a threat in the long term."
• "Adhesive labels and permanent ink markers can cause early failure of CDs through chemical interaction with the CD’s recording layer."

Gary Louie, University of WA Music School archivist, says:

"Don't trust DVDs. No one thinks they will last very long. Same for CDs and tapes. Generally, the data written to them is also not very clean, especially audio CDs."

3. Data Migration and Hardware Updating

Current hardware and software seem to go obsolete every 3 to 4 years. It is therefore extremely important to implement a Migration schedule.

The CDP Suggests:

• "Migrate your files frequently - at the very least every five years. Failure to incur these small incremental costs may lead to very large costs down the road when you need to migrate froman obsolete data format or medium."
• "Every sustainable digitization project should include the costs of data migration as a yearly budget line item."
• "At the time of migration, file integrity checks should be conducted... a “checksum” may be associated with a file when it is produced so that the checksum value may then be used to confirm the integrity of the file at the time of migration."

The subject of "checksum" makes for a great point of transition into the subject of quality control.

4. Quality Control

The CDP Recommends the following for quality control:

• All digitized audio files should be sampled for sound quality.
• Technicians charged with quality control should listen for consistency in the audio quality at a number of points in the recording, listening for distortions in the sound, for proper playback speeds, and for artifacts such as hiss and hum.
• Technicians should specifically check that the volume levels are set correctly.
• Recordings should also be checked for completeness.
• Any distortions in the sound or other inconsistencies in the recordings should be noted in the metadata.
• Metadata should also be checked for accuracy and completeness, with special attention paid to accuracy of file names, which can lead to the effective loss of files when recorded inaccurately.
• In addition, file integrity checks may be conducted. A “checksum” may be associated with a file when it is produced so that the checksum value may then be used to confirm the integrity of the file at each step from creation to migration.

"A checksum is an algorithm-based method of determining the integrity and authenticity of a digital data object, such as a digital audio file, used to check whether errors or alterations have occurred during the transmission or storage of a data object. Please note, however, that a checksum will only tell you there is an error. It cannot tell you what the error is, nor can it correct the error."

"It is important to understand that quality control of audio can be a very subjective process. It is recommended that technicians performing this process be selected on the basis of their familiarity with a variety of types of recorded sound, and preferably have experience in a field that has already helped train their ears. Staff with previous audio experience such as audio engineers, are a natural selection, as are staff with formal musical training."

5. File Format, Sample Rate, and Bit Depth Standards

While in general both WAV and AIF formats are widely used there appears to be the emergence of a new format the BWF which is exactly the same as the WAV format, except that it captures metadata.

The CDP says the following regarding WAV and AIF:

"The WAV file type was developed by Microsoft, is in widespread use, and is readable by virtually all audio software programs. The AIF file type was developed by Apple Computer and is also in widespread use. Both of these file types are uncompressed and acceptable for long-term file storage. The WAV file type has become a standard and is recommended. In addition, the WAV file type is also available in a professional flavor, broadcast WAV (BWF), which has the capability to store metadata in the file header. Although not all audio software programs are currently capable of reading or writing to the metadata header, the BWF format is emerging as the WAV file type of preference for archival audio projects."

For the sake of avoiding redundancy, I will simply say that all other sources considered thus far, are generally on the same page regarding WAV and AIF. However, since Vera also uses protools files (ptf) there are other issues to consider.

Regarding Protools files, Gary Louie said the following:

"If you are thinking long-term (10-50-100 years) who knows if anyone will be able to read ProTools files at all. The international recommendation for audio storage is the Broadcast WAV file, which holds the audio plus metadata for catalog and technical info. But that doesn't help much with PT sessions. We (i.e. University of Washington) don't do any PT work, so you should probably ask elsewhere (maybe the ProSoundWeb or Digidesign's forums) on how people archive their PT material."

I have yet to research archival standards for protools files, if there are any. When I find the answer I will update this post. However, all things considered, the new BWF seems to be the optimal choice for Vera's needs.

Below is a diagram from the CDP regarding Sample Rate and Bit Depth Standards:

6. Metadata Issues and Standards

As established above in my conversation with Jeffery, retention of metadata when working across platforms (Mac to PC) is a major issue. Depending on how thorough BWF is and if it is cross-compatible, there are other options to consider.

Darren White suggested the following:

For maximum portability of metadata, I would recommend that it be stored externally to the file itself. Even with, say, MP3 files and ID3 tags, cross- platform compatibility issues remain, even after all these years. So WiMP and iTunes might make entirely different sense/nonsense out of the same ID3 tags. I also use a Mac primarily and I think if we are talking Protools and AIFF files, Mac capabilities may be sufficient for basic metadata but to allow for expanding requirements, something like a CMS and database or even just an external XML file may make more sense in the long run.

The Arts and Humanities Data Service Perservation Handbook for Digital Audio suggests:

"Ensure that associated metadata integrated into the file itself can be extracted in the software tool and stored in the preservation format. If not, the information should be manually output to an ASCII text file."

ASCII or XML might be the way to bridge the gap between Mac and PC. If we cannot attach metadata and tags directly to the audio files we can create a text file with all the pertinent information about each file. Someone can either create one text file to match each audio file with an identical file name or just one text file for a group of audio files within one folder. I lean a bit more towards individual files.

Sound Directions, a joint project between the University of Indiana and Harvard College has an excellent resource on metadata standards, which is much to large to duplicate on this blog, so I have made it available to you here.

The complete publication is available here.

VI. Online Sources

1.) ARSC Technical Committee on Preservation of Archival Sound Recordings

http://www.arsc-audio.org/pdf/ARSCTC_preservation.pdf

2.) Arts and Humanities Data Service Perservation Handbook for Digital

Audio http://www.ahds.ac.uk/preservation/audio-preservation-handbook.pdf

3.) Colorado Digital Program Audio Best Practices

http://www.bcr.org/dps/cdp/best/digital-audio-bp.pdf

Briefly Back On The Air - An Archival "Res Gestae"

Hello everyone. For those of you not directly involved with this project, you may be wondering why I haven't updated this blog for a while. The answer is that I've become extremely busy with the actual work and as a result talking about the work has taken a back seat. At the moment I am on vacation, but i thought since I am mostly taking it easy today to recover from jet lag, I would give you all a few updates. A lot of has changed since the day before thanksgiving. Here is the Res Gestae, Latin for "Things having been done"

*I Submitted a progress report detailing the current state of both the physical and digital archives and my recommendations for moving forward.

*I Took on new responsibilities and shifted my priorities.

*First, we have 500 dollars on the Vera budget for storage supplies and Josh generously offered 200 to get things rolling.

*I Completed a rough inventory, measuring poster sizes and counting how many we have.

*Josh and I hashed out an estimate for supplies after finally deciding to go the folder route instead of the binder route.

*The Audio side of the Archives has now become a top priority so, I have been doing research into best practices.

*I will soon be coordinating volunteers to assist me in taking a more detailed inventory of the collection and to help implement the initial reorganization of the physical and digital archives.

*The work as of late has been very Vera heavy and while I am still cataloging zines for ZAPP, a conversation will soon have to take place to make sure Nora gets what she wants for ZAPP out of this internship as well. I know that this will eventually include activities such as coordinating Zine cataloging groups.

*My general duties from this point forward (at least on the Vera side) are the follow:
1.) Determine and layout standards and best practices for organizing and preserving all historical materials at Vera.
2.) Create clear and concise instructions and initiate implementation to set an example for future interns and current staff to implement these standards on their own.

For those of you who are a part of the project, be sure to inform if I am missing anything pertinent here. I do not have notes with me and this is a very basic round-up from memory. I just wanted to give a quick update.

Vera 11/25/09

After my conversation with Josh (see details below), I decided to return briefly to the physical archives to finish up the inventory there on the file cabinets. After a while of taking rough notes on what was in each drawer I realized that a previous inventory already existed. So instead of re-listing each item I looked at the existing inventory and matched it what i saw in each drawer and simply noted any additions. I decided to take some pictures of the drawers' contents to give you an idea of what I am encountering.

First of all... I want I want! Yes, I was excited to see these posters. I am quite a fan of these guys and the best part is we are from the same town - Chicago. Returning to what I actually came here for though, I was simultaneously unexcited to see these posters. As you can see, they are curling and they catch on the drawer above whenever you pull the drawer out. This means just to look at these posters you have to damage them. :'( Make it stop make it stop! ... wait, thats my job! Ok, I'll figure it out somehow. Anyway, there are also several other mixed items stored under these posters including various press materials, photos, videos, and other posters.

Skipping ahead, here is what I found in the 2007 drawer. A massive mixture of items are overfilled into this drawer and as you can see the manilla folders are brushing up against the drawer above. This was one of those moments where I began to feel a bit frustrated with inventory. Its difficult to decide the best way to approach a situation like this. I mostly have a strong urge to just start taking everything out and redoing everything, yet I realize these materials do not belong to me and I am reluctant to make changes until there is a practical option available that everyone has agreed upon. Ultimately, its this overwhelming lack of efficacy in the moment with the realization of my limits in time, funding, and authority. Ok, just take a deep breath, this is one of those moments where one's skill in deferral of gratification is most crucial.


Moving on, I opened the 2008 drawer and this is exactly as I found it. The way things are stacked in here, gravity will eventually (and has already) take(n) its tole. Basically, these materials will end up like someone who slouches at a computer all day (not me mind you, I have impeccable posture!) for 20 years and takes on a considerable hunchback. This is what made the existing inventory so relieving: I didn't have dig through piles of mixed materials. With this, I concluded my activities for the evening.

Summary:

5:00PM - Phone meeting with Josh

*Scope of Project

-Alix is supposed to be in charge of reorganizing the network drive - not me.

-I will be focusing on the Archives portion of the network drive and communicating with Alix regularly about any ideas I may have.

>for instance, we are considering having older materials on the network drive transfered into the archives.

-Josh suggested that I develop a policy for transferring files into the archives folder.

*Progress Report

-Josh will also be available December 9th

-Rough Outline:

I. Inventory

A. Existing organization

1. Physical Archives

2. Digital Archives

B. Condition of items in physical archives

C. Suggested changes

1. Physical and Digital Arrangement

2. Digitization standards

II. Interviews

A. General Trends

B. Individual staff folder trees / general usage areas (depends on space)

C. Staff suggestions

D. Special finds

III. Database

A. Skype Conference with Kevin from N-Power

ZAPP 11/25/09

Summary:

12:00PM - In The Archives

*Cataloging

-Popery #2, An anthology of comics by various authors

-Open Your Mind, a brief flyer by Max Trafk about socio-political issues of drug activity in Colombia

1:00PM - Meeting with Nora

*Progress: Positive and Negative Aspects of the project

-(positive) Research, for instance what i've learned about digitization standards and storage options. See research.

-(negative) aspects of the inventory such as the immensity of the network drive and realizing that I had conflated the Digital Archives with the entire network drive. When really its just a part of the network drive. Wrapping my head around the digital inventory has been rather daunting.

*Research suggestions

-Check multiple sources for standards on digitization

-Look at information organization standards

>Who handles IT at Vera? - contact these people.

-Apply research for Vera to ZAPP whenever possible

>this can include scanning and metadata standards.

-research the best option for bagging Zines

*Begin coordinating a progress report meeting

-Nora is available Wednesday December 9.

2:00PM - Back In The Archives

*Cataloging

- Steven's Comics, a collection of short comics about a boy who discovers he likes to play with dolls.

-the Unloved Potato Bug #1, a collection of comics by Aerick Mackintosh about dreams, tarot, finding coffee, and a goofy high seas adventure.

ZAPP 11/19/09

This evening it was only Alison and I in the Archives: Heather emailed us earlier today to inform us that she was not feeling well and would not be cataloging this evening. Other than that, it was another usual Thursday night at ZAPP.

Summary:

6:00PM - In the Archives

*Cataloging

- Vincent #7 by Rolan Bougie and Rebecca Dart, a comic about Vincent the Panda, a living life-sized stuffed animal who goes to see a movie in which he held a leading role among other stuffed animals.

-Win Tanes Bra by Stefan Neville and Clayton Noone, a comic about homelessness in New Zealand

-Noisy Desperation by various authors, zine related comics and essays and an essay from Henry David Thoreau about the Economy

-White Buffalo Gazette edited by Larned Justin, includes comics, drawings, and an essay about Imperial Japan.

Vera 11/19/09 - Mapping the Network Drive and Interview with Nick

Today I spent a significant amount of time exploring and mapping out the network drive, which contains the digital archives. I started by going back to Shannon, the managing director, and asking her to show me all the areas she uses within it. I thought this might be a good idea to do with all the staff - I could learn what areas are used most, where the work of various staff members overlaps, and what could be done away with. I also did this with Jeffery, the program coordinator, Tristan, the booking coordinator, and finally Nick, the development director. The latter with whom I also had an interview. By now you must be wondering what I've been learning from the staff interviews. This information is strictly proprietary at the moment. I decided it would be best to collate the individual results and post a comprehensive report all in one place. This report should hopefully be done by the end of the November and will also include my final assessment on the structure of the network drive. However, I will say that I have discovered some rather surprising and very helpful information... so stay tuned.

Summary:

1:30PM - Mapping out the network drive

*Revisit with Shannon

-obtained a list of network drive usage

*Revisit with Jeffery

-obtained a list of network drive usage

2:30PM - Interview with Nick

-details forthcoming

3:ooPM - Mapping out the network drive

*Revisit with Tristan

-obtained a list of network usage

Vera Research Progress Part 3

Iconclass: a classification system designed for art and iconography. It is the most widely accepted scientific tool for the description and retrieval of subjects represented in images (works of art, book illustrations, reproductions, photographs, etc.) and is used by museums and art institutions around the world.

comments: From what it appears, this system is incredibly thorough. I am uncertain how far I should delve into this, since I don't think the materials we are working with will require this much classification. Nevertheless, it is possible perhaps to structure a simpler system based on this one. ...More to come, stay tuned.

Introduction to Art Image Access - Issues, Tools, Standards, Stategies: An online publication exploring issues in image classification available at Getty

It Begins with the Cataloguer: Subject Access to Images and the Cataloguer's Perspective

1.) Primary Search tools: creator/author and subject

- many end-users tend to search for art images using the search criteria who created the work and what is the work of or about.

- comment: I think in Vera's case, for instance, band names would be the primary search for posters rather than the artist or subject - the latter being assumed.

However, I think subject matter for event photos would be crucial for all those instances where staff would need images that depict particular activities, people, or locations.

* Question: Does Vera even keep track of the artists who make the posters? If not, should we start doing this? - The same applies to event photos.

- titles or captions—alone or in combination with the subject matter element and other elements such as date, style, or period—are frequently employed by end-users seeking subject access to visual materials. End-user systems and user interfaces need to take this into account.

2.) Structure in Cataloguing

- Classification must move from a general visual analysis of the work as a whole to a detailed evaluation of its constituent parts.

- Slow down the cataloguer's mental processes in viewing the work of art, to ensure that it is being viewed in its entirety and not selectively

- Analyze the work of art in terms of overall subject matter and select a set of primary descriptive terms or keywords

- Catalogue the main visual elements, using a systematic approach to "reading" the work of art—left to right, right to left, top to bottom

- Determine and catalogue the meanings or symbolism in the subject matter as time, money, and expertise permit

..... to be continued .....

The International Poster Collection at the Colorado State University

comments: Someone plugged this on the Archives Listserv about 8 years ago and it looks like a great resource. It looks like they have most of their posters online and if you click on a thumbnail, a larger version comes up listing all the metadata below. Love it.

Here's a link to the search tool pictured at the left.

I used it and searched by keyword "music" and look at that, result number 13 is a poster for Seattle's very own Blood Brothers.

I contacted these folks to ask about their procedures for storage an digitization here is our email exchange:

Me:

I am working as an archives intern for a non-profit all-ages music venue here in Seattle called The Vera Project. Our collection here needs a lot of work, both physically and digitally, so I've been doing a lot of research into standards and best practices and how existing archives function. Of course as a music venue, our collection consists primarily of show posters and digital photos. So naturally, the International Poster Collection housed at your archives piqued my interest. I would love to know how you both physically store your posters and digitize them for your website. Thanks, I very much look forward to hearing from you.

----

Patty Rettig, Head Archivist at CSU:

Sounds like you have a challenge ahead of you!

Here, we store our posters flat in map cases. We keep 10 posters per mylar folder, arranged face-to-face, then back-to-back. We put tissue between the face-to-face sides, to help prevent acid migration and discoloration etc. We then keep about 10 folders per drawer.

When possible, we keep the smallest posters separate from the larger posters, so we can be more efficient with the storage space. For the smaller ones, we use half-size folders and can thus get two piles next to each other in the drawer.

When we have posters larger than the drawers, we roll them and wrap the roll in tissue for storage on top of the map cases.

For the digitization of the posters, we have a photographer do digital photography and create tiff images. We create lower resolution derivatives for web access. Though it hasn't been updated recently, this page http://lib.colostate.edu/posters/digitization.html tells more about the photography process.

Good luck with your work! Let me know if we can be of any further assistance.