Vera Audio Archives Report

At long last here it is - so feast your eyes. I'm overall very pleased with the result: my own work and its reception back in February. I managed to even keep it a little bit lively by throwing in a few jokes even though I was incredibly nervous going into it - it was my largest audience yet. In attendance were of course the usual suspects: Dustin Fujikawa - Vera's front desk manager, Nora Mukaihata - my advising librarian, Shannon Roach - Vera's Managing Director, and Josh Zimmerman - my advising Archivist. In addition to this powerhouse of absolutely talented and charming individuals, were Alexanne Brown - an observing Ethnomusicology student from the University of Washington, Rowdy Gleason - one of Vera's audio engineering interns, and Jeff McNulty - Vera's Program Coordinator. Jeff in particular has been great to work with on this project; we had many fruitful conversations throughout the research phase.

After I presented the report, the main team and I plotted out the timeline for the remainder of the internship. Since this time though a few changes have been made to the calendar and the images below reflect those changes.

...p.s. be sure to check out the appendix at the end of this post for some more tantalizing images!

I. Introduction – The Deliverables

1. Assess how much content there is, and how quickly the collection is likely to grow
2. Figure out the best storage for audio archives (naming conventions, file format, where to store them)
3. Figure out how to make these audio archives searchable
4. Document all of the above & explain it to Vera staff

II. Present Arrangement of Vera Audio Materials

File formats used: WAV, AIF, mp3, ptf (protools file), and digital audio tapes (DAT)

A. The Hard Drives:

All drives are located in the Vera Recording studio inside or on top of the PowerMac

1.) Internal 300GB HD - Live Shows & Studio Mixdowns

Main folder: Live Show Mixdowns

>Sub folder: Name of band, date, initials of who recorded it

• File name: whole show, date
• File name: individual song,

>Sub folder: Rough mixes (day of show)
>Sub Folder: Live shows mp3s
>Sub Folder: 2007 live shows

2.) External 500GB HD – Two partitions: mostly back-ups of live recordings

Main Folder: Vera Backup A-M
>Sub Folder: name of band, date, initials

Main Folder: Vera Backup N-Z
>Sub Folder: name of band, date, initials

3.) External Vera recording drive
Protools session files organized like so:

-One folder for a whole show named by headlining band, date, initials of who recorded it
This folder includes files for the opening bands as well

-Eventually all bands will have their own separate folder

B. Basic Workflow:

-The recording drives employ a system of highlighting

• Green: indicates a folder whose contents have been mixed
• Orange: indicates a folder that is ready to be backed up
• Red: indicates a folder that has been backed up

1. For live shows and recording sessions, they record to a recording drive
2. These files are then mixed
3. Then they are backed up onto a DVD (the protools files)
4. Then the protools files are copied from the recording drive to a Back-Up Drive

They like to keep the recording drives as empty as possible - nothing stays in there permanently

C. Amount of Content and projected Growth

There is about 500 GB of archived Audio. Jeff made a gratuitous prediction that we could record as much as a terabyte of audio for 2010 alone.

III. Recommendations for Individual Issues in Audio Preservation

A. Storage Hardware

RAIDS

The industry unanimously recommends storage of digital audio onto multiple hard disks in multiple locations. The most reliable storage system that exists is referred to as a RAID (Redundant Array of Independent Disks). Wikipedia defines a RAID as “an umbrella term for computer data storage schemes that can divide and replicate data among multiple hard disk drives”. Essentially, the RAID has two goals: data reliability and/or increased input/output performance. Each Hard Disk on a RAID configuration mirrors the other in case any single one should fail. Despite the fact that there are multiple hard disks, the RAID appears as a single disk from the point of view of the end user.

There are many different RAID configurations - each with their own purpose.[1] For our purposes, either RAID 5 or 6 appears to be the best configuration out there. First, they distribute their error correction data (parity data) over multiple disks. Second, they employ data striping, which distributes segments of a single file onto multiple disks allowing for greater performance. That is, while a CPU can often process information quicker than a single disk can supply it, the CPU can pull multiple segments of a file simultaneously rather than waiting for a single drive to supply each one on its own.

Recommended Storage Configuration for Vera:

1. The Studio PowerMac with three additional drive bays.

• External Drive: Recording Drive (Origin)
• Bay 1: Live Recordings A-M (Native – 1^st Backup)
• Bay 2: Live Recordings N-Z (Native – 1^st Backup)
• Bay 3: Studio Recordings (Native – 1^st Backup)

2. First RAID 5 or 6 (In-Studio)

• Identical to the PowerMac (local – 2^nd Backup)

3. Second RAID 5 or 6 NAS (Network-Attached Storage)

• Identical to the Power Mac (Remote – 3^rd Backup)
• Will be connected to the Vera Server

4. Optional Third RAID 5 or 6 (Off-Site)

• Identical to the Power Mac (Foreign – 4^th Backup)
• Possibly stored at the University of Washington

Once this system is in place, Vera should no longer back up protools files onto DVDs and create audio cds. DVDs and CDs are universally recognized as an unreliable, unsafe, and short-term preservation medium. They can be easily damaged and the logistics of storing them over time becomes extremely problematic. Additionally, the cost of DVDs/CDs over the long term is astronomical relative to hard drives. With the rising prevalence of purely digital formats and consumer devices designed to store and operate these, it is no longer advisable to use what is quickly becoming obsolete. The only exception to this I can imagine would be when bands specifically request a CD of their performance. By all means, one can be provided.

Low-Cost Setup

Vera can easily purchase low-cost PC towers with multiple bays then purchase the drives to populate them. For the NAS tower connected to the Vera server, it would require programming with a free Unix based program called FreeNAS: an embedded open-source NAS distribution program.[2] Furthermore, we could likely seek the services of a volunteer to perform this setup.

Aside from the programming aspect, PC Towers on Google Shopping come with as many as 10 bays and start as low as 30$. Individual hard drives on Google Shopping start as low as 70$ each for 1 Terabyte. RAID 5 and 6 require at least 3 or 4 drives respectively to setup. At these numbers, this would roughly equate to 720$ at the absolute lowest startup cost with the RAID 5 Configuration. RAID 6 is about 930$. I am guessing that the cost would very likely be higher. I would recommend starting with a RAID 6 on two 10-bay towers and starting out with a capacity of about 2TB. Then, whenever the collection reaches the RAID’s capacity, a new drive can be purchased and inserted into the next available slot.

B. Folder and File Naming Conventions

To meet the needs of the recording studio and Vera overall, it is absolutely imperative to adhere to a strict folder and file-naming scheme to enhance access to audio materials. If everyone uses the same language, whether they are creating audio or simply searching for it, they can all access it quickly and independently of one another. Therefore, with the guidance of Jeffery McNulty, I have laid out the following folder and file-naming conventions for the Vera Recording Studio.[3]

Studio Recording

Date, Artist Name, initials

>Song name (protools folder)

song.ptf

>Audio Files ß amount varies, may be multiples for each instrument numbering sys?

instrument.wav

instrument.wav

>Fade Files ß protools names these automatically, will never be accessed later

wav files

>Plug-In Settings ß not usually part of a project but should be

>Session File Back-Up

song (backup).ptf

>Rough mixes

1 – song name (rough mix).wav

2 – song name (rough mix).wav

>Final Mix (high-resolution)

1 – song name (high-res).wav

>Final Mix (low-resolution)

1 – song name (low-res).wav

>Album name (Masters)

1 – song name (master).wav

>Stereo Mixdowns ß for Mp3s and CDs

1 – song name (mixdown).wav

Live Recording

Date Artist Name Initials

Date Artist Name Initials.ptf

>Rough Mixes

Date Artist Name Initials.wav

>Show Template

>Session Back Up

1 – Date Artist name Initials (backup).ptf

>Stereo Mixdowns

Date Artist Name Initials.wav ß this will be the file for the whole show

1 – song name.wav ß these will be for individual songs extracted from the file above

2 – song name.wav

Mp3 collection (lives separate from a recording project)

Artist Name

>Album

1 - song.mp3 ß individual song

Artist Name Date initials ?.mp3 ß whole show

C. Format and Resolution Standards

The recording studio employs both aif and broadcast wav format almost indiscriminately. There does not appear to be any pattern to their usage. The only practical use for the aif format in the studio is to burn CDs and DVDs using a program called Toast. This program only burns these discs with aif files and it will automatically convert any other format to aif before burning a disc, which takes extra time. Now, both aif and wav are equally high quality, yet the latter is the more widely held standard and it captures basic metadata. Conversely, aif does not capture metadata and is oriented more towards Mac users who use toast. Therefore, I recommend universally applying the broadcast wav format for all files mixed from the original protools files and if possible, finding a CD burning program to burn CDs directly from wav files, if a band so happens to request one. The current practice is just far more confusing and disorganized than it needs to be when there exists a more rigorous and useable format. If such a CD burning program does not exist, I would then recommend at the very most only making the low-resolution (44.1 kHz / 16-bit) mixes into aif format. Furthermore, we simply do not need to create high-resolution aif files because RAIDs will be replacing the DVD backups.

Speaking of resolution, there is not a universally enforced standard for file resolution at Vera. The reason is that there really can’t be. While the industry-wide standard is now 96 kHz /24-bit, every audio engineer (at least for studio performances) has his or her own particular preferences. It is almost impossible to enforce a standard. Therefore I would recommend emphasizing the industry standard, but leaving it to each engineer to decide. Jeffery has indicated that for live shows it will be easier for us to apply a standard. He indicated a common usage of 88.2 kHz / 24-bit noting that the difference between 88.2 kHz and 96 kHz is negligible. The reason for recording at 88.2 kHz is for the ease of mixing down to 44.1 kHz 16-bit for mp3 and audio CDs. Therefore, I recommend that this be the recording standard for live shows.

D. Metadata

Audio engineers can track the following metadata using their tracking sheets. Later on, this information should be entered into a pdf file with its name identical to the file it describes. This will make the information contained within it searchable from both a Mac and pc. However you might be thinking it would be incredibly daunting and time consuming to enter all this metadata into pdf files over and over again. On the contrary, assuming that groups of files are created together, by the same person, at the same bit depth and sample rate, on the same day, and are the same event type, one could easily enter this information once into a pdf, save it as a temporary template to be copied over and over for each audio file in the group, only then requiring going back and entering the duration of each file.[4]

1.) Format (I realize that the recording studio won’t deal with LPs but since we have an LP in our
Archives, I’m guessing its possible that we might receive more in the future)

• Digital file

• LP Commercial microgroove disc

• CD – Compact audio disc.

• DVD

• DAT – Digital Audio Tape.

2.) File Type

(.wav) (.mp3) (.ptf) (.AIF)

3.) Bit Depth (Usually 24-bit standard)

The number of bits per sample for the audio content of the described audio object.

4.) Sample Rate (Usually 96 kHz standard)

The sample rate of the audio data for the described audio object.

5.) Checksum Value

A string indicating the checksum signature of the audio object.

7.) Checksum Creation Date

Indicates the time and date the signature in the checksum value element was
generated.

8.) Duration of File

9.) Date Recorded

10.) Recorded by

name or initials of the one who recorded it

11.) Mixed by

name or initials of the one who mixed it

12.) Artist

name of band(s) / performer(s)

13.) Event Type

Live or Studio Recording

E. Quality Control, Data Migration, and Optional Evaluation Metadata

Quality Control

It is recommended that an audio engineer periodically listen to files to check for any errors or alterations. Given the amount of audio that Vera has however, this would be an epic task. I therefore recommend that evaluations be performed during the mixdown process from this point forward. That is, if the audio engineer notices any anomalies in the recording, he or she should take note of them – perhaps on a separate evaluation sheet. Otherwise, there is a process called “Checksum,” an algorithm designed to check files for errors and alternations. This algorithm returns a series of numbers and letters. One can determine if an error has occurred when periodically running this algorithm and comparing the series of numbers. I recommend performing a checksum every time files are backed-up.

Unfortunately, I have not yet found a fully functioning and efficient method for performing checksums. The Disk Utility on a Mac is capable only of performing a checksum on an entire hard drive rather than a selection of files. This is rather problematic because say for instance there are multiple projects on the recording drive and only one of them is ready to be backed up. If you checksum the whole drive it will include the other projects. Then when you back up that single project onto another drive the checksum value for the new copy will of course be different because the new copy does not include those other projects.

There is another option for performing a checksum through the Mac’s automator program. I set this program up on my own Mac and executed it. I performed a checksum of a group of files on one drive, obtained the checksum value, then copied the files to another drive, and ran the checksum program again. Only it refused to give me the value. Further testing is required. However, if a solution cannot be found, I would recommend foregoing the Checksum task altogether.

Data Migration

There are two issues in data migration and the first is backing up data. The Vera recording studio records a studio or live performance to an external hard drive where it lives as a single project until it is mixed then backed up onto an external hard drive that lives in the studio. I however would recommend that at the end of each recording session, a project be immediately backed up – perhaps using the Mac’s Automator feature – lest the audio engineer have to perform an extra step. Otherwise, the engineer can just click and drag the project to the back-up drive. This way, if the recording drive by chance should fail, the recording still lives. Then, after the original recording has been fully mixed and it comes time to close out a project and back it up, the folders created during the mixdown process can be added amongst the original recording folders already living on the back-up drive.

The second issue in data migration is hardware obsolescence. All hard drives have both a natural lifetime and often before reaching the end of that lifetime, become obsolete. The recommended best practice is to replace hard drives every 4 to 5 years. In this case, I would recommend taking note of the purchase date for each hard drive and scheduling a time to replace it within 4 years. This could be done by placing labels on the outer shell of each RAID tower or even directly on the hard drives with a name and date for each drive.

Optional Metadata for Evaluations

I recommend that this set of metadata be used when the audio engineer discovers errors after the initial recording as he or she listens to it during the mixdown process.

1.) Evaluated By

Name of the person who completed the evaluation.

3.) Evaluation Date

Enter the date this evaluation was performed.

5.) Section Evaluated

Designate the section of the recording that you are evaluating. This will be either the entire
object or a specific Region or Stream.

6.) Problem[5]

Documents any anomalies present on the recording

7.) Notes

F. DAT Conversion (Special Addendum)

While the conversion of Vera’s DAT collection is already underway, I thought it might be useful include the following recommendations. There exists an abundance of information on the subject of converting audio materials from their physical form into electronic/digital. Sound Directions – a joint Audio preservation project between Indiana University and Harvard College – has the most in depth publication for this procedure, the more technical aspects of which are beyond the scope of my general archival purview. Therefore, I have included only the most relevant portion of their recommendations for the transfer of audio from a physical to electronic format in this report.[6]

Best Practice 1: Use audio engineers and technicians with solid technical skills and well-developed critical listening abilities at points in the preservation transfer workflow where their skill is required.

Best Practice 2: Perform preservation transfers in an appropriately designed, critical listening environment. [I.e. one without ambient interference] If such a space is not available, choose a room that is quiet and is removed from other work areas and traffic, and be acutely aware of its sonic deficiencies.

Best Practice 3: Route the signal from the playback machine to the analog-to-digital converter using the cleanest, most direct signal path possible. [I.e. ones designated specifically for preservation work.]

Best Practice 4: Design the monitoring chain to allow instant comparison of the signal from the playback machine to the signal that has passed through the analog-to-digital converter. The ability to monitor the signal from both the playback machine and post-A/D converter enables verification of the A/D conversion and allows easier diagnosis of potential problems heard during transfer

Best Practice 5: Preservation studios must include test/calibration equipment to test and monitor the transfer chain itself for noise as well as to test individual components for performance. During transfer, the test/calibration equipment shall not be inserted between the playback machine and the recorder.

Designing an audio preservation studio:

• Design the preservation studio as a critical listening environment and know its limitations
• All signal chain components must be tested so that they are known to be of professional-quality, that they are reliable, and that they do not alter the level or quality of the audio signal at unity
• The most direct and clean signal path from source to destination must be used at all times.
• There may be no unused devices in the signal path. If there are multiple destination formats for the transfer, then the signal must be routed in parallel without any daisy-chaining of devices.
• Signals shall be split or distributed using only calibrated, high-quality distribution amplifiers, routers, or properly designed and wired balanced cables and patchbays that demonstrably do not degrade the signal
• Use the highest quality signal format present on the source equipment and throughout the chain. For instance, use a balanced signal source rather than an unbalanced signal source

Cost: It may be necessary to engage the audio engineering community, as there does appear to exist an informal, short list of converters that engineers believe are of high-enough quality for preservation transfer work. These tend to range in price from around $1,000 to $10,000 and more.

IV. Access

With our end-users consisting of both staff and patrons, ease of access to audio materials is essential. With this in mind, I’ve been working diligently to determine the best way to improve access to Vera’s audio materials and there are two options: Adobe pdf and Salesforce.

A. Adobe PDF

The Adobe pdf file format is the simplest solution. The contents of a pdf can be searched both on the Windows Explorer search bar and the Mac Spotlight Search. Therefore, if Vera desires a cost-free and immediate option, I would strongly recommend creating one pdf file for each audio file and giving them identical names. Each pdf will then describe the audio file – whose name it shares – in the form of metadata. This means if anyone wants to search for all the files that were mixed or recorded by one individual or even all files recorded on a certain date, he or she may do so easily. Once gain, groups of audio files will inevitably have certain metadata in common, so filling out the fields should be a relatively quick process.

B. Salesforce

Salesforce is the second option, which would be very effective in tracking and describing audio at the project level instead of the individual file level (lest we clog their server). I can easily imagine this accommodating anyone outside the audio team who has no need or interest in searching hundreds of audio tracks of single instruments or different mixes.

For the past month, I have been tinkering with Salesforce and have found it relatively easy to customize. It resides on the Internet and can be accessed anywhere simply through an email and password login and a one time computer activation. In the demo version, there is a row of tabs across the top of the screen. The user can easily create a tab by clicking on the setup button just above these. In the setup page there is a side bar with several options. First, you create an object, which will serve as a template when you go to create a new tab. You give the object a name then proceed to the next page where you can add in or remove as many fields as you wish. Once complete, you then create a new tab and use the object you just created as its template. Any user can then go into any of the custom tabs, say audio for instance, and create as many entries as he or she wishes and run a search by keyword. So, if we create entries with fields that tell the end-user where an audio project lives on the server, he or she can then click the link to it and browse through the files at his or her leisure and if need be, read the individual metadata pdf files accompanying them.

V. Summary via Workflow

So you may be wondering how all of this fits into the big picture, below are a list of simple steps to give you a rough idea of how this could look.

A. Hardware setup

1. Purchase at least two 10-bay PC towers and eleven 1TB drives

2. Configure each tower to RAID 6 (each with four drives)

3. Install the remaining hard drives into the Mac.

4. Effective capacity should be 2TB and each RAID will contain the entire Vera audio collection.

5. Give a name or number to each drive, note its date of purchase – perhaps with labelson the outer shell of the RAID tower – and schedule their replacement 4 years ahead using the Mac’s built in calendar, which can automate the reminder.

6. When the collection reaches capacity, purchase another 1TB drive and insert into the next available bay.

7. Record its date of purchase and schedule replacement.

8. Repeat these steps until the RAID reaches physical capacity.

By the time the tower reaches capacity, it’s likely that individual internal hard drives will have higher capacity at which point, Vera can purchase 2 or 3TB drives to replace the 1TB drives. Otherwise, a whole new tower can be created following the same principles laid out above.

B. Protools Recording Session

1. Open Protools and starts a new session

2. Read the folder and file naming template provided[7] and start by creating a folder for each band recorded e.g. “2010-02-24 Tommy Salami JMM”

3. Obtain a metadata tracking sheet and fill in each field for each file you create.

4. At the end of a show or studio session, burn a cd for the band(s) if they request one.

5. Run a checksum on the session as a whole and record the value onto the tracking sheet.

6. Back-up the files onto the PowerMac drives and the in-studio RAID tower by simply

clicking and dragging the folders you created during session.

7. Run the checksum again on each back-up and compare the values.

8. If the values do not match, the files did not back-up correctly and need to be backed up

again.

9. Place the audio metadata tracking sheets into a place to be designated for an intern to

enter into pdf files later.

C. Protools Mixing Session

1. Obtain a metadata tracking sheet and fill in each field for each file you create.

2. Reopen the original recording session in protools and continue to follow the naming

template provided.

3. Record all of your high and low resolution mixes in broadcast wav format.

4. Record all high-resolution wav files for live recordings to 88.2 kHz 24-bit

5. Record all low-resolution wav files to 44.1 kHz 16-bit

6. If any errors are detected obtain an evaluation sheet (?) and take note of them. Attach

the evaluation to your regular metadata tracking sheet.

7. Create mp3s for completed songs then name and file them according to the naming template specifications.

8. Run a checksum on the mixes and record the value onto the metadata tracking sheet.

9. Back-up wav mixes and mp3s to the Mac tower and in-studio RAID Tower

10. Run the checksum again and compare the values, if the values do not match, the files must be backed up again.

11. You have officially completed an audio project.

D. Intern Data Entry Session

1. Obtain the metadata tracking sheet placed in the area designated as “completed sessions”

2. Open the Adobe PDF file titled “audio metadata template”

3. Enter all the metadata fields for groups of files that share certain metadata in common and save it as your temporary template.

4. For each audio file in a project, use one of these templates, fill in the remaining metadata from the tracking sheet, and resave the file with the same name as the file it describes.

5. Repeat these steps until you have recorded all metadata for each audio file.

6. Copy these PDF files to the Mac and in-studio RAID tower.

7. Perform a final back-up of the whole project to the NAS RAID tower.

8. Open the salesforce database and click on the tab titled “audio” then click on “New”

9. Fill in all the fields for the project entry and click “Save”

10. Congratulations an audio project has just been cataloged!

A brief teaser…

With my on-going conversations with Darren White about many of the technical aspects of digital audio, he informed me of a new storage method on the horizon: the Solid State Drive.

Here’s what he had to say:

As for SSDs, it's basically like a giant RAM chip, but non-volatile, so you don't lose data when you power down. No moving parts can mean a very long mean-time-between-failure as compared to a mechanical drive, like a hard disk. There is still a need for prudence hence redundancy and right now SSDs are prohibitively expensive for TB+ storage needs. But eventually, some type of solid state storage (or maybe bio- or quantum-storage) will overtake hard drives as we currently know them. I know things will look at lot different say, 5-7 years from now.

In light of this revelation, I would highly recommend keeping track of this trend as the first purchase of back up storage elements reaches its date of obsolescence.

---

[1] http://en.wikipedia.org/wiki/RAID

[2] Suggested by Darren White http://freenas.org/freenas

[3] See the appendix for more detailed examples of these conventions could look.

[4] There exists another more thorough and complicated option for metadata capture still under development by Sound Directions called the Audio Technical Metadata Collector. Visit the link provided below for more information.

[5] For standardized vocabulary be sure to visit the link provided below for the Sound Directions Audio Technical Metadata Collector and flip to page 21.

[6] For more in depth information about conversion, be sure to visit: http://www.dlib.indiana.edu/projects/sounddirections/papersPresent/sd_bp_07.pdf

and proceed to page 22, titled: “Personnel and Equipment for Preservation Transfer.” There are also specific recommendations for setting up equipment for DAT conversion on page 31.

[7] This can be placed in an area that is easily seen and readily accessible – a tack board for instance.

VI. Bibliography

A. Issues in Preservation

1. Mike Casey and Bruce Gordon, Sound Directions: Best Practices for Audio Preservation, Indiana University and Harvard University: 2007.

http://www.dlib.indiana.edu/projects/sounddirections/papersPresent/sd_bp_07.pdf

2. Mike Casey and John Ross, Technical Committee: Preservation of Archival Sound Recordings, Association for Recorded Sound Collections (ARSC): 2009.

http://www.arsc-audio.org/pdf/ARSCTC_preservation.pdf

3. Gareth Knight & John McHugh, Preservation Handbook: Digital Audio, Arts and Humanities Data Service: 2005.

http://ahds.ac.uk/preservation/audio-preservation-handbook.pdf

4. Various Authors, Digital Audio Best Practices Version 2.1, Collaborative Digitization Program: 2006.

http://www.bcr.org/dps/cdp/best/digital-audio-bp.pdf

5. Gary Louie, University of Washington Music School: louie@uw.edu

6. Darren White: deedubyah@mac.com

7. Sound Directions: The Audio Technical Metadata Collector http://www.dlib.indiana.edu/projects/sounddirections/papersPresent/sd_app1_v1.pdf

B. Information on RAIDS

1. General Definitions: http://www.wditech.com/wditech/?p=13

2. General Definitions: http://en.wikipedia.org/wiki/RAID

3. How to setup a low-cost RAID http://freenas.org/freenas

4. Where to buy low-cost towers: http://www.google.com/products?q=PC+Tower&oe=utf8&ved=0CDUQrQQwAg&show=dd&scoring=prd

5. Where to buy low-cost hard drives:

http://www.google.com/products?q=internal+hard+drive+1TB&scoring=p

C. Solid State Drives: A look to the future…

1. http://en.wikipedia.org/wiki/Solid-state_drive

2. http://www.google.com/products?q=solid%20state%20disk%20drive&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&um=1&ie=UTF-8&sa=N&hl=en&tab=wf

APPENDIX: The Potential Structure for Audio Projects

What's New at ZAPP: Storage and A/V Organization

Hey folks. First allow me to mention, depending where you view this, the text and image may look disastrously scattered. I apologize for this. I am just not that good at universalizing the html code. So I haven't been cataloging zines for a little while now - sad, I know - because I've taken on some new tasks. On Wednesday January 20, I began reorganizing the ZAPP storage closet. Oh don't you just wish you had my job? Pictured at the left is the first stage of this project.

The brown cabinet on the left was a disaster waiting to happen. It was bowing out at the center and thus the shelves resting on pegs at that point were extremely hazardous to the safety of its users and the condition of its contents. So I had to lower the shelf and prop it with various materials including a 2 x 4. Sketchy. I knew this tall obelisk of doom had to be removed.

Oh how I wish I could show you the rest of this tragedy. Oh and it gets worse! Accustomed as I am to employ "questionable" methods for the removal of to-be-trashed materials, I tore this thing apart. After all, this would make it easier to remove. Low and behold, the inner base of the cabinet was festering with MOLD! EWW. I couldn't believe my eyes. Phew, crisis averted.

What you see on the left here is the open space that remained in its wake.

Now here I am, carrying the new shelf from the Hugo House Member's Library into ZAPP. Once again, my stubborn will as usual far exceeds my physical capacities for sure enough, my back was a bit sore afterward. Also, as I'm sure you can notice, this shelf is not the only thing that rests over my shoulder. Look a bit more to the right and you'll see a bust of Mr. Richard Hugo himself; watching ominously as I single handedly dismantle his coveted Member's Library. He must be turning in his grave over this sad twist of events. Dearest Richie, I promise this was not my decision, I will be sure to pour the proper amount of libations in your honor.

Once I got the new shelf unit in, I had to make some more adjustments. Yes, this archivist is not afraid to get a little dirty. So I thought it best to remove the toilet seat and to cover the bowl with that faux marble panel behind me. This way, the items stored above it will not wobble over the seat and the remaining water in the bowl will not interact with the storage items as easily. Hopefully we can get both the toilet and sink removed altogether. We'll see. Thanks Brian, for the lovely shot. =)

Ah! The finished product (well, stage two anyway). Isn't it beautiful? I've been telling everyone that this space will soon be a pleasure dome - just you wait, I'll make it happen.

Below you see this cute little frog I found and I just had to give him a place to live. You see, he really likes his mail art, which is what lives inside this trunk that I decided to put on top of the new shelf for ease of access.

Aside from this project. I have a bit more of a serious project going on - more for the Hugo House in general as opposed to ZAPP specifically. For some time, Hugo House has been recording audio and video of the various events it holds - most often its Literary Series - onto mini DV tapes. I believe there are some other events that are recorded onto other mediums such as CDs and DVDs. I am in midst of developing a preservation plan for the materials based on my research for Vera's audio materials. The Goals are as follows:

Determine the size of the collection and how much it is expected to grow.

Determine standards and best practices for preserving the existing collection and expanding it.

Determine standards and best practices for its wider distribution.

When I make some major breakthroughs on this project I'll be sure to notify you all. Until then, thanks for reading!

Vera Photo Archives Tour - A Fun Weekend Project

Dear Readers, (the few, the brave, the proud)

I know its been quite a while since I've posted here and you all must be wondering what is happening as of late, well... that will have to wait. Ouch - total buzz kill, but hey it rhymes. Ok ok I promise when the work eases after the completion of my Audio Report and its presentation on February 24, (ooo a teaser!) I will at least try to put together some back entries based on my daily notes heretofore. Until then, I thought I'd share with you all (visually) a little brainstorm session I had about how best to organize Vera's show photos. Every time I come up with a method of organization and ruminate on it for a while, I start to find problems with. I think I've created and destroyed some 2 or 3 versions at this point. Well, here's my latest and I think its the best one so far. So I thought it would be fun to give you all a step-by-step tour and my justifications for it as we go along.

First, I use a Mac - as should everyone - so I could not exactly replicate how it will look on Vera's computers which are PC (gasp!), but this should give you the idea. So we start here in the Archives folder and you see the various archives elements: The Four Ps. For our purposes, we'll be looking at the Vera Photos element.

Here you can see that all photos are organized by year: simple enough. We'll be looking at some 2010 photos.

Inside you can see that photos are now broken down by month. This is just for the sake of example, its likely that various years will not have photos for a whole month - tragic to think. In that case, there will be no point in creating a folder for a particular month only to have it unfilled. We're going take a look at January.

So here we are in January: bitter, cold, gray, and white - just as you would expect. I decided that it would make the best sense to divide up the photos by band name with the date they played and an indicator for who was the headliner of the show. Otherwise, naming a folder like "2010-01-07 Show" is just too general and "2010-01-07 Whatever headlining band" will often be misleading. The latter because very often photographers do not shoot the headliner (as was the case with me). We don't want to name a folder of a whole show by its headlining band only to have end-users going in there and not finding what they were looking for.

Now, I really like Mac's folder highlight feature: its more visually pleasing and facilitates easier browsing. Assuming Vera's PCs offer the highlight feature, we can easily employ it to group together related folders (even though you can tell which one's are related by looking at the dates, this feature just makes it easier!) for each show then, all we need to do is alternate between gray and white. Seems simple enough. Next, we're going to take a look at the Baby Panda folder because I just so happened to be the photographer that evening. I know I know my extremely creative example "Band A" seems so much more exciting, but thats not what I invited you here today to learn about.

Here we are in the Baby Panda folder. You can see here that I've created separate folder one for me =) and one for a hypothetical second photographer. Now, one might object (as I originally did) that this would be extremely daunting for the Admin/Archives Intern to have to split up a single photographer's photo donation into different folders - forcing them to struggle to identify which band is which, if for instance, a single photographer shot (murderer!) the whole show. On the contrary, assuming we get all photographers to sign a release form, this form will provide a space for the photographer to specify which band(s) he shot. So, all the Admin/Archives Intern will have to do is refer to this release form. We're going to take a look at some of my photos because I think I'm so much better than Photographer X.

Ah yes, finally we're down the file level - the unctuous essence of it all. Here is how I'd like files to be named. Its simply common sense to at least have what the picture is of and to order them chronologically as they were shot. Even though very often Camera's will include a time stamp, I decided that its necessary to include the date in the file name. As you can see, my camera was mistakenly offset to 2009 rather than the correct year 2010. We simply cannot trust everyone's camera to always be on the correct time. Next, I'll give you the total perspective of the folder level.

Is your brain melting yet? Just for fun and to look at something even more boring than the infinitely exciting folder structure of the photo archives, I'll share with you a couple of favorite shots from this show.

Here we have Paul Gonzenbach former frontman of Oakland, CA's Jim Yoshii Pile Up performing for his new solo project Baby Panda. And next (to borrow the words of Jon Stewart) we move on to our "Moment of Zen"....

My dear friend Josiah Feinberg and cousin of Paul, playing drums in midst of his Enthousiasmos with the Muses of the Drums! A classic shot I must say so myself.

Vera Research Progress Part 4 - Audio

For those of us most concerned with Standards and Best Practices for Audio Preservation, here is the sum-total of my research organized and collated for your viewing pleasure. For your information, anytime I interject in the course of this post, I will use italics.

I. Introduction - The Deliverables

According to Shannon Roach these will be:

1. Assess how much content there is, and how quickly the collection is likely to grow
2. Figure out the best storage for audio archives (naming conventions, file format, where to store them)
3. Figure out how to make these audio archives searchable
4. Document all of the above & explain it to Vera staff

II. Present Arrangement of Vera Audio Materials - A Conversation with Jeff McNulty

Since for whatever reason I decided to take this rather charming picture before Jeff gave me the audio tour, I thought I'd include it here just for fun. For those of you who do not know, Jeff is on the left and Nelson, one of the Vera audio staff, is on the right.

File formats used: WAV, AIF, mp3, ptf (protools file), and digital audio tapes (DAT)

A. The Hard Drives:

1.) 300GB HD - Live Shows & Mixdowns

Main folder: Live Show Mixdowns

>Sub folder: Name of band, date, initials of who recorded it

• File name: whole show, date
• File name: individual song,

>Sub folder: Rough mixes (day of show)
>Sub Folder: Live shows mp3s
>Sub Folder: 2007 live shows

2.) 500GB HD – Two partitions: mostly show back-ups

Main Folder: Vera Backup A-M
>Sub Folder: name of band, date, initials

Main Folder: Vera Backup N-Z
>Sub Folder: name of band, date, initials

3.) Vera 1, 2, 3 recording drives
Protools session files organized like so:

-One folder for a whole show named by headlining band, date, initials of who recorded it
This folder includes files for the opening bands as well

-Eventually all bands will have their own separate folder

-The recording drives employ a system of highlighting

• Green: indicates a folder whose contents have been mixed
• Orange: indicates a folder that is ready to be backed up
• Red: indicates a folder that has been backed up

B. Basic Workflow:

1. For live shows and recording sessions, they record to a recording drive
2. These files are then mixed
3. Then they are backed up onto a DVD (the protools files)
4. Then the protools files are transfered from the recording drive to a Back-Up Drive

They like to keep the recording drives as empty as possible - nothing stays in there permanently

C. What Jeff wants to see in the future:

- Record all DATs to their backup hard drives

- 3-4 backups

1. DVD backup for all protools files,
2. One or two terabyte hard drives connected to the server
3. I think he may have said he also wanted drives that are not connected to the server
4. Hard Drive + CDs for audio backup this would be for mp3s, podcasting etc...

- All dvd and cd backups organized and cataloged
- Copies of records and CDs for bands who recorded at Vera

D. Current Issues:

1. Toast: the standard for CD and DVD burning in the Audio engineering community

• Jeff is concerned with future compatibility because there is a new version of Toast and it has recently changed the way one can access a DVD or CD Older CDs and DVDs burned using the old Version of toast cannot be viewed with the new version of Toast.
• I also think he said the new version of toast requires that you have toast to view the files I could be wrong.
• Jeff wants something for DVD burning that will be more future compatible Jeff also mentioned an issue with toast wanting to automatically convert wav files to aif and how there is a cross platform (PC vs. Mac) issue with this.

2. Issue of cross platform compatibility for tagging and metadata

• The audio folks use a mac. Mac has a different layout when you right click a file and check its properties. On a mac there is this thing called "Spotlight comments" where you can type in anything you wish about a file and it is then easily searchable.
• My personal experience with this is that it is not cross compatible. I have an external hard drive that i use to store all my internship files and personal files. I own a mac and use the spotlight comments especially for my audio recordings. When connecting to a PC and right clicking the file and choosing "properties," the comments do not show up. Not only that - and if i remember correctly - when I returned to my Mac and viewed the file, the spotlight comments had been erased. This is a major problem if we go the windows explorer tagging and metadata route.

III. Individual Issues in Audio Preservation

A. Storage

1. File naming conventions:

According to the Colorado Digital Program (CDP):

"Systematic file naming is important for system compatibility, interoperability, and to demonstrate ownership of the digital asset. File naming conventions specific to an institution may be used. These might include protocols that include institutional acronyms, collection identifiers, or part designators, among others. In general it is recommended that the characters in the file names be alpha-numeric, lowercase, and not utilize spaces, tabs, commas, periods or any other characters reserved for computer system use, such as slashes, asterisks, or question marks."

Seems pretty obvious enough to me.

2. Hardware:

Hard Drives and particularly RAIDs (Redundant Array of Independent Disks) seem to be the general consensus for Audio storage.

The CDP defines a RAID as:

"A self-contained collection of disk drives that act as a single large hard drive storage system. These configurations are designed to enable a system to operate when an individual drive within the array fails, thus minimizing and ideally eliminating the potential for the loss of data. There are ten types of RAID configurations. In general, the better choices are RAID 5, 6 or 10."

Gary Louie, University of WA Music School archivist has the following to say about RAIDs:

• "Hard drives are the only practical method now to store the size and amount of files we have. Hard drives are also poor for reliability, so any HD storage has to be RAID of some sort, and must be duplicated in at least one other physical location."
• "Some things we are looking at for local, small scale archive "starters" are small RAID hard drive towers with removable sleds, duplicated onto similar arrays via gigabit ethernet in another building."
• "You can certainly consider a really small scale HD storage method that will still be helpful to you on all counts at low cost. An external HD holder (like the Newertech Voyager) and a handful of bare HDs, with constant duplicates and stored at different locations. Could be in conjunction with a small server system."

ARSC Technical Committee on RAIDs:

"RAID arrays remove some of the disadvantages of individual hard drives while they provide larger capacities than individual drives. They are extremely easy to use, especially when supplied as network-attached storage, but still should be considered mechanical devices that may fail at some point in time. A single RAID array is not adequate for long-term storage."

For a more in-depth discussion of various RAID configurations go here.

Beyond the consensus on RAIDs as the best way to store digital audio, there is also a consensus that audio be stored in multiple locations and not on a single array.

The ARSC Technical Committee suggests:

"Generate local backup copies of all files as soon as possible after creation
Store backups on a separate device or separate media from the original.
Make at least two, or preferably three sets of archival master digital files of preserved content and store them in different locations, and possibly on different types of media."

The CDP suggests the same:

"It is highly recommended that organizations utilize a strategy employing multiple redundant copies of digital files in separate locations, as a fail-safe strategy for the failure or destruction of the digital media. Acceptable media may be optical disks, hard drives, or magnetic data tapes – all of which have particular strengths and weaknesses."

Depending on the needs and resources of an organization offsite storage is a potential option. This is where I believe our connection to UW could come in very handy.

The CDP says:

"Projects considering outsourcing should consider the costs of ingestion and ongoing maintenance fees; metadata requirements for outsourced files; workflow requirements; and accessibility requirements."

Vera's consulting "Technical Generalist" Darren White says:

"In terms of hard disk storage, VERA should be looking at a NAS/SAN on the VERA internal network, preferably with offsite backup. That way, if disaster strikes..."

Other options such as DVDs and CDs are not recommended.

The ARSC Technical Committee has the following to say about CDs and DVDs:

• "Optical discs [are an] unsafe preservation storage media. Using this format requires costly test equipment to check the quality of both blank media and digital recordings. Even if this equipment is already available to an archive, the cost of storage media may be higher than the other formats, except in very small volume – individual CDs or DVDs are not expensive, but the cost-per-megabyte is substantially greater than other media types."
• "The format cannot hold files at higher bit depth and sample rate than 16 bit, 44.1 kHz, which is far less than the de facto 24 bit, 96 kHz standard. In addition, migrating an archive stored on optical discs is expensive, because it requires much human intervention."

What the CDP says about CDs:

• "Storage and retrieval costs will escalate as CD collections become larger and more challenging to manage."
• "Limited physical life span (3 to 20 years) and the files stored on them are vulnerable due to physical deterioration, mishandling, improper storage and obsolescence."
• "Obsolete equipment for reading the media poses a threat in the long term."
• "Adhesive labels and permanent ink markers can cause early failure of CDs through chemical interaction with the CD’s recording layer."

Gary Louie, University of WA Music School archivist, says:

"Don't trust DVDs. No one thinks they will last very long. Same for CDs and tapes. Generally, the data written to them is also not very clean, especially audio CDs."

3. Data Migration and Hardware Updating

Current hardware and software seem to go obsolete every 3 to 4 years. It is therefore extremely important to implement a Migration schedule.

The CDP Suggests:

• "Migrate your files frequently - at the very least every five years. Failure to incur these small incremental costs may lead to very large costs down the road when you need to migrate froman obsolete data format or medium."
• "Every sustainable digitization project should include the costs of data migration as a yearly budget line item."
• "At the time of migration, file integrity checks should be conducted... a “checksum” may be associated with a file when it is produced so that the checksum value may then be used to confirm the integrity of the file at the time of migration."

The subject of "checksum" makes for a great point of transition into the subject of quality control.

4. Quality Control

The CDP Recommends the following for quality control:

• All digitized audio files should be sampled for sound quality.
• Technicians charged with quality control should listen for consistency in the audio quality at a number of points in the recording, listening for distortions in the sound, for proper playback speeds, and for artifacts such as hiss and hum.
• Technicians should specifically check that the volume levels are set correctly.
• Recordings should also be checked for completeness.
• Any distortions in the sound or other inconsistencies in the recordings should be noted in the metadata.
• Metadata should also be checked for accuracy and completeness, with special attention paid to accuracy of file names, which can lead to the effective loss of files when recorded inaccurately.
• In addition, file integrity checks may be conducted. A “checksum” may be associated with a file when it is produced so that the checksum value may then be used to confirm the integrity of the file at each step from creation to migration.

"A checksum is an algorithm-based method of determining the integrity and authenticity of a digital data object, such as a digital audio file, used to check whether errors or alterations have occurred during the transmission or storage of a data object. Please note, however, that a checksum will only tell you there is an error. It cannot tell you what the error is, nor can it correct the error."

"It is important to understand that quality control of audio can be a very subjective process. It is recommended that technicians performing this process be selected on the basis of their familiarity with a variety of types of recorded sound, and preferably have experience in a field that has already helped train their ears. Staff with previous audio experience such as audio engineers, are a natural selection, as are staff with formal musical training."

5. File Format, Sample Rate, and Bit Depth Standards

While in general both WAV and AIF formats are widely used there appears to be the emergence of a new format the BWF which is exactly the same as the WAV format, except that it captures metadata.

The CDP says the following regarding WAV and AIF:

"The WAV file type was developed by Microsoft, is in widespread use, and is readable by virtually all audio software programs. The AIF file type was developed by Apple Computer and is also in widespread use. Both of these file types are uncompressed and acceptable for long-term file storage. The WAV file type has become a standard and is recommended. In addition, the WAV file type is also available in a professional flavor, broadcast WAV (BWF), which has the capability to store metadata in the file header. Although not all audio software programs are currently capable of reading or writing to the metadata header, the BWF format is emerging as the WAV file type of preference for archival audio projects."

For the sake of avoiding redundancy, I will simply say that all other sources considered thus far, are generally on the same page regarding WAV and AIF. However, since Vera also uses protools files (ptf) there are other issues to consider.

Regarding Protools files, Gary Louie said the following:

"If you are thinking long-term (10-50-100 years) who knows if anyone will be able to read ProTools files at all. The international recommendation for audio storage is the Broadcast WAV file, which holds the audio plus metadata for catalog and technical info. But that doesn't help much with PT sessions. We (i.e. University of Washington) don't do any PT work, so you should probably ask elsewhere (maybe the ProSoundWeb or Digidesign's forums) on how people archive their PT material."

I have yet to research archival standards for protools files, if there are any. When I find the answer I will update this post. However, all things considered, the new BWF seems to be the optimal choice for Vera's needs.

Below is a diagram from the CDP regarding Sample Rate and Bit Depth Standards:

6. Metadata Issues and Standards

As established above in my conversation with Jeffery, retention of metadata when working across platforms (Mac to PC) is a major issue. Depending on how thorough BWF is and if it is cross-compatible, there are other options to consider.

Darren White suggested the following:

For maximum portability of metadata, I would recommend that it be stored externally to the file itself. Even with, say, MP3 files and ID3 tags, cross- platform compatibility issues remain, even after all these years. So WiMP and iTunes might make entirely different sense/nonsense out of the same ID3 tags. I also use a Mac primarily and I think if we are talking Protools and AIFF files, Mac capabilities may be sufficient for basic metadata but to allow for expanding requirements, something like a CMS and database or even just an external XML file may make more sense in the long run.

The Arts and Humanities Data Service Perservation Handbook for Digital Audio suggests:

"Ensure that associated metadata integrated into the file itself can be extracted in the software tool and stored in the preservation format. If not, the information should be manually output to an ASCII text file."

ASCII or XML might be the way to bridge the gap between Mac and PC. If we cannot attach metadata and tags directly to the audio files we can create a text file with all the pertinent information about each file. Someone can either create one text file to match each audio file with an identical file name or just one text file for a group of audio files within one folder. I lean a bit more towards individual files.

Sound Directions, a joint project between the University of Indiana and Harvard College has an excellent resource on metadata standards, which is much to large to duplicate on this blog, so I have made it available to you here.

The complete publication is available here.

VI. Online Sources

1.) ARSC Technical Committee on Preservation of Archival Sound Recordings

http://www.arsc-audio.org/pdf/ARSCTC_preservation.pdf

2.) Arts and Humanities Data Service Perservation Handbook for Digital

Audio http://www.ahds.ac.uk/preservation/audio-preservation-handbook.pdf

3.) Colorado Digital Program Audio Best Practices

http://www.bcr.org/dps/cdp/best/digital-audio-bp.pdf

Briefly Back On The Air - An Archival "Res Gestae"

Hello everyone. For those of you not directly involved with this project, you may be wondering why I haven't updated this blog for a while. The answer is that I've become extremely busy with the actual work and as a result talking about the work has taken a back seat. At the moment I am on vacation, but i thought since I am mostly taking it easy today to recover from jet lag, I would give you all a few updates. A lot of has changed since the day before thanksgiving. Here is the Res Gestae, Latin for "Things having been done"

*I Submitted a progress report detailing the current state of both the physical and digital archives and my recommendations for moving forward.

*I Took on new responsibilities and shifted my priorities.

*First, we have 500 dollars on the Vera budget for storage supplies and Josh generously offered 200 to get things rolling.

*I Completed a rough inventory, measuring poster sizes and counting how many we have.

*Josh and I hashed out an estimate for supplies after finally deciding to go the folder route instead of the binder route.

*The Audio side of the Archives has now become a top priority so, I have been doing research into best practices.

*I will soon be coordinating volunteers to assist me in taking a more detailed inventory of the collection and to help implement the initial reorganization of the physical and digital archives.

*The work as of late has been very Vera heavy and while I am still cataloging zines for ZAPP, a conversation will soon have to take place to make sure Nora gets what she wants for ZAPP out of this internship as well. I know that this will eventually include activities such as coordinating Zine cataloging groups.

*My general duties from this point forward (at least on the Vera side) are the follow:
1.) Determine and layout standards and best practices for organizing and preserving all historical materials at Vera.
2.) Create clear and concise instructions and initiate implementation to set an example for future interns and current staff to implement these standards on their own.

For those of you who are a part of the project, be sure to inform if I am missing anything pertinent here. I do not have notes with me and this is a very basic round-up from memory. I just wanted to give a quick update.