Fighting entropy and ISIL, one image at a time

The efforts of the United States to contain and destroy the terrorist group known as the Islamic State or ISIL is generating so much data that traditional disk media is being pushed to its limits, requiring new technologies to safely store all that information. Interestingly enough, one new technology being deployed in this fight is as old as time itself – carving information into stone.

Hitachi Data Systems has invented a new type of technology to preserve information on disks inside an infinitely expandable array. This new Hitachi Digital Preservation Platform uses M-DISCs that resist environmental conditions and can theoretically last for more than 1,000 years by permanently holding data inside a rocklike substrate. It’s currently being used in the fight against ISIL and could soon start to branch out into other areas of government that require permanent archiving of important data for all time.

We talked with Kimbry McClure, Hitachi Data Systems Federal Digital Preservation program manager, about the government’s new data preservation platform and the new media evolving, from traditional magnetic disks to Blu-rays to M-DISCs made of rock that can still be read 1,000 years in the future.

Editor’s Note: The following interview has been edited for clarity and length.

John Breeden II: What type of technology does the new Hitachi Digital Preservation Platform use for mass storage?

Hitachi Data Systems Federal Digital Preservation Program Manager Kimbry McClure

Kimbry McClure: Hitachi Digital Preservation Platform (HDPP) is Hitachi Data Systems Federal’s next generation long-term data preservation solution that leverages enterprise-grade optical drives and multiple types of standardized archival quality optical media. HDPP allows organizations to preserve and archive mission-critical data for decades using optical storage on either Blu-ray disc or M-DISC, proven enterprise mediums that have been used for over 30 years. HDPP currently supports Blu-ray technology, which enables data storage for up to 50 years, and will soon use M-DISC, which can store data for upwards of 100 years.

The longevity and durability of this medium meets the demands of the 21st century government by enabling civilian, Defense Department and intelligence agencies to securely archive mission-critical data. Optical storage allows the data to be accessed directly via random-access methods, intelligently searched and scale to meet the growing storage needs of an agency’s mission.

JBII: How does the Hitachi Digital Preservation Platform preserve federal data indefinitely, or at least for the next 1,000 years?

KM: All federal agencies are mandated to archive data anywhere from five to over 100 years, while others, such as the Executive Office of the President, are required to store data forever. To meet such a broad and changing storage mandate, agencies are evaluating the optical technology that HDPP is built upon.

Customers have two options when choosing an optical medium. For long-term storage capacity for up to 50 years, customers can employ Blu-ray XL discs. For agencies that need to store data indefinitely, the M-DISC optical medium’s data layer is composed of rocklike materials that can last for centuries. The data is written into the rocklike data layer, providing a permanent physical data record that is immune to data degradation caused by light, heat, humidity and temperature extremes. This new preservation tier represents the highest level of storage that can augment traditional archiving solutions.

JBII: Why are optical media used in this new system over traditional spinning disks or even flash media?

KM: Critical agency data is important and irreplaceable – not just to the federal government, but also to the public it serves. However, the total cost of ownership is also a major government and industry mandate. With the explosion of data volumes and the high costs of data centers and operations, optical medium represents both the highest reliability as well as the lowest overall cost of ownership representing superior savings in power, footprint and data reliability.

Agencies can supplement magnetic storage with optical media to create a preservation tier that enables agencies to migrate when they want, not when the technology forces them. Flash media for archiving purposes is cost-prohibitive and has only proven reliable up to 10 years under ideal conditions.

Magnetic tape is also susceptible to environmental conditions such as heat, moisture, dirt and electromagnetic events, a critical issue for both DOD and civilian agencies alike. For example, during Hurricane Katrina and Superstorm Sandy, many government agencies and businesses alike lost magnetically stored data in the widespread flooding, whereas nearly all optically stored data survived.

HDPP’s optical solutions have proven survivability and durability. In addition, optical storage requires only a small amount of energy to maintain, with each rack containing upwards of a petabyte of data, while only consuming approximately 1 kilowatt of power, resulting in increased energy efficiency.

JBII: That rocklike media of the M-DISC technology sounds really cool, kind of like an ancient stone writing. When will we start to see the switch from the existing Blu-ray technology into the M-DISC?

KM: In early 2015 our customers will be able to store data on the M-DISC medium. The biggest difference between Blu-ray and M-DISC is the composition of the M-DISC, which is an inorganic rocklike material that the data is written into. While Blu-ray is proven to last anywhere from 50-100 years, M-DISC can last upwards of 1,000 years and under far wider environmental ranges. Compared to the four- to six-year life span of tape, optical is a far superior technology. Data can be written dual-sided on both Blu-ray and M-DISC maximizing the amount of data per library, allowing agencies to shrink their footprint for archiving critical data.

JBII: Are there any limits to the amount of data that can be stored in the HDPP system?

KM: At this point, there aren’t any limitations on the amount of data that can be stored. As agencies scale the system, additional buffer servers will need to be added to mitigate any latency issues. It’s important to point out that adding a buffer server means we don’t actually have to touch the data. We never have to touch the optical discs when upgrading any of the electronics, drives or servers. Right now the system can handle eight 100-terabyte libraries per buffer server.

JBII: But how can the system allow for unlimited data to be accessed indefinitely? Does the storage array have to expand at some point? What about the controllers driving the system as the amount of data increases?

Part of the Hitachi Digital Preservation Platform.

KM: HDPP does not eliminate the need for migrations, instead, using optical discs allows IT managers to avoid forced migrations, moving data around an environment when they want to, not when the media dictates. This saves money and allows for more strategic long term planning.

We understand that solutions, data formats and other technologies will advance over the years. Hitachi Data Systems Federal will provide our customers with a generational roadmap for technologies, drives and data formats so they can meet changing needs and technology upgrades accordingly. Again, unlike magnetic tape, all of the electronics and drives are serviced from the back of the library, so when upgrades occur, the data is never touched.

JBII: Does a system that uses optical media for storage cost more than a more traditional storage array? Or can the government actually save money with it?

KM: Using the HDPP solution, federal agencies can preserve data for as long as necessary and access it whenever they want, all through lower operating costs that are achieved with lower media migration costs, wider environmental storage requirements, migration-free technology upgrades and high media longevity and durability.

At the present moment, the additional costs associated with HDPP’s more durable and reliable storage medium are recouped after the second forced technology refresh of tape. In a traditional Blu-ray environment lifespan of 50 years, tape would be forced to go through eight refreshes. In an M-DISC 1,000 year life span, tape would be forced to go through 166 refreshes. The cost savings is stark while the possibility of data loss is virtually eliminated.