Information spread across disparate hardware and workloads can prove costly, but can be orchestrated via a global namespace
Inaccessible data can prove costly for any business at any time. For research organisations, petabytes of unstructured imaging data needs to be properly sorted, orchestrated and managed through one unified namespace. After all, time is money, and data transfer delays can lead to discovery bottlenecks during projects.
Organisations routinely move information between storage systems and between on-premises to cloud infrastructure to support AI and HPC workloads. But the expenditure of data transfers and duplications, along with missed insights can all rapidly add up.
When dealing with limited funding, university bodies dealing in R&D need to find the right balance between granular insights and cost efficiency. This means having full visibility over management of resources, which can be achieved by unifying data and contextual metadata in one shared metadata namespace.
Hidden costs
Floyd Christofferson, Vice President Product Marketing at AI and HPC data platform Hammerspace, identified three key costs associated with storage silos that may be invisible to principal investigators and research staff.
“One is stranded capacity,” said Christofferson. “Research staff access their data via a file system metadata layer, but typically this metadata layer is bound into an individual storage type. So if data needs to move to a different storage type, that access path via the metadata becomes disconnected from users.
“The problem arises in that not all of that data needs to be on that particular storage system at all times, especially if it's very expensive high-performance storage. But often, organisations will keep the data there to avoid disrupting user and application access. The result is that other, less performant storage such as an object or archive store goes underutilised despite being far less costly, for fear of disrupting user and application access.”
This phenomenon is often called data gravity. Because when data does move, the result is significant operational complexity for system administrators, and potential disruption for users trying to track down their files. “This adds operational expense and a burden to your IT staff to shuffle data around between these different storage silos,” said Christofferson.
“And then the third and least easy to quantify - but the most significant - is risk. Every time you're moving data from one place to another, there's risk involved. Which data assets are you moving? Are you forking copies? Do you now have to manage multiple copies of the same file? These all need to be considered when determining risk.”
Shadow IT silos
Many researchers, especially those working remotely, rely on their own data tools that aren’t connected to the wider organisational infrastructure. Commonly, such tools are turned to as a result of budget constraints, or to bypass friction caused by IT restrictions. But these siloed areas of “shadow IT” can reduce team-wide visibility, as well as risking data protection and compliance gaps by being outside the direct control of company security measures.
Christofferson explained: “In most research institutions, you'll have central IT that manages infrastructure and data policies across the whole campus or organisation. Then outside that, you'll have specific departmental IT, such as a HPC centre or genomic research hub.
“A grant may require research bodies to purchase specific storage hardware, or out of impatience towards the restrictions of central IT, researchers will install their own network-attached storage for a project. Either way, shadow IT contributes to data fragmentation, which leads to increased costs and complexity. Alongside this, such isolated data often isn’t properly classified or protected.”
Implementing architecture that allows for full control over granular governance and protection processes across such heterogeneous storage environments is vital towards maintaining visibility and security. Partnering with a data platform provider like Hammerspace ensures that all data will be properly managed and organised by policy, regardless of the storage types it is on today, or moves to in the future.
Orchestration without movement
Navigating management of unstructured data is a long-standing challenge for organisations, and in R&D-focused sectors, time and funding must be managed without missteps.
A network-attached storage system may provide researchers with a centralised hub for accessing project documentation, but it does not eliminate the need to move data to other storage types over time. Additionally, each individual storage type becomes an island, lacking the shared metadata capabilities that bridge incompatible storage systems. This adds to the difficulty of keeping masses of unstructured data organised, and becomes a decades-long problem.
“Back in the olden days when we each had a PC, we used floppy discs to share data. To share a file with a colleague, we copied a file to a piece of physical media, which would include both the file essence and its metadata,” said Christofferson.
“Network-attached storage elevated the file system and metadata into a shared namespace on the network. So now everyone could see the same files via this common metadata layer, as long as all that data was in one storage type. The problem is that when those siloed storage devices with the embedded file system metadata became full, you were back to making a forked copy of both the data and its metadata. Essentially the siloed reality of data storage today goes back to the analog ways of putting data on a disc and handing it to somebody.”
This issue can arise when managing AI and other workloads that require unified access across multiple distributed storage types and locations.
Christofferson added: “High-performance computing (HPC) is a little bit more controlled, in that you will typically have a central compute infrastructure and high-performance parallel file system. But then you have scratch space or other external storage that you offload from your primary parallel file system architecture.”
The breakthrough comes when research organisations can manage their data across distributed systems through a shared global namespace. They are thus able to save on these costs by unifying access to all digital assets into a shared global namespace. This elevates that metadata access layer above the individual storage systems, unifying access, automating data placement by policy, and eliminating the disruption to users when data needs to move between storage types.
Think data, not storage
Christofferson advises that research leaders looking to optimise their data orchestration via such a unified metadata layer: “Think more up a level from storage, don’t get mired into what needs to be placed in slow or fast storage, and holistically consider what your data objectives are.
“Research organisations should focus on the qualitative aspects of data: the uses of the data; who needs to access and collaborate with it, and how it needs to be protected.”
From here, research organisations can treat their unstructured data as part of a unified, virtual data lake rather than purchasing new systems that exacerbate silo issues. Storage architectures can then be added more pragmatically, in line with specific needs, while workloads remain operational in the background.
“Hammerspace is unique in that we enable research organisations to get better utilisation out of the storage they already have. They don't have to worry about stranded capacity, shadow IT, or operational complexity because they can manage their data across any or all storage types via a unified policy-based global namespace that bridges them all.”
