HPE Data Management Framework 10PB 1EB+ 3-year Subscription E-LTU
HPE Data Management Framework 7 (DMF7) delivers centralized data management across High Performance Compute (HPC) and AI storage systems. Protects scalable, parallel file systems like Lustre and Spectrum Scale. Namespace reflection is used to create an independent snapshot of file system state, allowing administrators to recover file systems in a known good state from a point in time. Also, maintains file versions, allowing users to recover files from points in time reflecting successful job runs.
HPE DMF7 automates data movement between tiers in a storage hierarchy, e.g. between flash and disk. Administrators and users can also use HPE DMF7 to move files between file systems, for example when files must be moved from storage that is being retired. Improves utilization of expensive, high performance storage by automatically moving stale data to lower cost storage tiers, creating a virtual storage space under the file system that appears to scale beyond the physical capacity.
SKU # R1D45AAE
- A scalable database. The data base is distributed across nodes for higher system availability, is scalable to billions of files, and delivers high performance for queries.
- Namespace reflection. HPE DMF7 maintains a reflection of managed file systems and uses it for data management operations instead of compromising the performance of the file system metadata servers.
- Support for extensible metadata. Administrators can use metadata to customize queries, use extensible metadata for policy management and create personalized data sets from associated files.
- Data curation. Users can stage their files into HPE DMF7's independent back end data base for long term curation. Later, users can locate and recall their files to an entirely different file system.
- Query engine. Administrators, users and the HPE DMF7 policy management engine all use a common query tool to locate files and filter the results.
- Native file system integration. Native integration uses the standard tool set for Lustre and Spectrum Scale to increase utilization and automate data management operations.
HPC/AI Applications are Data Rich, and Data Must be Protected
In whole or in part, data can be lost for several reasons: user error, application failure, storage system failure, and malfeasance. The adage 'just rerun the application' isn't a useful strategy in present day HPC, especially when the data from sensors, satellites and signals may be ephemeral.
Loss of the file system due to failure has catastrophic impact upon availability of the high performance compute cluster. Even when a file system has tools for repairing, the complexity and time it takes to repair a broken file system can extend the compute outage beyond acceptable SLAs.
Up to now, protecting file systems and data has been an costly investment with imposing drawbacks, including the lack of backup windows, backup utilities that are sub-optimized optimized for PB-sized parallel file systems, and the negative impact upon performance from scanning file system metadata.
HPE DMF7 creates immutable versions of files and takes snapshots of the file system state via namespace reflection. Both operations are managed via administrator policy. Recovery can be optimized for lowest RTO from disk, lowest cost from tape, and/or from remote locations via S3/cloud.
The Emergence of Exascale Computing is Challenging the Scaling Limits of Legacy HPC Storage
The volume and diversity of data demanded by HPC/AI applications has fueled the growth of the "storage beast' that feeds on HPC budgets. At the same time traditional parallel file system architectures are struggling under the weight of relentless growth in the number of files and inodes.
Perhaps the challenge is felt most acutely by storage administrators, who struggle to maintain enough free space in costly high performance storage while users independently flood the file system with new files. Increasing the storage budget and/or deleting user files aren't practical remedies.
HPE DMF7 manages free space in storage by automatically moving 'stale' files out of the high performance storage, creating an underlying storage space that is bigger on the inside. Administrators easily manage this by policy settings, alleviating the need to take brute force actions.
Eventually, administrators need to work with users to prune unused files from the file system to ensure metadata performance isn't undermined. When old files are marked for removal, no data has to be moved since HPE DMF7 already preserves files and metadata in less costly back end storage.
HPC/AI Storage Environments are Diverse and Data has to be Portable
Storage systems are optimized for performance, capacity, and cost, and data is always in flight between these tiers. Application workflows demand that data follow the user and the application, and administrators are continually pressured to manage storage costs and push data down the hierarchy.
Managing HPC/AI data movement is an intimidating task. Tools aren't easy to use, they don't scale well, network pipe bandwidth is limited, and users may not have the needed skills. When data cannot be moved easily and the motivation to move it is low, the default choice is to leave it in place.
HPE DMF7 automatically migrates files down the storage system hierarchy without administrator interaction and recalls them up to high performance storage on demand. It uses parallel data movers and the high speed network to move files faster than standard desktop utilities and in a single step.
Technology migration is a common driver of data movement and HPE DMF7 future proofs against this risk. It automates the migration of back end objects from older, inefficient generation HDD/tape technologies and on to generations that have the highest density, reliability and performance.
When the file system must be retired, administrators move the file system and files onto HPE DMF7 back end devices. Once they are protected, the file system can be staged into an entirely new namespace and files can be staged into the new namespace or remain in curation by HPE DMF7.
Reducing HPC Storage Costs Means More Budget for Compute
The unabated growth of HPC data and the orchestration of large data sets for AI/machine learning are driving unprecedented growth in storage capacity needs. And the adoption of flash storage at a higher effective cost means that containing cost is still the primary goal for HPC storage buyers.
The storage beast works against that goal. The storage beast encourages HPC customers to replicate files on expensive proprietary storage, scale out the most expensive storage tier to satisfy data growth, and says it's easier to leave data in place on expensive storage than it is to move data.
The primary goal of HPE DMF7 is to reduce storage costs. Instead of purchasing more expensive storage, HPE DMF7 makes it easier for HPC customers to protect, scale and move data using lower cost storage and improve utilization and performance of mission critical HPC storage resources.
HPE DMF7 is the HPC storage management platform that automates data workflows and reduces HPC storage costs so HPC customers can spend their valuable project budgets on the infrastructure resource that matters most.
Linux® is a registered trademark of Linus Torvalds.
Find what you are looking for?