Contribution of New Hadoop on Lustre® Connector plugin to HPC Open Source Community Further Streamlines Big Data Workflows
From this week’s Lustre User Group 2015 Conference Seagate has announced it is contributing a new Hadoop on Lustre® Connector plugin to the Hadoop Open Source Community designed to simplify and accelerate Hadoop workflows for greater efficiencies and faster time-to-results.
The plugin is an enhancement to Seagate’s Hadoop on Lustre Connector, which enables Hadoop and HPC Lustre clusters to use the same data and eliminates the need to move the data between file systems and storage devices — a critical feature as more HPC environments use both Hadoop and Lustre as part of their data analysis workflows.
The new plugin provides users with more consistently high performance from their big data environments and simplifies use through key features, including:
- Improves performance through a basic connection from the Hadoop to Lustre client on the compute node.
- Includes additional code for MapReduce jobs to tighten the connection between Hadoop and Lustre clusters, enabling both to work together and target different data challenges.
- Supports customizable configurations for a variety of data environments and needs, including setting Lustre as the default file protocol or retrofitting any Hadoop cluster so it can access Lustre.
“The ability to drive truly valuable insights from massive amounts of big data hinges on the effectiveness of data-intensive computing, and the accompanying analytical approach using methods like Hadoop,” said Ken Claffey, vice president of ClusterStor, Seagate Cloud Systems and Solutions. “When it comes to data storage, today’s HPC environments are vast and varied — making the process of gaining actionable insights that much more complex. Through open source contributions like the Hadoop on Lustre® Connector plugin, Seagate is committed to providing new enabling technologies that help users gain the most out of their data while providing a pathway for continued growth and innovation.”
Seagate will discuss the Hadoop on Lustre Connector plugin as part of its presentation, “Understanding Hadoop Performance on Lustre,” at the Lustre User Group 2015 Conference on April 15.
More information is available here.
