Both my dad and I are big football fans; he roots for Manchester United and I’m a big Liverpool fan. Despite this seemingly unbridgeable chasm, it’s actually the healthy rivalry (ours and the teams’) that makes us enjoy the league more than if we were supporting the same team. We’ve become stronger supporters of our respective teams as we compete with one another. At the same time, these two proud football organizations push each to attain even greater levels of talent and performance that benefit each team, fan base and the English Premier League as a whole. It elevates everyone involved.
I see some similarities with Seagate’s recently announced collaboration with Intel to develop and support Lustre. Seagate will now adopt Intel’s Enterprise Edition Lustre (IEEL) as its baseline Lustre distribution, a strategic move that brings together the capabilities of the two leading Lustre distributions, development and support teams. With the power of our companies behind it, users can have confidence in ongoing investment in HPC and collaborating on features and improvements. It means these two industry giants are on the same page, all using the same playbook to ensure a win.
A Fractured History
Lustre began as a research project by Peter Braam in 1999; in 2001, he started the company Cluster File Systems. Six years later, Sun Microsystems acquired the company’s assets and intellectual property. Braam left Sun (having been one of the assets acquired) and started ClusterStor in 2009 where he was joined by a number of key Lustre developers and architects.
One year later, Oracle acquired Sun’s assets and intellectual property; at the end of 2010, the company announced that it would cease Lustre development and place Lustre into maintenance support only.
End of an Era? Not Quite.
At this point, the Open Community Model was created, providing an ongoing source of support and development of Lustre. OpenSFS, EOFS, Whamcloud and Xyratex were all contributors and beneficiaries of the open community. Here’s where things become even more complex. Once again Peter Braam’s company, now called ClusterStor, was acquired, this time by Xyratex whom themselves had hired key Lustre leader such as Peter Bojanic, Then in 2014, Xyratex was acquired by Seagate. Meanwhile (going back to 2010 when Xyratex acquired ClusterStor), Whamcloud was formed by Eric Barton and Brent Gorda, and that meant the bulk of the Lustre engineering talent was divided between Whamcloud and Xyratex. This paved the way for both healthy competition and user-focused collaboration – sometimes they ran up against one another, and sometimes they were on the same side of the ball.
What happened next? Intel acquired Whamcloud in 2012 and expanded Lustre beyond traditional HPC. Whamcloud was focused on software and support, and Xyratex on the purposed engineered system for HPC. Prior to the release of ClusterStor, which ships fully configured and tested, HPC systems deployment could take months, The first deployment of ClusterStor was in 2011, with Seagate’s partner Cray. It was for NCSA Blue Waters – and the first system to achieve 1TB/s (terabyte per second). Five years later, it’s still sustaining over 1TB/s and still one of the fastest systems in the world. Seagate powers five of the six 1TB/s systems in the world and is the #1 choice for new supercomputer deployments.
Distinct and Separate Communities
You can understand, based on this somewhat disjointed history, how different “teams” developed over the years. Seagate and Intel managed their own Lustre distributions based on the Open SFS community release (for which Whamcloud/Intel managed under contract from OpenSFS), and each provided the benefit of additional levels of testing and complementary software capabilities. Until now, there were “fans” who used IEEL, and those who used the Xyratex/Seagate version, which was the community release that was “hardened” by our team of Lustre experts/engineers – the team that was first to get Lustre to run at 1TB/s with the patches and bug fixes that Xyratex engineers provided.
Bridging The Divide
With this new collaboration between our teams, Seagate will incorporate Intel Enterprise Edition for Lustre into its market-leading ClusterStor storage architecture for HPC, a move that will strengthen Seagate’s HPC data storage product line and provide customers with a fully supported version of Lustre parallel file systems to help drive advancements in the HPC and big data market. Engineering and support from both sides unite to allow for those talents to form a powerhouse.
The Lustre journey has been fragmented, yet what started as a simple research project has been improved over the last 17 years with contributions from Sun, Oracle, the open community, Whamcloud, Xyratex, Intel and Seagate and others. What has been constant throughout the journey is the support and talents of the developers. They, along with the broader community including many of the US and European leading Supercomputing Centres have been heavily involved with Lustre at every step and now, as we join forces we see ourselves as fans who know rooting for “the league” will only make each team stronger. We’re all excited about the possibilities – for our organizations, certainly, but for Lustre users most of all.
