Here I am on day 3 at the Supercomputing summit and I have to say this has been one of the best IT conferences ever! I think I’ve been waiting for more than 15/20 years to re-live this kind of experience, which is very similar to what it was like going to conferences back in the 90’s.
Some of the sessions are off the chart (many of them too off for me!) and the expo area is the nerdiest, geekiest I’ve seen in many, many years. There are no booth babes, the women in the booths are engineers and really know their stuff. I love this show. All the technology is exposed, racks full of (working) equipment, CPUs, GPUs, real live demos all around and probably most important are the university booths and research centers displaying their HPC projects, achievements and services. The show floor hosts a large number of ver-small startups, and some of them have amazing technology that could be of real interest not just for HPC but for enterprise IT too. I know I’ve already said this, but I love this show!
Commercial HPC
Besides the HPC itself, with its super-mega clusters, hundreds of PBs of storage and unbelievable networking, what I find really interesting is the intersection between HPC and enterprise IT.
I’ve seen many use cases that transcend the classic Oil&Gas or Genomics. Vertical high end applications like Financial risk analysis as well as Big Data analytics (just to make two examples looking at the most common use cases) that are inheriting technology, architectures and products usually found in the HPC space.
Another interesting thing is that some of the people running HPC infrastructures are managing them like big clouds (without ever mentioning the word cloud!). In fact, they have many different projects running simultaneously and have to share resources between multiple teams. Everything resembles the cloud: self-service provisioning, specific HPC libs/tools that look like PaaS, Security, some sort of SLAs and so on. At the end of the day, it’s like managing a big private cloud which serves thousands of users at the same time! But, again, Openstack was not mentioned very often (sometimes by vendors, rarely by end users… probably OpenStack is just taken for granted, but I do have more meetings today and tomorrow and I’ll try to get a better grasp on how these clusters are actually managed).
In any case it’s very interesting to note that there are so many similarities between these two worlds.
HPC loves object storage
Just a few words about this (I don’t want to seem monothematic!). Almost all object storage vendors have a booth here. There are two reasons for that.
<1. Primary HPC storage is as fast as hell (parallel file systems, Infiniband everywhere, Disks and a few SSDs). It's also relatively cheap considering the kind of throughput it can sustain but the volumes of data produced are incredible and, once data is created and analyzed it has to be accessed from everywhere and archived (some times for a long period of time). Object storage is simply becoming more and more convenient from both the cost and usability side when compared to other technologies. Some integrations, like the one shown by DDN, makes the process really seamless.
2. Some vendors are working on an interesting integration between object storage and compute clusters. I talked about this possible scenario a while back, and it was more about Big Data Analytics in the enterprise… but, as I said, boundaries between HPC and high end enterprise IT (or commercial HPC?!) are blurring. A future design for these Analytics/HPC clusters could leverage some Flash-based cache on the nodes and a huge object storage repository at the backend. Even though, these kinds of filesystems (GPFS and Lustre are the most common) can manage a huge amount of files and very large capacities, they are complex and data need to be moved somewhere else when it’s time to archive it! And it will get more and more complex with clusters heading towards the Exabyte scale…
Closing the circle
For me this supercomputing conference is like a candy store for a kid. I’ll be excited when I get back with all that I’ve seen and will have a myriad of ideas to develop!
It’s really interesting to look at two different, but somewhat parallel, worlds like HPC and enterprise IT, find the similarities and try to figure out how some of this technology and know-how can be transferred from one world to the other.
Disclaimer: I was invited to this conference by DDN and they paid for travel and accommodation, I have not been compensated for my time and am not obliged to blog. Furthermore, the content is not reviewed, approved or edited by any other person than the Juku team.
