Do you remember Primary Data? Well, I loved the idea and the team but it didn’t go very well for them. It’s likely there are several reasons why it didn’t. In my opinion, it boiled down to the fact that very few people like storage virtualization. In fact, I expressed my fondness for Primary Data’s technology several times in the past, but when it comes to changing the way to operate complex, siloed, storage environments you come across huge resistance, at every level!
The good news is that Primary Data’s core team is back, with what looks like a smarter version of the original idea that can easily overcome the skepticism surrounding storage virtualization. In fact, they’ve moved beyond it and presented what looks like a multi-cloud controller with data virtualization features. Ok, they call it “Data as a Service”, but I prefer Data Virtualization… and that’s a bold move.
Data Virtualization (What and Why)
I’ve begun recounting this story by mentioning Primary Data first, because David Flynn (Now CEO of HammerSpace and former CTO of Primary Data) did not start this new venture from scratch. He bought the code which belonged to Primary Data and used it to build the foundation of his new product. That allowed him and his team to get on the market quickly with the first version of HammerSpace in a matter of months instead of years.
HammerSpace is brilliant just for one reason. It somehow solves or, better, hides the problem of data gravity and allows to virtualize data sets by presenting virtualized views of them available in a multi-cloud environment through standard protocols like NFS or S3. Ok, yes, at first glance it sounds like hot air and a bunch of buzzwords mixed together, but this is far from being the case here… watch the demo in the following video if you don’t trust me!
The solution is highly scalable and aimed at Big Data analytics and other performance workloads for which you need data close to the compute resource quickly, without thinking too much about how to move, sync and keep it updated with changing business needs.
Hammerspace solutions have several benefits but the top two on my list are:
- The minimization of egress costs. A common problem for those working in multi-cloud environments today. With HammerSpace, only necessary/essential data is moved where it is really needed.
- Reduced latency. It’s crazy to have an application running on a cloud that is far from where you have your data. Just to make an example, the other day I was writing about Oracle cloud, and how good they are at creating high speed bare-metal instances at a reasonable cost. A benefit that can be easily reduced if your data is created and stored in another cloud.
The magic of data virtualization
I won’t go through architectural and technical details, there are videos and documentation on HammerSpace’s website that address them (here and here). Instead, I want to mention one of the features that I liked the most.
Practically, you can query the metadata of your data volumes. These volumes can be anywhere, including your premises, and you can get a result in the form of a new volume that is then kept in sync with the original data. Everything you do on data and metadata is quickly reflected on child volumes. Isn’t it magic?!
What I liked the least, even if I understand the technical difficulties to implement it, is that this process is one-way when a local NAS is involved…… meaning that it is only a source of data and can’t be synced back from the cloud. There is a workaround however, and it might be solved in future releases of the product.
Closing the circle
HammerSpace exited stealth mode only a few days ago. I’m sure that by digging deeper into the product flaws and limitations will be found, and it is also true that the more advanced features are still only sketched on paper… but I can easily get excited by innovative technologies like this one and I’m confident that these issues will be fixed over time. I’ve been keeping an eye on multi-cloud storage solutions for a while – I’ve added them to my list now.
Multi-cloud data controllers and data virtualization are the focus of a report I wrote for GigaOm Research. If you are interested in finding out more about how data storage is evolving in the cloud era, subscribe to GigaOm Research for Future-Forward Advice on Data-Driven Technologies, Operations, and Business Strategies.
Disclaimer: I was invited to Tech Field Day 17 by GestaltIT and they paid for travel and accommodation, I have not been compensated for my time and am not obliged to blog. Furthermore, the content is not reviewed, approved or edited by any other person than the Juku team.
