Yesterday I read an article by Chad Sakac to which I can’t refrain from commenting. He wrote a very long post, adding a lot of details and trying to justify why XtremIO arrays need a totally disruptive upgrade (with a wipe out of all the data) to move from v2.4 to the new v3.0 software. To me it seems like he is only trying to cover his back.
Immaturity
The impression that XtremIO is an immature product is totally confirmed and it leaves a lot of space for competitors to comment and build some FUD, probably justified in this case.
Anyhow, the sole impression one gets from this update (fundamental for receiving new features) is that the product is still immature. I mentioned my doubts on the maturity of this product a while back, and then I tried to be more understanding in a recent post considering the efforts EMC is putting in its development. But, I have to admit, a debacle like this was totally unforeseen: after the last meeting at VMworld with the XtremIO team talking about “continuous upgrades”, no one could have imagined anything like this… is “continuous” meant to be a synonym of “disruptive”?
The upgrade
The amount of work put in behind a firmware update of this sort is massive. It means backups of all your data, stopping applications, resetting the system (which also means wiping out all the data!!!), restoring data and restarting the applications. Many things could go wrong and IT people don’t like that, do you?
It could take anywhere from hours to days (or even weeks if you don’t have the right maintenance windows!).
Yes, I can agree that hypervisors have non disruptive storage migration capabilities (like Storage vMotion) but it’s not always the case. What happens if you have only a XtremIO array and it is running mission critical workloads? Where will you temporarily store your active data?
I think it’s totally unacceptable (especially from the most important storage vendor), and I would like to know the reaction of EMC XtremIO customers. BTW, this is not the first time XtremIO applies this kind of disruptive upgrade.
Why it’s important
Immaturity of a product like this is always a problem for the end user and has its costs. It’s acceptable only if it is well known, you get a better price and the consequences are clear. At the end of the day it’s not bad being a beta tester, it has its advantages sometimes, but being an unaware guinea pig is another thing.
As an XtremIO customer, my reaction was annoyed at most. Do I wish it was online? Of course. Was upgrading hard, time consuming, or frustrating? No.
Also I don’t think Chad was trying to justify the need for a impactful upgrade but explain why it was the way it was. The article made it clear the goal of a non disruptive upgrade for XtremIO.
Chad’s latest comments has suggested free professional services and hardware to do the upgrade. For existing XtremIO customers, they can’t argue with that.
It is a problem, but EMC have committed to fixing it without cost to the end user.
You can’t really complain about that.
Every time you stop a mission critical application (even when planned) users get angry with IT, you have costs and business activity is compromised.
Not in my experience. That is why we have planned maintenance windows which are well publicized and approved by the business. When you build a constant story of success, end users are fine with planned application stoppage.
If it is such a mission critical of an application I would ensure it has something like a Storage VMotion option to ensure availability for when any data migrations need to happen. After all, eventually it will need to transition to a different storage array due to a tech refresh or some other event.
[Disclosure – I work for EMC XtremIO].
Enrico – there’s more information that I think is relevant to your readers. First, nobody is being forced to upgrade to XtremIO 3.0. It’s at customers’ discretion, we’ll continue to support 2.4 for years to come, and when customers want to upgrade to 3.0, EMC will be there with no-cost services to move their arrays from 2.4 to 3.0 using a number of tools (like Storage vMotion, PowerPath Migration Enabler, etc.) as required to be as seamless to applications as possible. EMC is taking care of everything.
Second, we’re delivering tremendous value to our customers with the 3.0 release. We consulted with a large number of customers during our release planning. They told us that inline compression has huge value and that they’d like to leverage it on the XtremIO arrays they already bought. Most of the time this kind of technology would be offered only on an entirely new product. We were able to deliver it on current hardware and we are not charging for the feature. A customer with an 80TB XtremIO array gets more usable capacity – it could exceed 160TB because of v3.0’s inline compression – and performance will go up as well. That’s a lot of usable XtremIO capacity for no cost outlay, and customers told us that as long as we handle the process of getting them onto 3.0 then they’d like us to deliver this capability now, and not wait.
Furthermore, customers told us that they’d prefer we do this now because they are only planning to use their XtremIO arrays for more and more applications in the future.
The media and our competitors of course want to make noise about this. Sensationalized reporting drives readership, and FUD helps when trying to sell against XtremIO. But the reality has nothing to do with maturity of the product – XtremIO is very mature and is serving the most difficult and challenging workloads on the planet every day – with amazing simplicity and better than five nines of field measured availability.
This was a decision made in the best long-term interest of our customers and with their input. For our customers it is about how we manage this with them more than anything else. EMC will make sure customers are happy and that XtremIO continues to be a runaway success and the fastest growing storage array ever.
Josh,
thank you for your comment.
My idea doesn’t change but it’s very good that you are working to minimize the consequences of this upgrade for your customers.
I appreciate your folks like Chad trying to handle this situation, as well as your services and support folks, but it’s hard to take anything you folks say seriously. The EMC xtremio website still claims:
Uptime & Non-Disruptive Upgrades: XtremIO eliminates the need for planned
downtime by providing non-disruptive software and firmware upgrades to ensure
7×24 continuous operations.
How can anyone take anything you say further as credible? You might as well claim it makes a perfect latte. It’s an insult to the venders who worked so hard building an enterprise class architecture with enterprise class firmware upgrade features. I’m sure they had trade-off’s as well, but picked enterprise availability over all else.
There have been multiple blog posts recently criticizing EMC for their totally deceptive marketing in regards the capabilities of XtremIO. Of all the comments on these blogs, the most interesting are from customers defending EMC over this practice. It appears they don’t mind being lied to or deceived or for paying more for a product than they should have based on its TRUE capabilities. Let’s be honest: If EMC had told the truth about their upgrades, do you think you might have talked them down just a little bit more on the purchase price? And I would also ask how do you think your company feels about you defending the company that just lied to you and thus was able to charge more for their product? You paid more than you should of and then you defend the vendor that screwed you? Wow! Now that is what I call a loyal customer! Not only do you not mind being lied to, but you even take time out of your busy day to defend EMC on blogs! Come to think of it, that doesn’t sound like any customer I’ve ever met.
This is all smoke and mirrors from EMC their contracts are so water tight many Customers would have difficulty in speaking out anyway. But I’m sure any vendor could find a method of persuading a couple of friendly faces, no matter how painful the situation.
If you take a step back it’s obvious EMC have been economical with the truth from the outset. When the original designers came up with XtremIO they were designing a platform that fitted a very narrow market space. For instance they never intended the platform to scale as high as EMC are now requiring, otherwise they would have done the simple math on 4K blocks and realized the addition of dedupe and other technologies meant their in memory feature would make for a non viable platform commercially.
So you can absolutely guarantee EMC engineering will have done the math and will have been aware of this limitation and the changes required to make this work from the outset, to say they weren’t would be naive at best. Also keep in mind the original design criteria (which differs from EMC’s use case) will continue to impact the platform going forward, meaning EMC’s shoehorning of XtremIO outside of it’s design goal immediately creates opportunities for a compromised platform.
Now given that EMC have been briefing Analysts, Customers & Bloggers, making youtube video’s & producing slick websites as well as the obligatory powerpoint decks for months and all stating non disruptive upgrades. Regardless of the help EMC are now offering, Customers really should feel aggrieved. Not only were they intentionally misled, but there’s more on the way as EMC move from their “get it out the door” V0.5 hardware to something resembling a V1,0 version platform.
As a customer I can tell you, i’m not aggravated because I was not lied to. I have known since getting my array that a destructive upgrade was coming. While I’m not saying i’m happy that this is the case – i’m saying the downside hasn’t outweighed the upside.
All this negative comments seem to come from other vendors. The customers i’ve seen and spoken with echo what i’ve said.
I’m not sure that’s really all that’s at issue here, regardless of what EMC told you, their public facing web site, presentations, videos and briefings have continued to state non disruptive upgrades throughout when this patently want the case.
Their get out of jail on this now is that all future upgrades will be non disruptive from this point on. Therefore we don’t need to change the messaging. But let’s face facts they only came clean publicly because they were exposed by a Customer.
It seems that despite their underhanded methods, their use of social media and recruitment of vocal apologists means EMC’s marketing machine win either way.
Happy or not, even if EMC were honest with you personally, they weren’t with other users to whom this has come as a surprise, and I’m sure others are yet to find out.
Based on the above websites, slideware, videos etc they definitely weren’t honest with the wider market and storage community as a whole. Which in effect provides them with an unfair advantage vs those who can offer non disruptive and non destructive upgrades and expansion to their products.
So regardless of the effect on you personally this little oversight has wider implications for the storage industry as a whole. iI vendors can just make stuff up with the full support of customers, who are we to believe going forward.
Almost all of the “backlash” I’ve seen has been limited to other vendors. Which is why I even choose weight in on the topic. I’ve seen the slides all talking about NDU, but it was always framed to be as forward looking.
Mark, in your first comment you said you were annoyed at the XtremIO destructive upgrade. And I think we could all understand that annoyance.
But why were you annoyed in the first place? Now we find out that the EMC sales rep admitted to you that their marketing was a lie, the upgrades were destructive, and you bought it anyway and got exactly what you paid for?
Mark, you are truly a storage vendor’s dream customer.
I can only speak to my experience, and that was any slide/presentation talking about NDU was forward looking in nature. I knew I would be facing this upgrade, and the juice was worth the squeeze. The value I’ve seen from the array outweighted an annoyance.
Dream customer? I doubt it. I’ve been ultra critical on my vendors on various things which do annoy me. This annoyance is simply not that big of a deal. It’s a non issue.
Mark not an issue for you, but let’s face it you speak only for yourself not the xtremio install base. if further proof were needed the initial complaint was made by a customer.
As for forward looking statements none of the public facing information carries a warning that Non disruptive upgrades and expansions are an aspirational feature.
As I asked above should vendors be allowed to just make stuff up ? It seems your not too bothered that others may be misled so long as your in the know and therefore it doesn’t impact you personally.
All of the public facing material suggests these are online and there’s definitely no suggestion that relatively basic features for a high end array are not just disruptive but also data destructive.
Reply of you like but your defense of EMC’s behavior on this is completely unjustifiable.
Mark, I strongly suspect that everyone reading this blog is storage savvy, so please don’t try to make a destructive upgrade in 2014 sound as unimportant as a faulty cable. Leave the spin and contortions to the EMC marketing people. FYI, here is a video from November 2013 with the EMC VP of marketing claiming NDU for XtremIO: https://www.youtube.com/watch?v=-uTkO758Wxw
Mark, SolidFire doesn’t have this problem with disruptive upgrades. Unlike what Chad said, there is a way to upgrade even metadata and underlying schema without causing downtime if your architecture is shared nothing. We’ve done it in our upgrades in the past and no customer had to know. We also allow you to mix versions and controller models, so you can upgrade and refresh on a rolling and once again, non-disruptive basis. Just saying…the juice was not really worth the squeeze…