The other day I was talking to a mid-level IT manager, who was in the process of procuring a large amount of expensive software. He explained that he was buying a Commercial Off-The Shelf (COTS) solution that he knew wasn't particularly good, but he had, as he said, "no choice."
"What do you mean, no choice? Why don't you use one of the several excellent freeware alternatives?"
Apparently, his company had a policy against using unsupported software, open source, or freeware. The policy was established by senior management, he said, because they needed a vendor standing behind a solution, so there was someone to accept liability in case the software didn't perform as advertised, and they needed to hold the vendor legally accountable.
Stop reading for a second, look away from your screen, and think about that.
So these large corporations are going to hold software vendors legally accountable? Have you ever heard of a case where a company has plowed a ton of money into a software deployment, and has utterly failed to get the software to work as advertised? Have you ever heard of a case where a company has bought a product and then discovered that, in order to work, it needed to spend an additional pile of money on system upgrades, or work-arounds, to get what it expected? Have you ever heard of software that needed an extra army of con$ultants to deploy it? Have you ever heard of a case where a company has spent a bunch of money on a product only to find out that a feature they were promised hadn't made it into the release? Have you ever heard of a company that bought some software and found out later that it was buggy garbage that required constant patching to keep it from being open season for hackers? Of course you have.
Have you ever heard of a vendor being taken to court because their crappy software didn't work as advertised? NO!
So, if I understand the logic of my friends' senior management, they refuse to allow use of free/open source/unsupported software, so that they can have all the benefits of something that they never have the guts to take advantage of. My friend is not alone, either - far from it - the great corporations of the world take a look at the "warranty" that comes with their software, and roll over on their backs with their soft underbellies exposed. On one hand, they pretend that the vendors are scared of their legal muscle, but on the other hand, they don't have the guts to challenge a click-wrap license agreement.
If these same "visionary" senior executives had the collective wisdom of a swarm of gnats, they'd realize that the customer is always right because the customer holds the purse-strings that the vendors depend upon. If even 50% of the FORTUNE 500 announced that they were going to stop buying software that came with (effectively) "warranty of no warranty" the vendors would scramble all over themselves to negotiate special back-room deals before word spread. Many vendors survive on the 10% to 20% annual "maintenance fee" that they collect. Many customers feel they have to pay that annual tax because it's the only way that they even have a prayer of eventually getting something that actually works. I know I'll probably be burned in effigy for putting that in writing , but it's the truth:
Have you ever heard of a customer keeping a product on maintenance because they hope that by installing the next version they might get something that actually works? Have you ever seen an IT professional praying that this one will not suck?
What was it that W.C. Fields used to say: "Never give a sucker an even break"? The vendors aren't going to give their stupid customers an even break. They don't deserve it.
CIOs/CTOs: If you aren't allowing the use of open source/freeware because you want the legal fallback - you'd better take advantage of it or just keep taking it on the chin. Get smart, guys.
Two years ago one of my consulting clients called me to discuss an online banking application. His employer (a major bank) had purchased an online banking application from a third party. Basically, it was one of those "turn key solutions" where all you need to do is install it, change the .GIF logos and a bit of HTML, point it at your mainframe, and you were ready to offer E-banking to your customer. They paid $500,000 for this software. Once they had it more or less working, they were ready to roll it out, and decided it was time to look at security. You can probably guess what happened next:
There was none.
Wait, I'm being ungenerous. There was a little security. They recommended that IIS be configured to use SSL ("Secure sockets layer") which provided the "security" part of "secure online banking."
I got into a couple of conference calls with the provider of the software and started asking probing questions about the security architecture of the product. At first, they simply cancelled the conference call because I had not signed their non-disclosure. We got all of that cleared away and then I got a chance to learn that the software basically exposed a stock NT4 Windows/IIS web server to The Internet. Putting additional software on the NT4 box would be "an unsupported configuration" and the software vendor would not answer support calls in case there were any questions about anything at all. The NT4 box talked over Microsoft's incredibly secure DCOM remote procedure calls to a "back end system" that translated the DCOM requests into SQL calls that were sent to a mainframe. Things in this configuration that had security problems:
The software provider's "supported configuration" was that the customer could put a "firewall" in front of the NT4/IIS machine and then "everything behind it will be protected." Apparently they had never heard of IIS bugs. Actually, they had apparently never heard of computer security.
So, then what happened? Did my client sue the software provider for misrepresenting their product? They might have had a case, right, since "security" is a property of "secure online banking" and the software provider billed their product as "secure". This is, after all, why CIOs and CTOs don't allow open source or "freeware" - it's so they can sue chumps like these and get their money back. Of course that's not what happened.
My client tried to design a security system for the software provider. Suddenly I found myself in lengthy con-calls trying to explain to the software provider things they could do to help shore up their pathetic "architecture." After a few days of this I stopped even recording my time spent; I couldn't bear to charge my customer for my time; they were already haemmoraging money on the project. After a week of trying to explain things to the software provider's "chief architect" I gave up and submitted a written opinion to my client that they should delay the roll-out and find an alternative E-banking solution. Instead, my client bought a firewall. At least they were smart enough not to just buy a stupid turbo stateful whatever firewall; they bought "web firewall" that could be programmed to permit or block specific URLs and which could be used as a layer of input validation in front of the NT4/IIS box. As it turns out, they have survived long enough with that configuration (more or less) and a lot of hard work, and now senior management thinks their security people are "alarmists." Their credibility is shot, and anyone with a business case can now steamroll over sensible security recommendations just by jumping over the security team's branch of the org chart.
What did all this wind up costing? $70,000 for a "web firewall" plus a consultant (me!) plus auditors plus additional "testing" plus "pen testers" - on top of the $500,000. Plus lost time. If whoever had picked the product in the first place had simply done the right thing and committed seppuku, a lot of trouble would have been saved. This, I call "throwing good money after bad." Security, when added as an afterthought can cost 20% to 50% more than "doing it right the first time." In this case, it added nearly $150,000 to a project that was already late and over its $1 million budget. And, if the software provider had a clue, they could have come up with an add-on "security module" that made their product not suck as much - for only $250,000. Don't laugh. It works for Microsoft.
The entire computing industry has become addicted to "throwing good money after bad" to the tune of billions of dollars. Take Microsoft Windows, for example: to be remotely tough enough to withstand Internet use, it needs automated patching, antivirus, a firewall, etc. Of course Microsoft provides mediocre built-in capabilities to meet those requirements, but only a complete doofus would field a Windows box on The Internet for a mission-critical system without spending at least as much again as Windows cost - in order to make it usable. Sure, it's a $100 operating system - but it comes with a $100 hidden liability cost, and a patching and maintenance cost that will just keep on costing you forever. If that's not stupid, can you tell me what stupid is?
I absolutely don't understand how it is that companies convince themselves that they are saving money by buying crapware and then paying a king's ransom to make it work. At one company where I used to work, our sales VP bought Siebel's COTS "customer relationship management" tool for about $250,000. Then it needed Oracle. And of course then it needed "customization" which meant it needed "consultants." By the time the plug was finally pulled (the VP got to keep his job, by the way) almost $1 million had been poured into this software pit and nothing had ever actually worked. In retrospect, I probably could have found a kid out of school to write something in PHP and mysql and it would have cost under $200,000 per year for him to sit around and drink coffee and mess with it - and I'd have been able to fire him if it didn't work, which is ever so much more satisfying than just writing the consultants a big check, telling them to leave, and looking at a Sun server, a pile of CDROM distributions, a rack of manuals - a $1 million write-off.
But the madness doesn't actually end if the software is actually cajoled into working. Suppose we had gotten Siebel working - why then we'd have had to maintain the customizations when each new version came out. Remember - all this stuff is customized. What's "off the shelf" about that?
Perhaps I should not even enter into the topic of US Government software boondoggles and how those are done. A measly $1 million write off is not even on the radar screen for those guys. Take, for example, the FBI's "Virtual Case File" (VCF) project, which is finally acknowledged to be a failure after having had $300 million on-the-record dollars sunk into it. "On-the-record" dollars are just the payout to the con$ultants and software companies and don't take into account the salaries of all the project managers who helped oversee the disaster, as well as related projects that were on separate budgets. Basically, VCF was a "GOTS" (Government Off the Shelf Software) project - an application that was going to be "integrated" atop COTS products (i.e.: Oracle, etc). VCF was probably a legitimately large project, and I am sure data conversion represented a large amount of the unsuccessfully spent money, but - what the hell - all this system was is a large database that can store graphics, manage and control who updated what, generate linkages and visualize relationships between files, and store text, video, images, OCR, etc. This is not rocket science, we're talking about. My guess is that if the smart guys who built Google decided to build something like "Google Virtual Case File" it would be up and running in about a year, for under $10 million. As far as I can tell, nobody has committed seppuku over the VCF disaster. Why not? This is an investment in technology that would break or make most businessess, it's crucial to FBI's continued operations - yet it's a $300 million 10-year long clusterf*ck. Now these clowns want more money so they can "try again."In the last 20 years, the fad in business has been to "stick with core competencies" and to not do anything in-house that can't be done for twice the price as a one-time charge (plus 20%/year maintenance). What that means is you pay several times what the product is worth and, at a 20% maintenance rate, you buy it all over again every 5 years.
"Being smart about technology" should be a "core competence" for a CTO, right? Maybe we need to outsource the CTOs to a managed technology visionary company or something. Think of the money we'd all save!
mjr.
Dulles Airport, Gate G6, Jan 30, 2005