A story of too many tools, part 2: Back to the future
It’s five years later, and our story continues. The shop is now recognized industrywide as a leader in integrated IT service management. Al (the former ITSM/ERP consultant) is now VP for IS Operations, talking to a review team from an industry partner sent to learn some lessons. The scene opens in a conference room with Al presenting…
“…The first thing we decided was that we absolutely had to decouple our IT business process owners from running their systems,” Al said.
“What? How could that work?” said Tom from the review team. “Our Incident Management group owns their tool, our Enterprise Architecture group owns theirs, and so forth. That’s just the way it has to be!”
“Not so fast. Do your business clients run their IT systems? Does Finance ‘own’ SAP? Does HR ‘own’ PeopleSoft?”
“Well, no – I mean, they are key customers and stakeholders, but those systems are run by IS…”
“My point exactly,” Al said. “Just because someone has worked their way up in an IS organization doesn’t mean they are any more qualified to engineer and run complex, enterprise-class systems for themselves than a business person. Not everyone is an applications software engineer, just like not all medical personnel are doctors. You need people whose career focus has been architecting, delivering, and operating application systems for clients; people who understand things like requirements management and the software analysis and design process.”
“So you’re saying we need an IS applications organization for IS?” asked Tom.
“Yes. For all kinds of reasons. First, in terms of regulatory compliance, you have frameworks like COBIT that insist on segregating development from operations. This is widely done; developers increasingly don’t have rights to implement production changes. But the one area we see where this is far too blurry is in the internal IT management tooling itself; because that is where the process is enforced, you have a ‘who watches the watcher’ issue. By separating Change, Config, and Release as process areas from the team that is responsible for tooling, you create the same sort of checks and balances we insist on for business-facing systems.”
“I’m still not convinced,” said Gene (another member of the review team). “There is so much tradition in IS that the functional areas get to own their tooling and work with it.”
“And how is that different from having Human Resources own the payroll software? A catastrophe resulting from that arrangement ten years ago is why we have the current IT organization -- HR tried to enhance their payroll system and didn't know what they were doing. We lost a year of data as a result and had to pay an IRS fine that nearly sunk the company. Stories like this were legion when unmanaged distributed systems started to appear outside the control of the old MIS organizations. If you're going to construct systems, you need software engineering specialists, professionals who focus on building automated solutions for requirements -- and this applies to packages as well as inhouse built stuff. Would we let Finance implement SAP?”
John from enterprise architecture chimed in, “We were doubtful at first, but then we really began to see it as an issue of ‘eating our own dog food.’ We’ve been frustrated for years with business areas that insist on functional silos with no architectural integration. We kept saying, ‘Just focus on your requirements and let IS determine the best tools mix.’ Then, when someone said the same thing to us, we pushed back. It became clear that we were being inconsistent. We’ve got processing and data requirements just like anyone else, and once we sat down and documented them the overlaps with all the other functional IS areas really started to become evident.”
“So, you actually architected yourself?” Gene asked?
“Absolutely. We came up with an enterprise process architecture for the IT capability itself, based generally on ITIL, COBIT, and CMM. We then did a parallel decomposition of the major data subject areas and assigned them to the process areas.
"Our metadata folks were really helpful here, as they’d been thinking about the data side of this problem for twenty years. Basically we matrixed metadata against the process areas. Then we started thinking about systems, and we ran our vendors through a wringer. There’s been too much voodoo in thinking about the systems requirements for IT; once we had it all down as an architecture we could see the redundancies.
"And man, were there a lot of them! For one thing, there’s little reason to have both Application Portfolio Management and Enterprise Architecture systems with their own repositories. Both deal with the same core entities, one from a planning perspective, the other more from an analytics perspective.
"On the other hand, we’ve been more aware of the process differences between these areas, and still have distinct folks working on these areas – APM being a more tactical capability feeding into Project Portfolio Management, while EA still focuses on the 18-36 month time frame.”
Tom asked, “Did the research companies – like Gartner – help out in any of this?”
Al just about blew his stack. “Those companies were doing more harm than good! I guess I can see their problem; they’re trying to make sense of a lot of complexity by subdividing it – but the end result is ridiculous. We had one analyst come in all bullish on CMDBs, and the next one come in and talk about the ups and downs of metadata repositories for the past twenty years. Working for the same firm, and they didn’t even know each other! We pointed out the overlaps in data subject areas between their areas, and they just looked at the wall and said they’d get back to us.
“I think their business model is to sell as many magic quadrants as possible, but our vendors tell us they really get pigeonholed sometimes. The enterprise IT tools space has been way too fluid for these firms to make sense of, and everything is colliding.”
“So, how did you make sense of it all finally?” questioned Gene.
“We focused on the software/systems development lifecycle, no surprise there,” said John. “Capturing requirements, architecting, building, deploying, and running & maintaining things is nothing new. The key thing we insisted on was a ground-floor assumption that all the systems supporting these areas were going to interoperate without excessive spending on interfaces. This led us straight to the industry standards. The second thing we insisted on was minimizing the number of repositories.”
“Why?” Jessica from the review team wanted to know. “I mean, of course simpler is better, but you folks seem really obsessed with this point. We build interfaces all over the place.”
Sheila the metadata manager chimed in, “It’s because of the nature of IT data. It’s not really comparable to the data we manage for our business partners. It’s more like the data you find in CAD/CAM systems used by engineers; it’s heavily interconnected. CMDBs are all about interconnections: ownership and dependency relationships. There’s particular difficulties in working with this kind of data, starting with the fact that even power users can’t cope with querying it – you need software engineers and specialized tooling.”
“Huh?”
“It has to do with recursive SQL and how graph information is stored…”
“That’s enough,” said Al. “We’re getting too far into the weeds here; you two can follow up offline on the technicalities. To continue – it was clear looking at the process frameworks that the number one priority was to bridge the development/operations canyon, which was really a nasty place here five years ago. Development and ops did NOT get along; the hostility was starting to impact our ability to get things done for the business.
"We had a cross-functional team look at this and the first thing they recommended was to integrate the processes and tooling. The ITIL standard clearly recommends this, but we weren’t seeing any ITIL suites that interfaced with the software development lifecycle. So, we went to our developers and asked them what they thought. They suggested using the Unified Modeling Language.”
“Why? What good is drawing pictures?” asked Jessica.
“Ah – but it’s not just pictures,” said John. “The UML has a standard XML format called XMI, and we had a developer write some transforms so that our CMDB could import it. After the CMDB vendor saw what we were doing, they started to support XMI directly.”
“Sounds great – what’s the catch?”
“Well, you still have to think your process through really carefully. It’s easy enough to move the first cut of something around; handling incremental changes and understanding the delta between two versions of a UML model was a real challenge until our vendors stepped up with some better solutions.”
“OK,” said Gene. “You’ve got your developers basically defining your configuration items, which is good – right now, we have the change management team doing it and I’ve noticed the developers give them these architectural diagrams to facilitate the CI identification; this cuts a step out of the process. You’re simply using the diagrams the development team has had all along as the data entry device.”
“Exactly,” said Al. “Of course, this doesn’t work for machines or packaged software, usually. But the custom-coded enterprise systems that we ourselves have to support, those are among the hardest configuration management challenges we have, with their complex interfaces and interrelationships. Now we can drill down to the application level and show systems, subsystems, and how they are integrated. It’s just amazing to me that we once upon a time had an enterprise architecture tool and CMDB that didn’t talk to each other; we still keep them separate for operational reasons but now they are kept in synchronization through data exchanges.”
“We also held our management framework vendor’s feet to the fire and insisted they align their service management hierarchy with the CMDB components; that had been another dual-maintenance headache. Once we realized this all was basically CI identification and elaboration, the integration problems became a lot clearer.”
Changing the subject, Tom asked, “So, I hear you have done some things also with project management.”
“Yes,” said Sandy from the project group. “Even ITIL doesn’t do too much with project management per se, but we’ve determined that integrating project management with your configuration management practice can be very valuable. CIs are generally project deliverables, and if you maintain the trace from a CI to the project that created or modified it you’ve got much better visibility into the software development lifecycle. We can tell, for example, the cumulative defect rates of projects using agile methods versus those using more traditional methods. Usually this information is only tracked during the project, but we can now hold projects accountable for poor software quality long after they’re closed.
“Also, after we started doing this, we found that our larger scale programs were doing a lot better. Having configuration management integrated with projects all the way through has given the project teams a lot more visibility between each other, which is key when a single component or subsystem is critical to the whole program – everyone can see its status, who owns it, what else is dependent on it, the current tasks affecting it, and so forth.”
“Wow,” said Gene. “Isn’t this a lot of overhead?”
“Absolutely, it’s a lot of work and was a big change in how people approached things,” said Sandy, “but when we started looking at present practice we saw it was actually more efficient than the endless spreadsheets and Visio diagrams getting fired back and forth across emails. Now there is a single source of truth that everyone relies on, and there started to be a ‘virtuous cycle’ of people holding each other accountable for its accuracy.”
Tom asked, “So, how does all this benefit you from a planning perspective?”
Al said, “I’m glad you asked that. It’s been just great. We have amazing visibility at a high level, with several digital dashboards for IS that are getting extensive use. For any service, we can now instantly show its total cost of acquisition and ownership, through our integration with the project management system’s financials as well as the usual ITSM reporting on related changes and incidents. It’s an integrated picture of the IS organization that really helps our leadership understand their world, and formulate strategy accordingly.”
“Any final recommendations?” asked Tom.
“The higher levels of the technology stack are harder, because they are subjective. No-one argues about a server or a component; everyone can see they are there. But what an ‘application’ is can be controversial, let alone a ‘process’ or a ‘service.’ Basically, you have to have a process in place to drive consensus, and a cultural understanding that you’ll never get these squishy concepts ‘right’ – you can only make them ‘useful.’ However, the fact that this is so hard also indicates its value. You’re building shared knowledge among your team, and never underestimate the power of this.
“Finally, don’t give up on challenging the vendors to support standards. They have no incentive to do so without your insistence, and being locked in to one vendor’s view of the world is not a pretty thing. This requires perpetual vigilance, the eternal price of freedom.”
