What we learned about EA implementations

(based on many people asking why I recommend the implementation approach I do)

The fact is that most EA implementations fail i.e. they fail to produce sustainable value and a useful asset i.e. one that can be used to: answer questions (e.g. impact analysis), maintain knowledge, support business operations and business change, etc. Over almost two decades I have seen dozens fail (often undertaken by some of the brightest, most able, and most diligent people I have met).

I have seen EA initiatives in many types of organisations (forestry, health, insurance, government, finance, health and safety, eCommerce, retail, education, telecommunications, etc) and using many different generic frameworks and methods (e.g. Zachman, TOGAF, etc.) - and the quality of the result does not seem to be associated with either the type of organisation or which of these generic approaches is used.

It is true that in many cases inadequate tooling is the root cause, and EA "consultants", "experts" are quite happy to continue using office documents and essentially manual approaches because for service companies it is far more profitable to do the fishing, than sell a rod and teach the person how to fish for themselves.

Where poor tooling is not the cause what I usually have seen is work on an EA implementation commence:

• lead by people who only understand about 10-15% of what the technology being used can do (and have a partial understanding of its intended use) or lead by people who understand the tooling reasonably well but who have very limited domain expertise (i.e. in enterprise architecture, ICT strategy, business/IT transformation etc.).
• without a clear understanding of what is sought (requirements)
• without a clear understanding of how the solution will be implemented (all the components, how design will be undertaken, how operational procedures will be defined etc.)
• without a well defined (tried and tested) implementation plan
• without an understanding of the organisations change impedence issues (the organisational behaviours that impede change) associated with improving touchpoint processes and roles.

Unsurprisingly these implementations seldom proceed well and usually erroneous conclusions drawn from what is delivered (from those who "don't get it" or sometimes "don't want to get it").

If one doesn't start with a clear view of the outcomes being sought - it is unlikely the benefits will effectively be achieved (i.e. if you don't understand the requirements it is unlikely the solution you will deliver something that achieves them).

Therefore what is required upfront is a focus on what outcomes should be sought, i.e.

• who are the stakeholders and/or beneficiaries ?
• what are the scenarios of use they have ?
• what are the business questions they need answered ?
• and in roughly what stage of an implementation ?

Some EA approaches aim to achieve this but they are so generic (i.e. they don't make an assumption that suitable tooling will be used) so they can't be precise enough about the nature of the requirements captured, or how the design will proceed.

This is why we have developed a detailed implementation approach. This defines all the major areas of work, the key roles, logical milestones etc. that is specific to the implementation technology we use and can be instantiated quickly for a specific project

People often think that there must be a single detailed approach to developing an EA that will suit every organisation. There is a common meta-approach - but the detailed approaches differs based on the organisation and the goal. The problem is that different organisations are focused on making different types of changes e.g. new products or new geographies, mergers, risk reductions (including regulatory compliance, DR, BCP), cost reduction etc. so their focus is naturally different e.g. I have seen EA implementations with many different orientations e.g.

• Business and/or technology strategy (usually aiming at getting an explicit consensus as to what changes should be initiated and why).
• Business process redesign (adjusting the way the business operates, as a precursor to looking at technology changes)
• Application architecture (usually focusing on rationalisation, as a result of merger of unmanaged proliferation)
• Service architecture (usually focusing on developing approaches for service governance)
• Integration architecture (often now related to SOA initiatives, but in the past oriented around ESB, EAI initiatives).
• Security architecture (unfortunately usually as an after thought).
• Meta data and data architecture (unfortunately usually disconnected from the processes the data exists to support)
• Rules architecture (usually oriented at understanding how strategies and policies are implemented in processes, procedures and systems)
• Technology platform architecture (usually focused on getting better utilisation from servers, and/or planning for building extra server capacity)
• Storage architecture (usually focused on preparing for the addition of extra storage capacity).
• Standards management (usually oriented at governance)
• Programme management (usually oriented at understanding how a set of capital initiatives, or business change initiatives relate to how the business seeks to operate in future).
• Service level management (usually oriented at SLA templates, contracts and delivery exception management)
• New product or channel definition (usually products that involve systems at the heart of their delivery)
• Disaster recovery planning (usually leveraging off existing information about processes, applications, platforms and teams)
• Business continuity planning (as for DR planning)
• Requirements management (in the context of how the business seeks to operate)
• Package implementation management (usually focusing on understanding how business operates, and how this relates to the package in question - so either the package can be changed or the business can change how it does things (or both).
• Compliance management (understanding how regulatory or legislative compliance is being achieved)
• IT Skills management (understanding what skills are required for what systems, usually to allow rationalisation or as a risk management exercise).

If one looks around an IT organisations one will see information about all of the above being created, collected, used and disposed of (whether that is intent or not). It is usually presented in a diverse set of office documents (business plans, business cases, project charters/statements of work, technology documents, software design documents and models, infrastructure design documents and models, contracts, requirements documents, business continuity planning documents). This suits each person producing the specific artefact admirably and keeps lots of staff and consultants employed. To change this situation take time and needs to be done incrementally and with purpose.

Often when you ask an organisation what they want to focus on they will say: applications, infrastructures (servers, storage, intregration) etc. but be unclear about how they will make intelligent decisions about these things without understanding: business/technology strategy, products/services, and the business processes, information, rules, organisations etc. that these solutions are designed to support. Or you will get organisations that say they want to focus on all the above - but struggle to work out what will the initial focus should be so that the stakeholders with greatest immediate need will be provided answers to the burning questions they have, while knowledge is gathered and organised in such a way that it is reusable in down stream phases to answer questions that are currently of secondary or tertiary importance (or not even thought of).

The art of a successful implementation is prioritising - based on a complete understanding of what the tooling can do, a good understanding of EA best practice and clear understanding of what the organisations current goals and challenges are.

What we sought in a tool for business/technology transformation and EA

(based on many people asking what the requirements are for a tool)

People often ask me what lead us to working with the tool sets we use for enterprise architecture and business transformation (EA/BT). The following is a short background.

Obviously the overall objective is to improve an organisation's effectiveness (reduce costs, improve income/outputs, ensure regulatory compliance, build asset value etc.). This requires managed change. I had focused how to improve the quality of transitions from RHE's inception (the best people, methods and tooling). We had tried using a wide range of OTS tools (e.g. CASE, Ontology, Document management and of course Office suites) and building tools to allow us to manage this information. After many years of trying to determine how EA/BT work could be done (in a way produce sustainable value for clients) and a long term focus on methods (including frameworks) we determined the tooling was inadequate (and the tool limitations fundamentally affected the method).

In 2002 we reassessed what tools were available that could be used to support:

• business strategies: drivers, plans, markets, products/services, locations, organisations, resources, change/transitions, projects/initiatives;
• business operations: communciations, services, processes, rules, information, organisation etc;
• technology architecture: enterprise, solutions, components/systems, etc. Supporting any: styles, topology pattern, interaction models etc. And integrating detailed design models (e.g. UML, ER, BMPL/N, SOA) etc. and integrate data from other sources (e.g. CMDBs);
• business and technology alignment: relationships and dependencies between the business and technology;
• requirements (business/systems): at various levels of detail supporting an understanding of acceptance/contracts/terms and design/delivery;
• change over time: time based views of all of the above (as-is, to-be, transitional);
• change initiatives: programmes, projects (e.g. costs, risks, resources, timeframes) and business cases (e.g. benefits, fit to current environment etc.);

I wanted to ensure that knowledge in all areas (usually oriented at different audiences, for different purposes, but ripe with opportunities for reuse) could be inter-related i.e. so that information doesn't just reside in isolated silos (e.g. a plethora of office documents, or isolated models). I also wanted a focus on models that record explicitly the essential semantics (rather than just how pretty the pictures, are or how elegant the wording is, as often happens in free form documents and diagrams). This knowledge hub, with various interchange information mechanisms, would:

• manage the semantics: model any object, and relate it to any other object; for a predefined library of object we would have a predefined set of properties, methods and relationships to other objects; it would be easy to change the modelling language (objects, relationships, properties, methods, appearance, nesting etc.); and we would be able to structure the knowledge to support any framework (taxonomy e.g. Zachman, FEAR, TOGAF, eTOM, etc.) so these could be tailored to specific customers/project needs etc.)
• communicate: create visulations based on any sets of objects with the appearance, level of detail, layout, type information displayed, etc. changed to suit the audience/purpose; allow all authorised people to see the subset of the information they are interested in and annotate/revision-mark that information; allow publishing in any format e.g. office documents, images etc. and via customisable web interfaces (e.g. to support mashups etc.)
• integrate: with other sources so we could exchange data with modelling tools; databases/applications; systems tools (e.g. CMDBs); portfolio management tools; programme management tools; dedicated modellers (e.g. project plan, UML, BPM, Data/ER, SW architecture analysis tools); etc.
• easy to use and automatable: allow users to very quickly and efficient create models with very little knowledge of the tool (and very little training), to support workflows associated with maintenance and use of the knowledge.
• open and extensible: to be standards compliant and implementation technology and technical architecture agnostic. To be able extensible so it can be customised to exactly what we or a client believes is needed.

Eventually (in 2003) we found tooling that provided a client visual modelling capability and a simple server side repository. During the initial implementations it became apparent that to be successful a combination of visual modelling, forms (web forms/dashboards) would be required to make implementations effective for large organisations (and to overcome organisational impedance). It was also recognised that essentially what the solutions represented was effectively a data warehouse /BI solution for the CIO (and IT) and that typically this was the one area of the enterprise that was not served well by IT. So in addition to the above what was needed was:

• powerful analysis and reporting: to be able produce reports (using standard reporting tools) in whatever form (visual, textual, dashboards) and type (Word, PDF, web forms etc) is required and access these reports from portals and client tools so the knowledge can be visible to all stakeholders in a way to suits there specific needs, roles at each point of interaction.
• tool to ensure the quality and currency of information: so we could automate the collection of data (where there is another source of record e.g. ERP, CMDB, etc.), to maintain policies on data (that allow data quality to be monitored/assessed), to allow easily ad-hoc updates by many users (who can do so as part of their day to day job) and proactively survey/poll users (when it is believed the data is out of date).
• secure, scaleable and accessible: to support a large number of ad-hoc users (many that will only interact with the systems for a small % of the time but all of whom have an interest or are guardians of bits of the jigsaw puzzle.
• tailored to specific purposes: solutions (based on tailored metamodels, user interfaces, system interfaces) to support specific jobs e.g. enterprise strategy or architecture; business cases and investment plans; technical architecture e.g. solution/integration/service/data/storage; or managing requirements, metadata, design, acceptance, programmes of work, security, business continuity
• sophisticated management of multiple states: as in reality large complex enterprises have many initiatives operating each of which is expected to change the current state (at different points in time) and automate the way the changes will ripple through those states.

So in summary our current requirements could be summarised as:

• suited to business (and technology) orientation - so ideally one could start with the business side e.g. able to manage business drivers and requirements and solutions.
• flexible, extensible and customisable (language of modelling, user interfaces, data interfaces, datamarts, analysis and reporting) - so modelling can be done in terms/language suited to the business
• powerful visual analysis and representations (for large amounts of data, power users, and design)
• supporting a broad and diverse community of users (with suitable roles and security controls for those different groups of users)
• automated data collection
• communicating in various ways: diagrams, charting &  analytics, reporting etc.
• multiple states and transitions between states

Having resolved the tooling issues – we then focused back methods – because it was clear that many initiatives in this area failed because people did not have a well developed implementation methodology. See (EA implementation)

Contemporary technology platform design

(shameless plagiarised from this work on SOA - real time enterprise)

IT Platform strategy needs to be driven by an organization’s business priorities. It is these priorities that set the drivers from which an application, system and network infrastructure strategy can be created. The increasing importance of IT as the digital value chain of the business, positions IT executives and their organizations to be more strategic and critical than ever before. IT organizations must transform from a “lights-on” culture to a “strategic differentiation through IT” culture.

An IT platform should:

• combine decision support and just in time fulfillment with real time execution & information availability
• deliver services when needed- as needed based on the explicit service requirements of the business.
• afford organizations the ability to implement, new, unique products or differentiated capabilities in a timely manner for purposes of competitive advantage & operational control
• leverage and reuse common component services (both business & infrastructure) for productivity, efficiency and competitive advantage purposes.
• provide the necessary “plumbing” – rich user experience, dynamic execution environment, intelligent mediation platforms, real time data frameworks, optimal system footprints, dynamic network coordination, automated orchestration and tooling to enable autonomic management, monitoring & reporting
  

Some common questions driving a move to new platform are how to:

• become more flexible and responsive to the dynamic needs of the business
• simplify & reduce the complexity of the IT environment
• get more value out of project & operational IT spend – both systems & people
• reduce costs while delivering improved service
• eliminate dedicated silos of data, systems and infrastructure as they exist today
• reduce the time it takes to build and deploy new business services
• implement and sustain predictable qualities of service

A platform has to produce services that respond to the business in four key areas:

• business services – (information, automated logic, intelligent analysis)
• infrastructure services – (federated query, caching, execution, messaging)
• infrastructure resources – (storage, network, compute)
• systems performance management – (service execution that alleviates/minimize any compute, memory, I/O or bandwidth limitations and meets service levels)

This helps address datacenter management challenges of:

• Data Center optimisation: space, power, cooling
• Network bandwidth optimisation and management
• Enterprise systems management
• Dynamic infrastructure management (operational processes and technical skills)

Consequently the key attributes sought in the design of IT platform are:

Service orientation

• components and services are loosely coupled
• components and services are not locked to an implementation technology
• business logic and data is not be hard-wired to data stores or a technology/vendor

Real time infrastructure – a virtual, dynamic runtime environment

• service requests are via: message framework (publish and subscribe), grid service (scheduled, on-demand, adhoc), fabric service (application server container originated)
• services are dynamically allocated based on: service contract requirements of speed, throughput, load, calendar, wall clock, costs/margin rules etc.
• performance management (applications, network) and dynamic load-balancing and message brokering
• consumption and usage monitoring, logging (task, resource, user, transaction, job, etc.) and reporting (usage, service level compliance, trends, service allocation, efficiency, etc.)
• policy driven infrastructure provisioning & configuration management
• real time transaction workload management, transactional data caching and synchronization.
• meta-data repository and data transformation services, meta-rule repository with real rule evaluation.
  

Service oriented infrastructure utility – policy driven consumption and fulfillment

• ITIL/ITOM best practices
• Proactive capacity planning
• Multiple service fulfillment strategies
• Dynamic network routing

Service oriented product management – repeatable discipline that defines the products, services, policies and infrastructure for service oriented business.

• explicit management of the alignment of the business (strategies and operations) and IT - Service management that is suited to the granularity of services provided and consumed (service level management, service life cycles, service catalogues)
• systems integration architecture and team
• utility infrastructure model oriented at end to end optimization
• asset management (for reuse).

Communicating the big ideas to a broad community of people.

The value of large format diagrams on walls is underestimated as a way of communicating essential concepts to a large community of people.

For a millenia (e.g. frescos, murals etc.) have been used to communicate common concepts to communities where the individuals didn't have the ability (let alone the time) to read the texts that provided the details.

It could argued that icongraphy proved an effective way of ensuring the main christian churches remained united (with people all seeing the same big picture), and that the divisions arose when people started reading the texts (and focusing on details of interpretation).

The essential elememts of many complex designs (of all kinds) can often best be described diagramatically e.g. buildings (plans, elevations, sections, perspectives), cars, electrical schematics etc.

It is surprising that this method of communicating the design of businesses and IT systems is often under valued.

IT must be the only complex design discipline in world that tries to communicate complex plans, designs, roadmaps etc. in a A4/Letter sized pages. In some absurd homage to Word (or similar document formats). Or in the futile attempt to pretend that these kinds documents are suited to communicating complex models, plans and designs (when their real strength is narrative i.e. they are good at telling stories)

The problem in this context i.e. complex design and planning is that there is an almost infinite number of possible stories (for different audiences, different perspectives etc.) and the underlying parameters change (albeit incrementally) - so any narrative ends up be at best partial and usually misleadingly or inaccurate.

Analysis and modelling

Analysis modelling

Basically modelling consists of doing the analysis - and recording the results in a modelling tool of some kind, whereas analysis consists of doing the analysis and recording the results on paper or narrative document/presentation of some kind. If the analysis is done soundly it represents 90%+ of the effort or modelling, and modelling ensures that the analysis is done corrected, is recorded, and can be examined and updated.

Why then do people have the impression that doing analysis on paper is far faster than modelling? It is because they do half baked job of analysis and the method of presenting the analysis (narrative document/presentation) allows this. Often in fact it encourages partial analysis because the "analyst" has formed conclusions (based on prejudice e.g. what worked last time, what would suit them) and good analysis could undermine these conclusions.

There are a number of ways of doing and presenting analysis (e.g. business analysis, technology issues analysis etc.). Often analysis is done in the head and results presented using unstructured data e.g. Office documents (Word, Powerpoint etc.). When this is done there are a number of problems - the exact relationship between data and the conclusions is often not recorded explicitly and in detail; and this almost inevitably leads to short being taken (which exposes the a major weakness of analysis done in this way). This is find for the analysts (as the data/relationship - to the extent they are known - exist in their heads), it is not OK for the person for whom the analysis is done.

Analysis consists
- determining the metamodel and semantics (based on the what is being analysed)
- gathering the items data
- relating the items data
- recording the data/relationships
- reporting on the conclusions (based on an analysis of the data and how it is related)
- presenting the conclusions and usually the basis of them (data/relationships)

The data/relationships includes (facts, beliefs etc. and goals, preferences) and the conclusions are usually in the form of recommendations.

Modelling basically consists of exactly the same steps - but the principle difference is a modelling tool enforces the semantics (which goes a long way to validating the data/relationships) and makes it obvious when data/relationships are incomplete and or inconsistent.

This means that in practice often gathering and relating the data with rigour (i.e. so that it is accurate, explicit, weighted etc.) is more difficult than is anticipated. This is not an issue with modelling - it is an issue with doing analysis properly.


See also:
- Why paper based approaches don't work
- Why EA can't be done on paper

TRMs and change

Categorisation systems

As with any system of categorisation when the data to be categorised is known and/or static it is easier to develop a useful system of categorisation, and the systems of categorisation seldom needs to be changed. When the data to categorised is unknown of changing quickly it may be necessary to change the categoisation system i.e. based on changes in the nature of the data.

Technical Reference models (TRM) are often used to categorise an enterprises technologies or the of types of technologies that could be used by an enterprise (often along with product catalogues, which indicate which specific vendors products are being employed).

TRMs for Technology Visioning

TRMs are put to a number of uses one of which is technology visioning (where the TRM provides categories for fact, beliefs, implications, standards, patterns etc.). TRMs are also used for investment planning (when the things categorised are assets, products etc.).

When looking ay technology visioning i.e. looking at technology trends and directions based on new and emerging technology products and announcement almost by definition the data to be categorised is changing quickly and/or unknown. So in this area it is likely that a TRM will need to evolve.

In areas where technology is fairly static the TRM will probably be fairly useful (correct, accurate, well balanced) because the technology is fairly static it will not be the focusing of technology visioning i.e. trends, directions and changes.

In areas where the technology is changing quickly the TRM will often need to be reviewed as technology and business paradigms change i.e. new categories may be needed (e.g. with old categories splitting or merging).

To make this feasible one needs a way of changing the TRM but keeping the associations of elements (facts, technologies, assets, products) to the TRM.

5 things SOA vendors are missing, and 5 things customers need

(prompted by 5 things SOA Vendors are missing)

This item is refreshing direct and points out that SOA is an architecture (a complex distributed architecture, leveraging many of the traditional EA concepts) and or a meta an architectural pattern. What is needed are products that are elements in an SOA, not products that in and of themselves aim "to be" the SOA. It says to Vendors:
1. Make sure your product works.
2. Make sure you know what SOA is - most who sell SOA technology don't understand the first thing about SOA and typically play "buzzword bingo" reciting terms e.g. agility, reuse
3. Get wise about the approach to SOA - how should each customer approach SOA.
4. Don't sell yourself as "one stop SOA shopping." - in reality, nobody is a one-stop SOA shop.
5. Consider the future - Architectures are journeys, not projects. You need to think long term when you work with a customer and a customer's architecture inserting yourself at key points in the process

What Customer need is people who:
1. Know how to make the product works. Or at least can confirm they don't work, or they are not working as they are meant to. This usually comes from someone who has worked with a number of products of each class.
2. Know what SOA is - often these will be people whose goal is not to sell a particular technology and rather focus on what the business aims to achieve.
3. Are wise about the approach to SOA - and know how to engage with customers at many different stages (before they need a product set, when they do need and are selecting a product set, when they have a product set).
4. Know "one stop SOA shopping." - doesn't make sense, and at it worst ends up in the customer being captured by the vendor (which is most vendors' goal)
5. Can consider the future - will be around for the journey, and are able to think long term when looking at a customer's and a customer's architecture.

It is staggering to think that Customer's would look to the major product vendors and outsourcers for independent advice in this area.