What we sought in a tool for business/technology transformation and EA

(based on many people asking what the requirements are for a tool)

People often ask me what lead us to working with the tool sets we use for enterprise architecture and business transformation (EA/BT). The following is a short background.

Obviously the overall objective is to improve an organisation's effectiveness (reduce costs, improve income/outputs, ensure regulatory compliance, build asset value etc.). This requires managed change. I had focused how to improve the quality of transitions from RHE's inception (the best people, methods and tooling). We had tried using a wide range of OTS tools (e.g. CASE, Ontology, Document management and of course Office suites) and building tools to allow us to manage this information. After many years of trying to determine how EA/BT work could be done (in a way produce sustainable value for clients) and a long term focus on methods (including frameworks) we determined the tooling was inadequate (and the tool limitations fundamentally affected the method).

In 2002 we reassessed what tools were available that could be used to support:

• business strategies: drivers, plans, markets, products/services, locations, organisations, resources, change/transitions, projects/initiatives;
• business operations: communciations, services, processes, rules, information, organisation etc;
• technology architecture: enterprise, solutions, components/systems, etc. Supporting any: styles, topology pattern, interaction models etc. And integrating detailed design models (e.g. UML, ER, BMPL/N, SOA) etc. and integrate data from other sources (e.g. CMDBs);
• business and technology alignment: relationships and dependencies between the business and technology;
• requirements (business/systems): at various levels of detail supporting an understanding of acceptance/contracts/terms and design/delivery;
• change over time: time based views of all of the above (as-is, to-be, transitional);
• change initiatives: programmes, projects (e.g. costs, risks, resources, timeframes) and business cases (e.g. benefits, fit to current environment etc.);

I wanted to ensure that knowledge in all areas (usually oriented at different audiences, for different purposes, but ripe with opportunities for reuse) could be inter-related i.e. so that information doesn't just reside in isolated silos (e.g. a plethora of office documents, or isolated models). I also wanted a focus on models that record explicitly the essential semantics (rather than just how pretty the pictures, are or how elegant the wording is, as often happens in free form documents and diagrams). This knowledge hub, with various interchange information mechanisms, would:

• manage the semantics: model any object, and relate it to any other object; for a predefined library of object we would have a predefined set of properties, methods and relationships to other objects; it would be easy to change the modelling language (objects, relationships, properties, methods, appearance, nesting etc.); and we would be able to structure the knowledge to support any framework (taxonomy e.g. Zachman, FEAR, TOGAF, eTOM, etc.) so these could be tailored to specific customers/project needs etc.)
• communicate: create visulations based on any sets of objects with the appearance, level of detail, layout, type information displayed, etc. changed to suit the audience/purpose; allow all authorised people to see the subset of the information they are interested in and annotate/revision-mark that information; allow publishing in any format e.g. office documents, images etc. and via customisable web interfaces (e.g. to support mashups etc.)
• integrate: with other sources so we could exchange data with modelling tools; databases/applications; systems tools (e.g. CMDBs); portfolio management tools; programme management tools; dedicated modellers (e.g. project plan, UML, BPM, Data/ER, SW architecture analysis tools); etc.
• easy to use and automatable: allow users to very quickly and efficient create models with very little knowledge of the tool (and very little training), to support workflows associated with maintenance and use of the knowledge.
• open and extensible: to be standards compliant and implementation technology and technical architecture agnostic. To be able extensible so it can be customised to exactly what we or a client believes is needed.

Eventually (in 2003) we found tooling that provided a client visual modelling capability and a simple server side repository. During the initial implementations it became apparent that to be successful a combination of visual modelling, forms (web forms/dashboards) would be required to make implementations effective for large organisations (and to overcome organisational impedance). It was also recognised that essentially what the solutions represented was effectively a data warehouse /BI solution for the CIO (and IT) and that typically this was the one area of the enterprise that was not served well by IT. So in addition to the above what was needed was:

• powerful analysis and reporting: to be able produce reports (using standard reporting tools) in whatever form (visual, textual, dashboards) and type (Word, PDF, web forms etc) is required and access these reports from portals and client tools so the knowledge can be visible to all stakeholders in a way to suits there specific needs, roles at each point of interaction.
• tool to ensure the quality and currency of information: so we could automate the collection of data (where there is another source of record e.g. ERP, CMDB, etc.), to maintain policies on data (that allow data quality to be monitored/assessed), to allow easily ad-hoc updates by many users (who can do so as part of their day to day job) and proactively survey/poll users (when it is believed the data is out of date).
• secure, scaleable and accessible: to support a large number of ad-hoc users (many that will only interact with the systems for a small % of the time but all of whom have an interest or are guardians of bits of the jigsaw puzzle.
• tailored to specific purposes: solutions (based on tailored metamodels, user interfaces, system interfaces) to support specific jobs e.g. enterprise strategy or architecture; business cases and investment plans; technical architecture e.g. solution/integration/service/data/storage; or managing requirements, metadata, design, acceptance, programmes of work, security, business continuity
• sophisticated management of multiple states: as in reality large complex enterprises have many initiatives operating each of which is expected to change the current state (at different points in time) and automate the way the changes will ripple through those states.

So in summary our current requirements could be summarised as:

• suited to business (and technology) orientation - so ideally one could start with the business side e.g. able to manage business drivers and requirements and solutions.
• flexible, extensible and customisable (language of modelling, user interfaces, data interfaces, datamarts, analysis and reporting) - so modelling can be done in terms/language suited to the business
• powerful visual analysis and representations (for large amounts of data, power users, and design)
• supporting a broad and diverse community of users (with suitable roles and security controls for those different groups of users)
• automated data collection
• communicating in various ways: diagrams, charting &  analytics, reporting etc.
• multiple states and transitions between states

Having resolved the tooling issues – we then focused back methods – because it was clear that many initiatives in this area failed because people did not have a well developed implementation methodology. See (EA implementation)

Contemporary technology platform design

(shameless plagiarised from this work on SOA - real time enterprise)

IT Platform strategy needs to be driven by an organization’s business priorities. It is these priorities that set the drivers from which an application, system and network infrastructure strategy can be created. The increasing importance of IT as the digital value chain of the business, positions IT executives and their organizations to be more strategic and critical than ever before. IT organizations must transform from a “lights-on” culture to a “strategic differentiation through IT” culture.

An IT platform should:

• combine decision support and just in time fulfillment with real time execution & information availability
• deliver services when needed- as needed based on the explicit service requirements of the business.
• afford organizations the ability to implement, new, unique products or differentiated capabilities in a timely manner for purposes of competitive advantage & operational control
• leverage and reuse common component services (both business & infrastructure) for productivity, efficiency and competitive advantage purposes.
• provide the necessary “plumbing” – rich user experience, dynamic execution environment, intelligent mediation platforms, real time data frameworks, optimal system footprints, dynamic network coordination, automated orchestration and tooling to enable autonomic management, monitoring & reporting
  

Some common questions driving a move to new platform are how to:

• become more flexible and responsive to the dynamic needs of the business
• simplify & reduce the complexity of the IT environment
• get more value out of project & operational IT spend – both systems & people
• reduce costs while delivering improved service
• eliminate dedicated silos of data, systems and infrastructure as they exist today
• reduce the time it takes to build and deploy new business services
• implement and sustain predictable qualities of service

A platform has to produce services that respond to the business in four key areas:

• business services – (information, automated logic, intelligent analysis)
• infrastructure services – (federated query, caching, execution, messaging)
• infrastructure resources – (storage, network, compute)
• systems performance management – (service execution that alleviates/minimize any compute, memory, I/O or bandwidth limitations and meets service levels)

This helps address datacenter management challenges of:

• Data Center optimisation: space, power, cooling
• Network bandwidth optimisation and management
• Enterprise systems management
• Dynamic infrastructure management (operational processes and technical skills)

Consequently the key attributes sought in the design of IT platform are:

Service orientation

• components and services are loosely coupled
• components and services are not locked to an implementation technology
• business logic and data is not be hard-wired to data stores or a technology/vendor

Real time infrastructure – a virtual, dynamic runtime environment

• service requests are via: message framework (publish and subscribe), grid service (scheduled, on-demand, adhoc), fabric service (application server container originated)
• services are dynamically allocated based on: service contract requirements of speed, throughput, load, calendar, wall clock, costs/margin rules etc.
• performance management (applications, network) and dynamic load-balancing and message brokering
• consumption and usage monitoring, logging (task, resource, user, transaction, job, etc.) and reporting (usage, service level compliance, trends, service allocation, efficiency, etc.)
• policy driven infrastructure provisioning & configuration management
• real time transaction workload management, transactional data caching and synchronization.
• meta-data repository and data transformation services, meta-rule repository with real rule evaluation.
  

Service oriented infrastructure utility – policy driven consumption and fulfillment

• ITIL/ITOM best practices
• Proactive capacity planning
• Multiple service fulfillment strategies
• Dynamic network routing

Service oriented product management – repeatable discipline that defines the products, services, policies and infrastructure for service oriented business.

• explicit management of the alignment of the business (strategies and operations) and IT - Service management that is suited to the granularity of services provided and consumed (service level management, service life cycles, service catalogues)
• systems integration architecture and team
• utility infrastructure model oriented at end to end optimization
• asset management (for reuse).

Communicating the big ideas to a broad community of people.

The value of large format diagrams on walls is underestimated as a way of communicating essential concepts to a large community of people.

For a millenia (e.g. frescos, murals etc.) have been used to communicate common concepts to communities where the individuals didn't have the ability (let alone the time) to read the texts that provided the details.

It could argued that icongraphy proved an effective way of ensuring the main christian churches remained united (with people all seeing the same big picture), and that the divisions arose when people started reading the texts (and focusing on details of interpretation).

The essential elememts of many complex designs (of all kinds) can often best be described diagramatically e.g. buildings (plans, elevations, sections, perspectives), cars, electrical schematics etc.

It is surprising that this method of communicating the design of businesses and IT systems is often under valued.

IT must be the only complex design discipline in world that tries to communicate complex plans, designs, roadmaps etc. in a A4/Letter sized pages. In some absurd homage to Word (or similar document formats). Or in the futile attempt to pretend that these kinds documents are suited to communicating complex models, plans and designs (when their real strength is narrative i.e. they are good at telling stories)

The problem in this context i.e. complex design and planning is that there is an almost infinite number of possible stories (for different audiences, different perspectives etc.) and the underlying parameters change (albeit incrementally) - so any narrative ends up be at best partial and usually misleadingly or inaccurate.

Analysis and modelling

Analysis modelling

Basically modelling consists of doing the analysis - and recording the results in a modelling tool of some kind, whereas analysis consists of doing the analysis and recording the results on paper or narrative document/presentation of some kind. If the analysis is done soundly it represents 90%+ of the effort or modelling, and modelling ensures that the analysis is done corrected, is recorded, and can be examined and updated.

Why then do people have the impression that doing analysis on paper is far faster than modelling? It is because they do half baked job of analysis and the method of presenting the analysis (narrative document/presentation) allows this. Often in fact it encourages partial analysis because the "analyst" has formed conclusions (based on prejudice e.g. what worked last time, what would suit them) and good analysis could undermine these conclusions.

There are a number of ways of doing and presenting analysis (e.g. business analysis, technology issues analysis etc.). Often analysis is done in the head and results presented using unstructured data e.g. Office documents (Word, Powerpoint etc.). When this is done there are a number of problems - the exact relationship between data and the conclusions is often not recorded explicitly and in detail; and this almost inevitably leads to short being taken (which exposes the a major weakness of analysis done in this way). This is find for the analysts (as the data/relationship - to the extent they are known - exist in their heads), it is not OK for the person for whom the analysis is done.

Analysis consists
- determining the metamodel and semantics (based on the what is being analysed)
- gathering the items data
- relating the items data
- recording the data/relationships
- reporting on the conclusions (based on an analysis of the data and how it is related)
- presenting the conclusions and usually the basis of them (data/relationships)

The data/relationships includes (facts, beliefs etc. and goals, preferences) and the conclusions are usually in the form of recommendations.

Modelling basically consists of exactly the same steps - but the principle difference is a modelling tool enforces the semantics (which goes a long way to validating the data/relationships) and makes it obvious when data/relationships are incomplete and or inconsistent.

This means that in practice often gathering and relating the data with rigour (i.e. so that it is accurate, explicit, weighted etc.) is more difficult than is anticipated. This is not an issue with modelling - it is an issue with doing analysis properly.


See also:
- Why paper based approaches don't work
- Why EA can't be done on paper

TRMs and change

Categorisation systems

As with any system of categorisation when the data to be categorised is known and/or static it is easier to develop a useful system of categorisation, and the systems of categorisation seldom needs to be changed. When the data to categorised is unknown of changing quickly it may be necessary to change the categoisation system i.e. based on changes in the nature of the data.

Technical Reference models (TRM) are often used to categorise an enterprises technologies or the of types of technologies that could be used by an enterprise (often along with product catalogues, which indicate which specific vendors products are being employed).

TRMs for Technology Visioning

TRMs are put to a number of uses one of which is technology visioning (where the TRM provides categories for fact, beliefs, implications, standards, patterns etc.). TRMs are also used for investment planning (when the things categorised are assets, products etc.).

When looking ay technology visioning i.e. looking at technology trends and directions based on new and emerging technology products and announcement almost by definition the data to be categorised is changing quickly and/or unknown. So in this area it is likely that a TRM will need to evolve.

In areas where technology is fairly static the TRM will probably be fairly useful (correct, accurate, well balanced) because the technology is fairly static it will not be the focusing of technology visioning i.e. trends, directions and changes.

In areas where the technology is changing quickly the TRM will often need to be reviewed as technology and business paradigms change i.e. new categories may be needed (e.g. with old categories splitting or merging).

To make this feasible one needs a way of changing the TRM but keeping the associations of elements (facts, technologies, assets, products) to the TRM.

5 things SOA vendors are missing, and 5 things customers need

(prompted by 5 things SOA Vendors are missing)

This item is refreshing direct and points out that SOA is an architecture (a complex distributed architecture, leveraging many of the traditional EA concepts) and or a meta an architectural pattern. What is needed are products that are elements in an SOA, not products that in and of themselves aim "to be" the SOA. It says to Vendors:
1. Make sure your product works.
2. Make sure you know what SOA is - most who sell SOA technology don't understand the first thing about SOA and typically play "buzzword bingo" reciting terms e.g. agility, reuse
3. Get wise about the approach to SOA - how should each customer approach SOA.
4. Don't sell yourself as "one stop SOA shopping." - in reality, nobody is a one-stop SOA shop.
5. Consider the future - Architectures are journeys, not projects. You need to think long term when you work with a customer and a customer's architecture inserting yourself at key points in the process

What Customer need is people who:
1. Know how to make the product works. Or at least can confirm they don't work, or they are not working as they are meant to. This usually comes from someone who has worked with a number of products of each class.
2. Know what SOA is - often these will be people whose goal is not to sell a particular technology and rather focus on what the business aims to achieve.
3. Are wise about the approach to SOA - and know how to engage with customers at many different stages (before they need a product set, when they do need and are selecting a product set, when they have a product set).
4. Know "one stop SOA shopping." - doesn't make sense, and at it worst ends up in the customer being captured by the vendor (which is most vendors' goal)
5. Can consider the future - will be around for the journey, and are able to think long term when looking at a customer's and a customer's architecture.

It is staggering to think that Customer's would look to the major product vendors and outsourcers for independent advice in this area.

Linking SW Architectures with Enterprise Architectures

(prompted by an interface between Lattix and Troux)

In an Enterprise Architecture (e.g. Troux) an organisation would record its applications (hundreds to thousands). The level of detail they would record about applications would vary. Often not much detail is available (i.e. to hand) regarding the internal aspects of the applications (i.e. its internal modules/components and how these inter-relate).

Application portfolio optimisation work is typically focus on rationalising the set of applications at a macro level (e.g. often duplication of function arises as a result of mergers).

A next level of analysis would involve looking at the modules (or components that application is constructed from). This may present opportunities for more atomic refactoring of the application portfolio and will insights into candidates for service representation.

When an application is analysed using Lattix (SW architecture analysis) - we can use the information provided by Lattix (based on an examination of the actual code) to augment the information in the Enterprise Architecture (i.e. add the modules).

See also Enhancing EA