Data Governance - Maturity Model

(synopsis of: 7 Stages for Effective Data Governance by Martha Dember)

There is a tendency in organizations to be complacent about data quality and integrity issues (which could compromise credibility of the organization's information).

Data Governance:

• is the development and integration of a set of rules/policies, guidelines, and standards - for managing data (not just a collection of ad-hoc data quality projects)
• provides the framework for IT and business to worktogether to establish confidence and credibility in the enterprise's information.
• is implemented by a data governance management team, of IT and business associates, unified by a common goal i.e. to ensure: data is what it is supposed to be (Data Quality); data is in the correct context (Data Integrity); data and its associated metadata are accessible (Data Usability)
• ensures the authority to manage data is properly delegated from the senior-most levels, and that parties are held accountable for executing governance policies as required by their respective mandates.
• has policies and procedures that balance effective information access with appropriate use of the information.
• is a programme (i.e. is not an application, that can be purchased, installed, and implemented with a specified end date) but a process that, over time, affects the culture and the way an organization conducts business.

Data Governance Maturity Model - requires a plan or framework (along with other things). In a data governance maturity model one can define seven stages of maturity .

1. Strategy and Framework
The Strategy identifies data issues, their causes and effects, and methods to solve the issues.
The Framework defines the roles/responsibilities of the data governance team and the relationships and dependencies between the data governance team and the data architecture.

2. Scenarios and Validation
Tests the data governance strategy and framework e.g. takes a data issue (reactively) and determines the cause/effect of the issue, and propose a solution to: refine the processes/framework; determine the communications (how best to implement the steps, and who should play roles).
Benefits: PoC using industry best practices that can be adapted to the organisation

Stage 3: Formalized Organization and Responsive Process Rollout
Formally define the roles (e.g. job descriptons)
Benefits: Accountability for establishing and maintaining data quality.

Stage 4: Proactive Process Rollout
Identify business events or activities that causes problems (data issues)
Benefits: Proactive processes improvements, better communication between business and IT (people can collaboratively manage data)

Stage 5: Expanded Business Involvement
Explicit buy-in from key stakeholders and executive management in the data governance program. Standards compliance monitoring is incorporated as a part of performance measurement; and data-specific technology, processes, and organizational components are aligned with the company's most important business objectives.
Benefits: Continuous improvement efforts are measured and monitored (metrics);Associated processes (e.g. SDLCs) can be improved; Data governance efficiency is improved (the knowledge base is used to reduce future project effort/duration)

Stage 6: Stewardship Culture
Governance across the Enterprise reconciles priorities, expedites conflict resolution, and builds cooperation in support of data quality as a common objective. Data quality education and awareness programs are an integral part of the on-going employee training programs.
Benefits: There is a common focus and delivery and everyone acts as a data steward (extending to business partners/channels)

Stage 7: Strategic Governance
Data governance and compliance becomes real-time, change-driven, on-demand process that continually assess risks, update policies, and manage resources across the enterprise. People, processes, and technology working together organically and autonomically that result in an effective data governance program.
Benefits: Utility of information can now drive flexibility and agility of the organization and result in a streamlined/efficient organization.

Thinking SOA (are the issues really that new?)

(prompted by thinking SOA)

I agree with the conclusions i.e.

- SOA is a core aspect of EA (and EAs that get this will get SOA more quickly)
- SOA does have new some challenges especially around lifecycles, service quality management and inter-party agreements - but think that in but in many cases the challenges have always been there - now it is just harder to bury your mistakes.

I disagree with some of the things implied i.e.
- Good EA has always been about interoperability (driven by business goals such as agility) - and never about layers of technology. So this is not something new with SOA.
- "Traditional enterprise architecture" is Technology Architecture (NOT EA). Many of frameworks, methods and tools - have been manifestly unsuited to real EA for a decade (hence the how few really success EA projects there are). So this is not something new with SOA either

See also: EA/SOA a shareholders issue?

EA and SOA shareholder issues?

(prompted by: Could Lack of SOA Drive Shareholder Lawsuits)

[WIP]
I can't really bring myself to agree with the conclusions in this item - that the absence of good EA's could drive lawsuits.

Key assertions from this:
1. Shareholders are looking at enterprise architecture efficiencies
2. Many major public companies don't have efficient enterprise architectures
3. Stockholers assume that IT/Architects, are doing their best to make the architecture optimal for the business. However, in many cases, that is not true.
4. Years of neglect, lack of talent and understanding, etc. have created dysfunctional EAs.
5. Bad EAs hindering corporations ability to make money and return value to shareholders
6. This could drive lawsuits where bad EAs may need to explained in depositions (during law suits).
7. SOA are not a fix for bad EAs - they will just make bad ones worse.
8. What is needed is a long term cogent SOA strategy and this is complex and hard, but worth it if you do it correctly.
9. Shareholders may ask for complete audit be done on the methods and practices in building an effective EA

Comments re the points above:
2, 3, 4, 5, 7, 8 - I think are fairly well known i.e. - Most major enterprise don't have efficient EAs and years of neglect, lack of talent/understanding are a cause. Few IT orgs are doing their best to fix this. Bad EAs do hinder ability to return value to shareholders and SOAs will make bad EAs worse (adding complexity) and what is needed is a long term cogent strategy. These strategies and architectures are complex and hard to do.

I think 1/6 - are unlikely. That shareholders will look at architectures and focus law suits on the deficiencies seems improbable to me for many reasons.

The last point 9 - may well make sense. They may ask for audits to be done on the methods and practices in building an effective EA But the challenge will be finding auditors with the knowledge and experience to do the auditing who are actually impartial e.g. most of the organisations that are recognised as leadings in EA seem to have unignorable conflicts of interest i.e. they are primarily vendors of systems or software; they generate huge incomes from the implementation ERP systems (often following services associated with "impartial" evaluation of ERP solutions); they don't really understand some of the issues associated with EA.

See also SOA's must be underpinned by better management

James Rogers Breakfast Presentation

For those of you who missed James' breakfast sessions in Sydney and Melbourne, a copy of his presentation is available. Please email us at info@rhe.com.au for a copy.

One of his main points is that EA, and EA tools like Troux, are increasingly being viewed as a key enabler of business transformation projects. Without an aggregated view of an organisation it is very difficult to make data based decisions that span possibly multiple lines of business and IT.

From the perspective of Enterprise Architects, who often struggle for budget and to prove their value to the organisation, attaching EA to Business Transformation initiatives is a good way to prove the value of the process and to incrementally build out a view of the enterprise architecture.

James also showed us some highlights from the soon to be released Troux 7 products.

Guy

EA can't be done with document or CASE tools

To be a success EA requires the correct tooling/technology. All reputable EA experts would recognise this (though some were slow coming to this fairly obvious conclusion).

What are some of the things a tool needs to do?

The correct selection of the technology requires a clear understanding of what is required. This includes:

• Communicating- to a broad set of stakeholders (not aligned to any specific technology, and most not aligned to any technology). This requires communication in many forms (diagrams, forms, reports, documents etc.) and using many media. This in turn requires a range of publication techniques.
• Accessible - if the information held in the EA is not accessible (e.g. via a Web portal) and searchable (i.e. based on the specific, and immediate, area of interest of the person looking for information).
• Analyzable - one must be able to analyse the information (this requires that concepts and relationships are explicitly recorded.
  
• Aggregating - the EA will contain data owned by many constituencies (people and systems) and it is necessary to be able to aggregate the information from the various sources. The systems include operational systems (e.g. CMDB), design-silos (e.g. ER models, UML models, BP models), and business people (with the plans, products etc.).
  
• Maintainable - as a natural part of day to day activity i.e. without requiring any substantial additional work by any of the data owners.
• Secure - reflecting who owns the data, and who can access it)
  
• Auditable -
  
• Consistent - so that if a change is made to a concept (e.g. a business goal, a technology component) it is reflected everywhere.
  
• Integrated and cohesive - so that all connections known are recorded.
  
• Explicit and precise - this requires that the semantics are able to be explicitly recorded i.e. the core concepts and the relationships between them (their types, properties, weighting, etc.).
  
• Business oriented - this requires that languages suitable for communication with non-technical people be used (e.g. not languages just used by design people UML, ER , BP etc.).
  
• Technically connected - this requires that the EA can represent the semantics that occur in the various technology design silos (UML, ER, BP, etc.) so that explicit connections can be made to things elaborated in related tools (e.g. CASE) and eventually leads to executable code.

What about people who say it can be done without tools?

If an EA professional advocates doing EA without suitable tooling - I would characterize it as unprofessional i.e. either they are ignorant (of what EA is, and what is required to achieve it) or they are being disingenuous (i.e. they know what is required, but don't see it in their interest to make it clear).

By way of comparison - if someone's profession was creating complex models in another domain e.g. large project plans, complex budgets, complex building design - they could not (in any of these domains) credibly advocate doing this just in say Powerpoint, Word or Visio (and at the same time claim to know a great about project planning, budgeting, complex building design).

What about an Office suite as a tool?

EA tools must have the detailed representation of information that BI requires, have the communication and ontological capabilities of knowledge management tools, be able to support the detailed semantics required for connection to dedicated design domains, and be able to graphically communicate complex designs.

Office suites fail for reasons such as:

• Communicating - Documents can't practically be produced for each and every stakeholder focusing on the small subset of the information they are interested in at any point in time, and presenting it in precisely the way they want to see it (diagram, forms, report, chart) etc.
• Analyzable - Documents (Word, Powerpoints etc.) can't be analysed (except by people)
• Aggregating - Documents don't lend themselves to aggregating data.
  
• Maintainable - no one can credibly suggest that all data held in documents is practically maintainable.
  
• Explicit and precise - Documents seldom explicitly record all the relationships within them and never record all the relationships across concepts held in sets of them.

The information from an EA should be able to be conveyed via documents (from Office suites). In this sense these documents (Word, Excel, Powerpoint, Visio etc.) represent reports. However the fact that reports are required does not mean that the reports themselves are the locally repository for the data.

What about CASE tools?

People' who start looking for EA solutions by asking about CASE capabilities - indicate that they have not understood the essential nature of the problem see:
http://ea-in-anz.blogspot.com/2007/08/business-modelling-with-domain-specific.html
http://ea-in-anz.blogspot.com/2007/08/uml-is-not-language-suited-to-most.html

Business and IT Transformation

The term "EA" has developed some unfortunate connotations e.g. it has been confused with "technology architecture", "systems architecture", "solution architecture", "software architecture", "business process modelling" etc.

This is reminds one a little of the famous strong of the "blind men describing the elephant".

If one uses the analogy of town planning - it is a bit like confusing town planning with: architecture, electrical installation, landscaping, interior design, plumbing etc. When asking for a tool selection to support townplanning - one is then asked if it has a electrical load modelling capability; hydrological modelling capability; a lighting model for examing interiors, or a CAD tool for drawing beams [see http://ea-in-anz.blogspot.com/2007/09/ea-and-analogies-with-built-environment.html]

In EA at present one is often asked if it the EA tool is a CASE tool (ER modeller) or process simulation tool - and what this illustrates is that the person asking the questions does not a good grasp of what EA actually is. [see http://ea-in-anz.blogspot.com/2007/09/ea-cant-be-done-with-document-or-case.html]

EA also is often seen as some ivory tower activity that at best produces lots of large complex documents (that always out of date, and that are of little value to anyone other the authors) - mainly because EA is done in the wrong way.
[see http://ea-in-anz.blogspot.com/2007/08/myth-of-heavy-weight-ea.html]

It may well be that the pragmatic way around this is to abandon the term, and the
focus on the end goal which is business and IT transformation.

UML is not a language suited to most aspects of EA

(prompted by discussions with EAs and "How UML is Used")

UML is often one of the first words that comes out of IT people's mouths when one discusses EA (in a bizarre attempt at logocide one organisation evens call its UML modelling tools Enterprise architect).

UML may be very useful, but not for most aspects of EA. This is because most of EA has little to do with OO modelling oriented at OO development (or SW development and per se) and because UML is just not effective for communicate with most of the business (i.e. not technical people).

The analysis that needs to be done in EA is not really related to: the different states that objects can exist in, the messages that objects used to communicate, or low level technical "use cases". Obviously an EA needs to consider scenarios of use (how things are used in different situations by different roles or systems). And one could use the very oddly named "use case" for this, however many other mechanisms are available usually better suited to use in EA. In my experience the quickest way to alienate most business people is to start of by using abstract terms like "actor", or talk to them Swedlish. Most people in the business prefer natural business concepts like: business process, business role or organization, etc.

That people attempt to apply UML to enterprising architecture probably indicates the mistaken belief that EA is somehow an extension of software development - and goes a long way to explaining why some many EA initiatives fail.

There are various aspects of detailed SW design oriented analysis that we may wish to relate to from an enterprise architecture or we may wish bring together for analysis the disparate views of various development silos e.g.
- process orchestration development views which use BPEL that may relate to BP models
- object development views which may relate to UML models
- data development views which may relate to ER models
etc.
This is more in the solution architecture rather than enterprise space. What an EA can do is bring together these disparate silos. For example we could consider the relationship between a business process or a business role/organization, and a business application (and elaborate that relationship using UML - if it really added value). But for most people getting a canonical list of business processes or business applications would be a great starting point - rather than elaborate great detail one interaction

We could also consider the broader issue of the utility of UML in capturing requirements per se. UML is also often also one of the first words that comes out of IT people's mouths when one discusses business requirements.  This is mostly the case when IT people with a SW engineering background are involved. Or when executives want to show they an "in" with technical people by using their arcane terminology,

UML not useful for most, if any, aspects of business requirements management. Business requirements may result in software development (the majority will not). Of those that do require development only a subset will require OO (e.g. others may require BI). Often when development is required it will be outsourced or done under contract. What the business needs is boundary of responsibility that allows those solutions elements delivery to be assessed.

By way of analogy. If one has transport requirements - one doesn't to be asked what use case of a brake pedal is. One wants either a transport solution (streets, tracks and trains, roads and vehicles, ships and ports), or a services (e.g. buses, taxis), or possibly an in-house asset car, a bike, a plane. In the rare circumstances where a car is engineered for a client - it would be a very brave (or foolish) client indeed who would try to specify it, or its components, using Use Case, Sequence diagrams.

In "How UML is Used" by Brian Dobing and Jeffrey Parsons - it seems reasonable to assume that as the data references in this article is gathered from UML practitioners/clients they are not unreasonably prejudiced against UML. What bemuses me is that the authors - despite the facts in front of them - doesn't seem to just say "UML is not great for client facing requirements capture" or for communicating with non-technical audiences. This doesn't mean UML may not be great for technical communications e.g. between designers and developers.

What can we learn about "How UML is Used" regarding requirements capture?
- UML may be too complex for clients (no kidding) and those who are not technical (i.e. who don't want to participate in development).
- UC are not thought that good for communicating with clients (business people)
- Clients (business people) don't want to review the artifacts beyond UC narratives (and why would they, when these are really development/developer oriented models)?
- UC diagrams are rated as least useful in providing additional information to developers.
- Despite the above developers, undeterred, seem to believe that UML diagrams can be understood by clients?

Contrary to claims in the popular literature, developers appear to believe that UML diagrams can be understood by clients. 87% of respondents rated UC Narratives as useful for "verifying and validating requirements with client representatives on the project team.", but when asked whether the UML facilitated communication with clients, 55% said it was at best "Moderately Successful."

Domain specific business languages (or Enterprise specific languages) may be a better answer. To instantiate these dynamically so they are available for modelling, one needs a metamodeller.

EA requires an ability to communicate on many subjects with a diverse range of stakeholders. Selecting an arcane language oriented at object oriented analysis is unsuited to the task i.e. both the semantics and the notation are a poor fit for the vast of the information used to describe an enterprise.

If UML struggles to capture the requirements for project in which development undertaken (i.e. to act an effective mode of communication) - surely people can see that if the task is describing an enterprise (where development is not the major goal i.e. it is a necessary evil) - it is unlikely to be a good fit.

Of course long before this Anthony J H Simons, Ian Graham wrote about: 30 THINGS THAT GO WRONG IN OBJECT MODELLING WITH UML 1.3. In this the authors catalogue problems experienced by developers, using various object modelling techniques brought into prominence by the widespread adoption of UML. Developers still seem to create inordinate problems for themselves by pursuing unproductive development strategies that are apparently fostered by UML. This article shows how the biggest problem by far is cognitive misdirection, or the apparent ease with which the rush to build UML models may distract the developer from important perspectives on a system. This problem is more serious than the outstanding inconsistencies and ambiguities which still exist in UML. While UML itself is mostly neutral with respect to good or bad designs, their are consequences of allowing UML to drive the development process.

The cognitive misdirection problem is more serious when UML is applied in domains other than OO design. In these case the pursuit of unproductive strategies oriented around UML can only consider "the triumph of hope over experience" (Samuel Johnson).

See www.semanticprecision.com for a better solution.

The myth of "heavy weight" EA

There is a common myth that maintaining an effective EA introduces a lot of cost to the Enterprise. An EA implemented in the right way reduces the level of effort and cost in an enterprise overall.

An effective EA needs to be supported by the right tools, and the right changes in associated processes (each of these processes in turn becomes more efficient and accurate) and be comprehensive in the set of stakeholders it supports.

A "Light weight" EA - where a superficial and partial set of information is assembled in the traditional way (i.e. without the right tooling, and without changing the touch point processes) may suit "light weight thinkers" - but as they fail to deliver substantial value and they add additional cost - they don't make any sense.

Establishing the changes in an enterprise that will allow an effective and complete EA to develop (and establishing the tooling, and changing associated processes) does have a cost but this cost is comparatively small if led from the right level within the enterprise, undertaken over a period of time (i.e. most cultural change is hard to effect quickly).

By way of one imperfect analogy - books and libraries
Assume your current strategy for managing books (i.e. knowledge - putting aside for now the fact that documents - especially in narrative form seldom make the semantics explicit, or relate knowledge in one book explicitly and atomically to knowledge in another book) is to leave the books scattered all over every office (and this is the form i.e. documents, and is the strategy essentially advocated at present; for most the design/governance artefacts associated with an enterprise).
Most people would find it quite difficult to find the correct information on most subjects, and if they wanted to record new information about subjects so that others could find it they would not know where to put it.

Say a proposed strategy is to establish a library function e.g. a classification system (Cf Zachman's framework), some places (physical or virtually) and some simple processes (that you would like people to follow). Now the actual cost of finding knowledge on any subject (i.e. what is said by a range of authors, with different perspectives) is reduced, and knowledge on subjects can know accrete in a natural way. Yes there is a set up cost.

A problem with this analogy is that most of the documents created in an enterprise are created by their authors for a very small and select audience (which in itself is odd), whereas most authors of books would ideally like their readership to be as broad as possible.

Another imperfect analogy - maps and the environment (built and natural)
If your current strategy for managing information about the environment (from rivers and mountains, to buildings and reticulation systems) is to leave documents scattered all over a wide range of offices/enterprise (and this was the form that the various stakeholders used until very recently). And a proprosed strategy is to establish a mapping function e.g. a classification system (a co-ordinate system, or a map projection), a collation mechanism and processes. The actual cost of finding out about any areas is reduced, and knowledge (on all areas) can accrete in a natural way (i.e. as the plans on buildings, cables, new information on soil types, etc. are deposited). Yes there is a set up cost.

No new information needs to be created - and in fact the effort associated with creating meaningful new information is reduced (e.g. it is easier to design changes if I can see clearly what already exists - buildings, cables, geological structures etc.) at the start of the exercise - rather than as I start construction (which is the ICT industry's approach).

Many artefacts are created by a function or unit for a very specific purpose and for a very small and select audience. The artefacts are usually focused on a narrow task (usually related to some delivery and/or construction activity), and specific timeframe (e.g. usually the immediate future). They are not oriented at: the ongoing maintenance, the effect on others, their discovery in future, the eventual replacement of what they is being delivering etc.). Governance (town planning functions) is slowly overcoming the impediments that the various engineering and cartographic specialists put in the way of this knowledge base by organising and sharing that knowledge.

Justifying architecture and strategy

There are always challenges in valuing, and therefore justifying, good (or often any) design, strategies, plans. The problem lies in a number of areas e.g.:

• Challenges with counterfactual analysis: in most situations (enterprises, buildings, cities, endeavours), one can't easily compare the result of the designs/strategies/plans that was actually taken - with other hypothetical designs/strategies/plans that could have been taken. People either need to intuitively value sound well reasoned designs, strategies, plans - or not i.e. the ROI is either obvious or it is not. So there is no point of comparison.
• Self-interest: This can militate against sharing knowledge. Knowledge is power. This can be seen in many professions to a greater or lessor extent. If ones shares and/or make available the knowledge one has (or the basis of a decision), one may diminish ones value (as a holder of knowledge), and become open to criticism (e.g. regarding the veracity of the data or the soundness of the analysis).
• Secrecy: Sometimes one does not wish to disclose a strategy/plan. Often the downside of secrecy is that people in ones own team or side, don't understand what should be done, why, when etc. (i.e. the benefit of secrecy is outweighed by its impact on efficacy).
• Thinking is intrinsically difficult: These things (strategy/design/planning, and modelling) are often quite difficult to do well i.e. they require quite a lot of thought. To make matters worse the result can be seen, with hindsight, as trivial and not justifying the effort required.
• Thinking takes time: It is always quicker just to act
• Thinking costs money: As results may be seen as trivial e.g (F = M x A) and could not be worth justifying the effort.
• Future is someone else's problem: That what is constructed is a poor fit, doesn't last, is expensive to operate, expensive to change or adapt is often not the problem of the initiator, the initial designer, or the builder (and perhaps the initial user).

These issues can be seen all areas of life e.g. social planning, cities, buildings, cars, and of course businesses and Information and Communication Technology (ICT). ICT is particularly bad as it has not yet matured into a profession (where people have the requisite learning, training and discipline and profess a code of ethics).

People wishing to ensure better strategies, designs and plans, need to look for others who:

• Intrinsically value better strategies, designs and plans, and can consider the future.
• Can overcome self-interest of knowledge holders and know what must be kept secret
• Know thinking is difficult, takes time and money and are prepared for the effort.

Michael.