Do you need Data Governance over a Data Lake?

There continues to be a lot of excitement about data lakes and the possibilities that they offer, particularly about with analytics, data visualizations, AI and machine learning. As such, I’m increasingly being asked whether you really need Data Governance over a data lake.  After all, a data lake is a centralised repository that allows you to store all your structured and unstructured data on a scalable basis.

Unlike a data warehouse, in a data lake you can store your data as-is without having to structure it first.  This has resulted in many organisations “dumping” lots of data into data lakes in an uncontrolled and thoughtless manner.  The result is what many people are calling “Data Swamps” which have not provided the amazing insights they hoped for.

So the simple answer to the question is yes – you do need Data Governance over data lakes to prevent them from becoming data swamps that users don’t access because they don’t know what data is there, they can’t find it, or they just don’t trust it.  If you have Data Governance in place over your data lake, then you and your users can be confident that it contains clean data which can found and used appropriately.

But I don’t expect you to just take my word for it; let’s have a look at some of the reasons why you want to implement Data Governance on data being ingested into your data lake:

Data Owners Are Agreed

Data Owners should be approving whether the data they own is appropriate to be loaded to the Data Lake e.g. is it sensitive data, should it be anonymised before loading?

In addition, users of the data lake need to know who to contact if they have any questions about the data and what it can or can’t be used for.

Data Definitions

Whilst data definitions are desirable in all situations, they are even more necessary for data lakes.  In the absence of definitions, users of data in more structured databases can use the context of that data to glean some idea of what the data may be.  As a data lake is by its nature unstructured, there is no such context.

A lack of data definitions means that users may not be able to find or understand the data, or alternatively use the wrong data for their analysis.  A data lake could provide a ready source of data, but a lack of understanding about it means that it can not be used quickly and easily. This means that opportunities are missed and use of the data lake ends up confined to a small number of expert users.

Data Quality Standards

Data Quality Standards enable you to monitor and report on the quality of the data held in the data lake.  While you do not always need perfect data when analysing high volumes, users do need to be aware of the quality of the data. Without standards (and the ability to monitor against them) it will be impossible for users to know whether the data is good enough for their analysis.

Data Cleansing

Any data cleansing done in an automated manner inside the data lake needs to be agreed with Data Owners and Data Consumers. This is to ensure that all such actions undertaken comply with the definition and standards and that it does not cause the data to be unusable for certain analysis purposes— e.g. defaulting missing date of births to an agreed date could skew an analysis that involved looking at the ages of customers.

Data Quality Issue Resolution

While there may be some cases where automated data cleansing inside the data lake may be appropriate, all identified data quality issues in the data lake should be managed through the existing process to ensure that the most appropriate solution is agreed by the Data Owner and the Data Consumers.

Data Lineage

Having data flows documented is always valuable, but in order to meet certain regulatory requirements, (including EU GDPR) organisations need to prove that they know where data is and how it flows throughout their company.

One of the key data governance deliverables are data lineage diagrams. Critical or sensitive data being ingested into the data lake should be documented on data flow diagrams.  This will add to the understanding of the Data Consumers by highlighting the source of that data.  Such documentation also helps prevent duplicate data being loaded into the data lake in the future.

I hope I have convinced you that if you want a data lake to support your business decisions, then Data Governance is absolutely critical.  Albeit that it may not need to be as granular as the definitions and documentation that you would put in place for a data warehouse, it is needed to ensure that you create and maintain a data lake and not a data swamp!

Ingesting data into data lakes without first understanding that data, is just one of many data governance mistakes that are often made. You can find out the most common mistakes and, more importantly, how to avoid them by downloading my free report here.

How Long Will My Data Governance Initiative Take?

In this blog, I want to answer a question that I am asked several times every week. To be honest, it’s not an unreasonable question, but it’s not an easy one to answer!

Before I go into any detail trying to answer the question, I want to make one thing very clear: there is no end date on Data Governance.

Data Governance should be something that you are implementing and embedding within your organisation, so that it becomes part of business as usual. For this reason, as anyone who has worked with me or attended my training courses will know, I make a point of impressing upon everyone that Data Governance is NOT a project. If you truly embed Data Governance into your organisation it should never end.

However, having said that, it is entirely possible that you may want to do a project (or project-like initiative) in order to design and implement a Data Governance Framework in the first place. So perhaps the question should be “how long will it take to design and implement a data governance framework and start delivering some benefits?”

But to be honest, that questions isn’t any easier to answer and you could say that both are “how long is a piece of string” questions. Last year, I was lucky enough to be on a panel debate at Data2020 in Stockholm with David Dadoun from Aldo and Andrew Joss from Informatica. Whenever I participate in a panel debate, I always start with a sense of trepidation as to whether my fellow panelists will have the same views as me or not. In this case I did not have to worry because both David and Andrew were very experienced in Data Governance and had seen many of the same challenges that I had over the years. This meant that we all agreed that there is no such thing as a standard Data Governance Framework or a standard approach to implement it. It also meant that— much to the frustration of the Chairman— we took it in turns to answer many of the questions with “it depends.” The panel debate was filmed and you can watch it here if you’re interested.

The reason I tell you this is that whenever I am asked this question, I am always tempted to respond with “it depends.” However, this would not be useful for the person asking the question, so instead, I have to follow up with some supplementary questions. These will include things like:

• Do you have an agreement to commence a Data governance initiative?

• How many resources have you got to work on the initiative?

• What is the scope of your initiative?

• How big is your organisation?

• How open to change is your organisation?

And depending on the answers to the above, I may well ask “is your organisation ready for Data Governance?” Please note this final question is not the same as “does your organisation need data governance?”

Back in 2014, the Data Governance guru Gwen Thomas (founder of the Data Governance Institute) wrote a fantastic article called “When You’re Not Ready for Data Governance.” I frequently direct people to have a look at this post to help get their head around whether now really is the right time for them to commence Data Governance, because sometimes you just have to accept that now is not the right time.

So having asked the first round of supplementary questions (detailed above), if I am convinced that an organisation is ready and able to commence designing and implementing Data Governance, then I need to answer further questions. These are around what they are aiming for and where they are starting from. To help answer these questions, a lot of companies turn to a data governance maturity assessment of some kind. These are very valuable tools in helping an organisation decide how mature they need to be, and in identifying where they currently are.

Please be aware that sometimes organisations can get tied up in “analysis paralysis” and spend inordinate amounts of time and effort on completing a maturity assessment. This is not useful, and care should be taken to only go to the level of detail needed to understand what capabilities your company is hoping to attain, plus identifying its current state.

There are multiple different maturity assessments available. As with all things Data Governance  I prefer a simple approach and you can download a very quick and easy Data Governance Health check questionnaire for free here. If a more detailed assessment suits the culture of your organisation better, I recommend you look at the freely available maturity assessment published by Stanford University. Sadly they recently removed their assessment from their website, but Alex Leigh has created an excel spreadsheet version that you can download from his website.

It is only after you have gone through the analysis outlined above that you will be in a position to estimate how long implementing Data Governance is going to take in your organisation. Now clearly the timescales are going to vary, but in my experience, it is going to take you the best part of a year (and probably longer) to design and implement a Data Governance Framework over at least some part of your data or organisation. This doesn’t mean that you won’t be able to deliver some quick wins during this period, but it will take a reasonable amount of time and effort before your Data Governance Framework starts to deliver value on a regular basis.

I don’t say this to put you off starting in the first place, but I have seen so many people underestimate the amount of effort and time that a Data Governance initiative takes, and it is vital that you manage your stakeholder’s expectations from the outset.

So whilst I can’t give you an easy answer that works for everyone, I hope I’ve given you some insight into how to work out the answer for yourself.

How to use a Lean Approach to Data Governance

Getting a data governance initiative started can be extremely challenging.  This is especially true if you work in an unstructured manner, trying to start too many tasks at the same time or doing things in the wrong order.

Over the years I have developed my own methodology for implementing Data Governance successfully.  This is based on my experience of what has worked successfully (as well as what hasn’t) over many years of implementing Data Governance.  Although I got into Data Governance by accident when I was a Project Manager, I had never particularly considered whether a lean approach could be applied.  So, when I was first asked whether a Lean approach would work for implementing Data Governance, I decided to look into it.

It didn’t take me long to realize that I have incorporated some of the lean principles into my approach without realizing it and that a lean approach would definitely be a good way to structure a Data Governance implementation.

To be successful, data governance needs to be implemented iteratively and efficiently and that is why applying lean principles to data governance works so well.

What is Lean?

Lean was originally created by Toyota to eliminate waste and inefficiency in its manufacturing operations. The process became so successful that it has been embraced in manufacturing sectors around the world and is now used in many different industries as it can improve how teams work together.

The goal of lean is to eliminate waste, i.e. the non-value-added components in any process. The idea being that until a process has gone through lean multiple times, it contains some element of waste. When done correctly, lean can create huge improvements in efficiency, cycle time, and productivity, which leads to lower costs and improved competitiveness.

The Lean Enterprise Institute (LEI), founded by James P. Womack and Daniel T. Jones in 1997, is considered the go-to resource for lean wisdom if you want to learn more about the details, but for this blog I want to focus on why it is a good approach for Data Governance.

Lean principles are all about the following initiatives:

• Empowering small teams

• Reducing cycle times

• Gradually eliminating waste

• Focusing on value

So let’s look at each of these in turn:

Empower Small Teams

Lean improvements start with people, and the same can be said for data governance. Applying lean principles allows you to focus on your team first.

Instead of creating a huge project team to implement data governance, setting up a small central team to support users across your whole business will have greater success.

Rather than being responsible for all data, a small data governance team can focus on:

• Identifying and maintaining existing data management activities

• Providing a framework for managing and aligning existing data management activities, and planning for future activities

• Coordinating the implementation of the data governance framework

• Acting as a liaison between the Business and IT to verify that business requirements are fully understood by IT and ensure the business is fully engaged in IT led projects

Reduce Cycle Times

Many data governance initiatives fail because they are too big in scope, cost, and timescales. Working on small phases or projects will more likely lead to success.  For example, try executing one process in a business area. When that is completed, implement that same process in the next business area.  Don’t try to achieve too much at once; if the data governance programme is too large and unstructured, the benefits will not be delivered efficiently and the entire programme might get stopped.  If you focus on small areas of scope, you are likely to achieve small but consistent successes in the implementation of your data governance framework.

Gradually Eliminate Waste

Implementing a data governance program through small frequent phases (or projects) allows you to use the lean problem solving approach: The Plan – Do – Check – Adjust (PDCA) Cycle.

For example, the initial plan step would review the business area you are intending to implement data governance in and take measures to fully understand the current situation.  What are the priorities and challenges in that area? This knowledge will enable you to plan the implementation for a data governance framework that will benefit that area.

In the do phase, you will implement the framework and identify and brief various stakeholders about their roles and responsibilities. During this phase, prepare stakeholders to start following one of the data governance processes or activities, such as defining data items for a data glossary, or using a data quality issue resolution.

Next you check the results of the do phase, confirm they align with expectations, and identify what can be learned from the experience. Determine if anything should be done differently next time.

Finally based on the insight gained, you can either adjust your approach before planning to implement the framework in a new business area or move back to the do stage to make changes in the first business area.

Focus on Value

Taking inspiration from George Orwell’s Animal Farm, we could say that all data is equal, but some data is more equal than others.  Lean uses prioritization techniques to focus on areas where you can gain the most value, and the same approach can be applied to managing data.  You can’t achieve value from managing all of your data with the same level of monitoring and control.  The highest level of monitoring and control should only be applied to the most critical data, needed to successfully run your business (I talked about this in more detail in this recent blog).  Applying lean principles and prioritising data governance activities for data that adds the highest value for the lowest effort will help engage stakeholders and demonstrate the benefits of data governance, while long-term activities (i.e. high value and high effort) progress.

Business engagement is absolutely vital to the success of all data governance activities, and underpinning these activities with the solid foundation of a data governance framework will help you achieve lasting data governance success.

Don’t try to do too much at one time. If your data governance initiative is unwieldy, it will be too big to get started and too slow to deliver benefits.  Applying a lean approach to data governance can help you work iteratively, checking and improving as you go and focusing your efforts on activities that will deliver the greatest value to your organization.

My Data Governance Checklist gives you a structured approach to design and implement a successful Data Governance Framework. You can download the free version of this checklist here.

Why You Need Data Governance

In this blog I’m going to look at why you really should do data governance. When I tell people what I do, I get a mixed response. Some people seem genuinely surprised that everyone isn’t already doing Data Governance, and an awful lot of people ask why would you need that?

Now I’m biased, as I believe that every organization would benefit from implementing data governance. It may not solve all problems, but it really does provide a framework which can be used to proactively manage your data.

A few years ago the main driver of Data Governance initiatives was regulatory compliance and while that is definitely still a factor, there is a move towards companies embracing Data Governance for the business value which it can enable. For example if your organisation is starting a digital transformation or wants to become “data driven”, you are not going to be successful if your data is currently not well understood, managed and is of poor quality.

If you embrace Data Governance and achieve better quality data, all sorts of benefits start to be seen. But you don’t have to take my word for it; take the DAMA DMBoK Wheel for instance: 

As you can see, it lists all the Data Management disciplines around the outside of the wheel. There in the middle, at the heart of it all, is Data Governance.  Now it didn’t just get put in the middle because there were no more spaces on the outside of the wheel – it’s there for a reason. Data Governance provides the foundation for all other data management disciplines.

Let’s look at a few of these disciplines to illustrate the point:

Data Quality

Without Data Governance all data quality efforts tend to be tactical at best. This means a company will be constantly cleaning or fixing data, perhaps adding default values when a key field has been left blank. With Data Governance in place, you will have processes, roles, and responsibilities to ensure that the root causes of poor data quality are identified and fixed so that data cleansing is not necessary on an on-going basis.

Reference and Master Data

Anyone who has been involved in any master data projects will have no doubt heard or read numerous dire warnings about the dangers of attempting these without having Data Governance in place. While I am not a fan of wholesale scaremongering to get people to embrace Data Governance, these warnings are genuine. For master data projects to be successful, you need data owners identified and definitions of all the fields involved drafted and agreed, as well as processes for how suspect matches will be dealt with. Without these things (which of course Data Governance provides) you are likely to be faced with a mess of under, over or mismatching!

Data Security

Of course Data Security is primarily an IT managed area, but it makes things a lot easier to manage consistently if there are agreed Data Owners in place to make decisions on who should and should not have access to a given set of data.

I hope you agree that these examples and explanations make sense, but don’t forget that is theory; and explaining this in data management terms to your senior stakeholders in order to get agreement to start a Data Governance initiative is unlikely to be successful. Instead, you are going to need to explain it in terms of the benefits it will bring. The primary reason to do Data Governance is to improve the quality of data.  So the benefits of Data Governance are those things that will improve, if the quality of your data improves.  This can cover a whole myriad of areas including the following:

Improved Efficiency

Have a look around your company. How many “work-arounds” exist because of issues with data? What costs could be reduced if all the manual cleansing and fixing of data were reduced or even eliminated?

Better Decisions

We have to assume that the senior management in your organization intends to make the best decisions. But what happens if they make those decisions based on reports that contain poor quality data? Better quality data leads to more accurate reporting.

Compliance

Very few organizations operate in an industry that does not have to comply with some regulation, and many regulations now require that you manage your data better. Indeed, GDPR (the General Data Protection Regulation) impacts everyone who holds data on EU Citizens (customers and employees), and having a solid Data Governance Framework in place will enable you to manage your data better and meet regulatory requirements.

So, at this point you are probably thinking, “isn’t it just a generic best practice thing that everyone ought to do?” And the answer is, yes – I do believe that every organization could benefit from having a Data Governance Framework that is appropriate for its needs.

What Happens if you Don’t Have Data Governance?

Well I’ll leave that to you have a look around you and decide what the likely consequences for your company could be, but it is usually the opposite of the benefits that can be achieved.

Remember data is used for dealing with your customers, making decisions, generating reports, understanding revenue and expenditures. Everyone from the Customer Service Team to your Senior Executive Team use data and rely on it being good enough to use.

Data Governance provides the foundation so that everything else can work.  This will include obvious “data” activities like Master Data Management, Business Intelligence, Big Data Analytics, Machine Learning, and Artificial Intelligence.  But don’t get stuck thinking only in terms of data.  Lots of processes in your organization can go wrong if the data is wrong, leading to customer complaints, damaged stock, and halted production lines. Don’t limit your thinking to only data activities.

If your organization is using data (and to be honest, which companies aren’t?) you need Data Governance.  Some people may not believe that Data Governance is sexy, but it is important for everyone.  It need not (in fact it should not) be an overly complex burden that adds controls and obstacles to getting things done. Data Governance should be a practical thing, designed to proactively manage the data that is important to your organization.

Just one final word of advice: I hope that this article has convinced you that your organization needs to embrace Data Governance; but if that is the case, please don’t just spout the generic benefits and examples I have shared here in your efforts to gain stakeholder buy in. It is very important to spend time working out the specific reasons your company should be doing Data Governance. You can find more advice on that and how to engage your senior stakeholders here.

Does it have to be called Data Governance?

This is a question that I get asked fairly regularly. After all it is not an exciting title and in no way conveys the benefits that an organisation can achieve by implementing Data Governance. Sadly however, there is no easy yes or no answer. There are a number of reasons for this:

1. Data governance is a misunderstood and misused data management term

Naturally I am biased, but in my view, data governance is the foundation of all other data management disciplines (and of course therefore the most important). But the fact remains that despite an increasing focus on the topic, it remains a largely misunderstood discipline.

On top of this, it is a term which is frequently misused. A few years ago, a number of Data Security software vendors were using the term to describe their products. More recently the focus on meeting the EU GDPR requirements has led to a lot of confusion as to whether Data Protection and Data Governance are the same thing and I find that the terms are being used interchangeably. (For the record, having Data Governance in place does help you meet a chunk of the GDPR requirements, but they are not the same thing).

Having more people talking about Data Governance is definitely a good thing, but unless they are all meaning the same thing, it leads to much confusion over what data governance really is.

I explored this topic in a bit more detail in this blog: Why are there so many Data Governance Definitions?

In order to understand whether Data Governance is the right title for your organisation to call it, I would start with looking at how you define data governance. And this step leads nicely to the next item for consideration.

1. Sometimes it is right to include things which are not pure data governance in the scope of your data governance initiative.

This is a topic that I covered in my last blog which you can read here.

To summarize that article, it is just not possible to have one or more people focus purely on Data Governance in smaller organisations. It’s a luxury of large organizations to be able to have separate teams responsible for each different data management discipline (e.g. Data Architecture, Data Modelling or Data Security).  Going back to my point above, if data governance is the foundation for all other data management disciplines, it is only natural that the line between them can sometimes get a little blurred. As a result of this, the responsibilities of the Data Governance Team can get expanded.

So consider what is included within the scope of your data governance initiative and decide whether it be more appropriate to name the initiative and your team (either or both)  something that is more aligned to the wider scope of the initiative and activities of the team.

Is the name going to make cultural change harder to achieve?

Achieving a sustainable cultural change is one of the biggest challenges in implementing data governance and insisting on calling it “data governance” could make achieving that cultural change more difficult if the term doesn’t resonate within your organization. This is related to a topic that I explored in another old blog Do we have to call them Data Owners?

Whether we’re talking about the roles, the team, or even the initiative the same principles are true. It is better to choose a name that works for the culture in your organization than to waste considerable effort trying to convince people that the “correct” terminology is the only one to use.

It would be my preference to explain that the initiative is to design and implement a Data Governance Framework, but if the primary reason for implementing data governance is to improve the quality of your data, perhaps calling it the “Data Quality Team” and “Data Quality Initiative” would fit better? After all, that very much focuses on the outcome of what you’re doing.  It also addresses the question that everybody asks (or should ask) when approached to get involved in data governance of “why are we doing this,” which is usually followed by “what’s in it for me?”

When having these conversations, I explain the initiative in terms of its outcomes (e.g. better quality data which will lead to more efficient ways of working, reduced costs and better customer service). That is a far easier concept to sell rather than implementing a governance structure, which can sound dull and boring.

Is the name causing confusion?

In the early days of a data governance initiative, the talk is all about designing and implementing a data governance framework. Once this work has been achieved you start designing and implementing processes which have “Data Quality” in their titles:

• Data Quality Issue Resolution

• Data Quality Reporting

I have been fortunate enough to work with organizations in the past who have had both a Data Governance Team (supporting the Data Owners and Data Stewards) and a Data Quality Team (responsible for the processes mentioned above) but that is fairly unusual in my experience. It is more common for the Data Governance Team to support the above processes. So it is worth considering whether it would confuse people if they had to report data quality issues to the Data Governance Team?

In summary, I would not want to miss the opportunity to educate more people on what Data Governance really is. But the banner under which it is delivered can be altered to make your data governance implementation both more successful and more sustainable. So if having considered all the points above in respect of your organization and you want to call it something else, then that is fine with me.

Deciding what to call your initiative is only the start of many things you need to do to make your Data Governance initiative successful.   You can download a free checklist of the things you need to do here. (Don't forget this is a high level summary view, but everyone who attends either my face to face or online training gets  a copy of the complete detailed checklist which I use when working with my clients.)

What should you include in a Data Governance initiative?

One of the many challenges you will have to face when implementing Data Governance is agreeing the scope of the initial phase of your initiative. By this I don’t just mean which data domains or business functions are going to be in scope. I’m thinking of associated activities like data retention, end-user computing, and data protection. Being a bit of a Data Governance purist I maintain that such activities are most definitely NOT data governance. It is easy therefore to make the logical conclusion that they should not be in the scope of your initiative. So what I say next may surprise you:

Do not immediately go on the defensive and refuse to take any (or even all) of these activities into the scope of your initiative!

Now you may be wondering why someone who spends her time educating people on what Data Governance is would say that! Well, when I’m training and coaching people it is important that they understand what Data Governance is, but when I’m implementing Data Governance in practice, I take a pragmatic approach.

However, I would not want you to think that I would just say yes to an ever-expanding scope. There are a number of factors that would make me consider bringing these additional data activities into the scope of my data governance work, which include:

• If you work for a small organization that does not have the luxury of separate specialist teams to cover each data management discipline;

• If they overlap with other projects ongoing at the same time;

• Or if a senior stakeholder requests it.

Whilst you may become aware of other activities that you want to bring into scope, they are most likely to come to your attention through your senior stakeholders – so let’s consider this question:

How do you manage senior stakeholders who ask you to extend the scope of your initiative?

Now whilst it may be tempting to protect the scope of your initiative, remember they have their own agenda. They are not trying to derail your plans, they just have concerns of their own or issues that they need addressed. The first thing you are going to need to do is to listen and understand what their concerns are before you try to educate or influence them. After all, how can you properly allay their concerns if you don’t fully understand them?

But remember whilst it is imperative that you understand why they’re asking you to extend the scope, when I say educate or influence them, I don’t mean your initial stance is to say no! When talking to your senior stakeholder, ask lots of questions and constantly consider the following:

• What exactly does this person need done?

• Does it have any alignment or overlap with your data governance work?

• What will happen if this additional work does not get done? (And in particular will it cause a problem for your data governance initiative?)

Even if the answer to this last question is no, it may still be necessary for you to consider that if you say no, that this senior stakeholder could divert resources currently allocated to your initiative to address this other issue.

Are there benefits and/or efficiencies to be achieved by taking on this work? This can be especially true if you are talking to the same stakeholders.

My advice is to look for solutions that help everyone. This is not about you or them winning. This is about doing the right thing for your organization. Find out why he/she is concerned about these other topics. Is it because they are not being done, or is it that they are being done but are not visible or are being done but not well enough or quickly enough?

Now obviously I’m biased, but I truly believe that well implemented data governance can be the framework against which you align an awful lot of other activities in your organization (well at least those concerning data)! Once in place, you can use your data governance framework to coordinate, oversee, and escalate other data matters to the appropriate people. That said, it is not the answer to everything and you should resist taking on everything (unless of course you are Superman/Superwoman), or at least agree to timescales for adding additional scope once the implementation of your data governance framework has reached a certain stage.

If you do take on something that perhaps you feel is not in the area of your expertise, that is ok – just be honest and clear on the matter. Explain that whilst, for example, you may not be a data retention expert, you see how including that in your data governance initiative has benefits for the organization. Confirm that you are happy to do the necessary research and support the work if you are given the necessary expert support (for example from your Legal Department).

Remember that whether your data governance initiative is small and focused or has gained additional scope, stakeholder engagement is absolutely vital for success. You need to spend a lot of effort engaging your stakeholders. If you could lose their support by not addressing their other concerns, it’s got to be worth considering whether the additional work is something that you can take on.

Finally, if you want ideas on how to go about engaging your stakeholders, you can download my top tips on stakeholder engagement for free if you click here.

Originally posted on TDAN.com