Bad Data Quality Can Be Helped by Decoded Data Lineage

What do you hope to get out of data lineage? After all, no one is merely having a good time. Data lineage is a crucial asset for regulatory reporting/governance, decision-making trust, and on-premise to cloud migrations, to name a few. (data science course Malaysia)

Moreover, data lineage tools track the flow of corporate data from its source through all stages of its lifespan to its final destination. Technical data transformation logic can also be tracked using data lineage tools. A graphic representation allows you to see the overall flow in a more intuitive way.

The mandate of today’s CDO – and the challenge – is familiarity with the fundamentals of data capability. The foundational elements that keep leaders up at night, according to an interview with Ramesh Nair, North America Financial Services leader at Accenture Applied Intelligence, are extracting value from current big data investments, preparing for the future, and Metadata Management, Data Quality, Data Governance, and data lineage. Despite this, according to one survey, 66% of CDOs have not used Data Lineage.

Data And Enterprise (data science course Malaysia)

In the severely regulated financial services sector, Christopher Butler, the CDO for Asia-Pacific International Markets at HSBC, has spoken on the importance of data lineage. He explained that components of HSBC are developing the company’s data lineage programme in order to have a detailed picture of data across the entire business, allowing HSBC to extract crucial pieces and identify features like the data’s owner.

According to Gartner’s Magic Quadrant for Metadata Management Solutions, communication of a clear lineage for data and its use results in improved risk management and better assessment by decision-makers with regard to the impact of change within an enterprise.

The Importance of Analyzing System Metadata (data science course Malaysia)

The ability to extract lineage and data flow information for tracing data pathways from system metadata – such as parsing sophisticated SQL scripts, ETL setups, or report definitions – spans the complete spectrum of which data can be entered for computing other data. This is due to the fact that it analyses each and every piece of logic. Decoded lineage is a type of lineage that can be used in conjunction with data similarity lineage (examining the data values of schemas to look for similarities without accessing code).

MANTA, Octopai, and Spline are among the companies that offer decoded lineage capabilities. Ernie Ostic, MANTA SVP of Products, elaborates on the use cases for decoded lineage for the organisation and its CDO, including privacy compliance needs. “We don’t look at data; we look at code,” Ostic explains. Data lake, data vault, or data warehouse feeds and reports: “You need to be able to show them you have a complete grasp of how it got there.”

We can demonstrate it using a decoded lineage tool.

“The issues and dilemmas faced by those that require lineage are significantly higher than the expense of any programme,” he argues, especially for large institutions under regulatory scrutiny. “To avoid fines, they must answer lineage and exhibit complete control and awareness of their data.”

For efficient decision-making, you need pure, unadulterated trust in data, but data is frequently dubious. Ostic depicts the all-too-common scenario of dealing with a key decision-maker who wants to know how to calculate a report. The leader believes the number is incorrect and wants to discover where it came from by proving it through lineage. He observes that the typical response is, ‘Let me check.’ It’s a long chain from identifying the colleague who was supposed to have authored the report to discovering that it was actually produced by someone else and then emailing all the way down to the mainframe person who runs the report every week… so on and so forth

“You’re crossing multiple groups in a broad organisation,” Ostic explains. “How long it takes has an impact on the executive’s ability to make choices and conduct business.” The ability to automate the image of lineage through programme analysis can considerably speed things up.

MANTA

MANTA can also save a temporal slice of lineage. For example, if a corporation needs to examine how to calculate a report on a specific day last year, a simple click of the mouse can take the user to that lineage and compare it to the current lineage to see if there are any differences.

Decoded lineage also applies to cloud migrations, as it reduces the amount of effort and resources spent on coding decisions. One of the financial organisations with whom MANTA collaborated required to move data from the ground to the cloud. It had approximately 2,000 tables in Microsoft SQL Server and DB2, as well as a few hundred reports and 73 additional ETL scripts. When we reduced the number of crucial reports to seventeen, the firm only had to lineage those reports. MANTA might build a complete representation of the data lineage by scanning the metadata and analysing all of the SQL programming code and logic placed in it to scope the migration and reduce the duration.

In addition to MANTA’s standalone product, it may help users that have invested in a catalogue tool such as IBM’s Information Governance Platform, Collibra, Informatica, TopQuadrant, or others package and push lineage through their native API.

“For customers, lineage is really significant,” adds Ostic. “It’s about what you can do with it, not simply the bloodline.”

Source: data science course malaysia , data science in malaysia