APIs are often described as a technical integration mechanism: one system sends a request, another returns data or performs an action.

That is true, but it is incomplete.

In an enterprise environment, an API is also a security boundary, an operational dependency and a product that needs ownership over time.

That distinction matters because many API problems are not caused by the endpoint itself. They come from weak identity, unclear ownership, unmanaged versions, poor observability, or choosing an API for an interaction that would have been better handled another way.

A useful way to think about an enterprise API is:

Consumer → Gateway → Authenticate → Authorise → Service → Backend → Observe → Govern

The code may sit in the middle, but the reliability of the overall service depends on the whole chain.

Start with the business interaction

Before choosing technology, clarify what is actually happening between the systems.

Is a user waiting for an immediate response? Is a partner sending a structured document? Is a system moving a large analytical data set? Is the process long-running and tolerant of delay?

Those questions determine the integration pattern.

A synchronous API fits well when a consumer needs an immediate request/response interaction: retrieve an account, submit a transaction, check availability or invoke a business capability.

But an API is not automatically the right answer for every integration.

• High-volume analytical movement may be better handled through ETL/ELT or streaming.
• Long-running or loosely coupled workflows may be better suited to events or queues.
• Structured B2B document exchange may still fit EDI.
• Simple scheduled transfers can sometimes be handled safely through managed file transfer.

The goal is not to maximise the number of APIs. It is to use the integration pattern that matches the business interaction.

The API contract is part of the architecture

An API creates a defined contract between a consumer and a service.

That contract lets the consumer use a business capability without needing to understand the internal implementation of the backend system.

Done well, this reduces coupling.

A mobile application should not need to know the database schema of the customer system. A partner should not need privileged access to an internal platform. A new channel should not require every backend system to be redesigned.

The API creates a controlled boundary.

That boundary is useful precisely because it can remain relatively stable even when the implementation behind it changes.

But stability requires discipline. The contract needs to be documented, changes need to be managed, and consumers need enough warning when a version is going away.

An undocumented API with unknown consumers is not really a reusable service. It is an invisible dependency.

Authentication and authorisation are different controls

This is one of the simplest distinctions in API security, but also one of the most important.

Authentication asks: who or what is calling?

Authorisation asks: what are they allowed to do?

A valid identity is not the same as permission to perform every action.

Enterprise APIs should use strong identities for users and services rather than shared accounts or credentials embedded in application code. Depending on the environment, that may involve OAuth 2.0, OpenID Connect, managed identities, service principals or other fit-for-purpose identity patterns.

Once identity is established, permissions should still be constrained.

A service that only needs to read customer status should not receive write access to customer records. A partner integration should not inherit broad backend permissions simply because integration is easier that way.

The principle is familiar:

least privilege applies to APIs as much as it applies to people.

The gateway is useful, but it is not the application

An API gateway can provide a common entry point for routing and cross-cutting controls.

It may centralise:

• authentication integration;
• rate limiting and throttling;
• request filtering;
• policy enforcement;
• logging;
• routing; and
• common security controls.

That is valuable because it reduces the need to reinvent the same controls for every service.

But the gateway should not become the place where all business logic ends up.

The gateway can decide whether a request is permitted to reach a service. The service should still be responsible for its own business rules, validation and behaviour.

Otherwise the organisation simply moves application complexity into a different layer.

An API is part of the attack surface

Once an API is exposed, it becomes another route into organisational systems and data.

The basic controls are not exotic:

• encrypt traffic in transit with TLS;
• validate inputs and reject unexpected data;
• minimise the data returned to the consumer;
• keep secrets in managed secret stores rather than source code;
• apply appropriate rate limiting;
• use strong identity and least privilege;
• log meaningful events; and
• monitor for unusual access patterns.

The important point is that security is not something added after the interface has been designed.

The interface itself determines what capability is exposed, to whom, and under what conditions.

A well-designed API can actually improve security by giving consumers a narrow, controlled path to a business capability instead of direct access to the backend.

A poorly governed API can do the opposite.

Versioning is a business issue, not just a developer issue

Changing an API can break every consumer that depends on it.

That makes versioning and deprecation an operational concern.

Before changing an interface, an organisation should know:

• who consumes it;
• which versions are in use;
• whether the change is backward compatible;
• when an older version can be retired;
• how consumers will be notified; and
• who owns the migration.

This is where an API inventory or catalogue becomes valuable.

It does not need to become a large bureaucracy. The objective is simply to stop critical interfaces becoming invisible.

If nobody knows who owns an API or who consumes it, every change becomes a discovery exercise.

Observability is part of the service

A technically successful request is only one part of operational health.

For an important API, I want to understand:

• availability;
• response times;
• error rates;
• usage patterns;
• authentication failures;
• capacity or throttling events; and
• unusual access behaviour.

Logs, metrics and traces allow a team to distinguish between a consumer problem, gateway issue, service defect and backend dependency.

Without that visibility, incident response becomes guesswork.

This is why I prefer to think of APIs as operational products, not simply integration code.

Someone needs to own the service.

Give the API an owner

Every material API should have clear accountability for:

• availability;
• security;
• defects;
• documentation;
• versioning;
• consumer communication; and
• lifecycle decisions.

That does not mean one individual does all of the work.

It means there is an accountable point from which technical, security and operational responsibilities can be coordinated.

The same idea applies to the consumer side. A critical integration should not depend on two teams each assuming the other is responsible.

The leadership view

At leadership level, I do not need to write every endpoint to ask useful questions about an API estate.

I need to know whether we can answer:

1. What business capability is this interface providing?
2. Who or what can call it?
3. How are authentication and authorisation handled?
4. What data is exposed?
5. Who owns the API?
6. Who consumes it?
7. How are changes and versions managed?
8. How do we know when it is unhealthy or being misused?
9. What backend dependency does it create?
10. Is an API actually the right integration pattern here?

Those questions connect architecture, security and operations.

And that is the point.

An enterprise API is not successful simply because two systems can talk to each other.

It is successful when the interaction is appropriate, secure, observable, supportable and governable over time.

Related reference: Enterprise APIs: Architecture, Security & Governance

More technology leadership material: https://rakeshranderia.com.au/technology-leadership.html

For critical data, it is better understood as a dependency and control map.

If we cannot explain where important data came from, what happened to it, and where it goes next, it becomes very difficult to govern that data properly.

A simple lineage model

Source → Ingest → Transform → Store → Serve → Consume → Retain / Delete

This model is deliberately simple.

The technology behind each stage may be very different, but the governance questions remain broadly the same.

At each stage, capture:

• System / Dataset
• Authoritative Source
• Owner
• Classification
• Interface
• Transformation
• Downstream Dependency
• Access Model
• Quality Control
• Retention
• Evidence
• Criticality
• Review Date / Trigger

The aim is not to document every field in every system.

The aim is to understand the important movement, transformation and dependency points well enough to manage risk, change and accountability.

Start with the authoritative source

One of the first questions should be:

Where did this data originally come from?

That may sound obvious, but once information has moved through several applications, integrations, data platforms and reporting layers, the answer can become surprisingly unclear.

A useful lineage record should identify the authoritative source or system of record.

Without that, teams may end up making decisions using copied, stale or transformed versions of data without understanding how they differ from the original.

Document how the data moves

Data can move through many different integration patterns:

• APIs
• Events and messaging
• EDI
• Managed file transfer
• SFTP
• ETL / ELT pipelines
• Database replication
• SaaS integrations
• Manual spreadsheets

The interface itself matters because different patterns introduce different operational and security considerations.

For example, an API may introduce authentication, authorisation and availability dependencies, while a scheduled file transfer may introduce reconciliation and timing risks.

The technology is important, but lineage should capture the business dependency created by that technology.

Transformations matter

The data arriving at a downstream system may not be the same data that left the source.

Transformations can include:

• field mappings
• calculations
• enrichment
• aggregation
• masking
• tokenisation
• de-identification
• currency conversion
• timezone conversion
• identity matching
• data consolidation
• quality corrections
• business-rule application

These changes can affect meaning as well as format.

If a downstream report shows a value that differs from the source, lineage should help explain why.

This is particularly important where data drives financial reporting, regulatory obligations, customer decisions or automated processes.

Classification should follow the data

Classification and lineage are closely connected.

Classification tells us the required control baseline.

Lineage tells us where that control requirement needs to follow.

If a dataset is classified as Confidential or Restricted, copying it into another system should not automatically reduce that requirement.

A practical rule is:

Downstream data should inherit the source classification unless there is a documented and approved reason to change it.

There may be legitimate reasons for reclassification, such as verified aggregation, masking, tokenisation, de-identification or removal of sensitive fields.

But the change should be deliberate rather than accidental.

Downstream dependency is often the missing piece

Lineage should not stop once data reaches a database or data warehouse.

It should continue into the systems and processes that rely on it.

That may include:

• operational applications
• reports and dashboards
• customer-facing services
• APIs
• exported files
• third parties
• automated business processes
• machine-learning models
• AI agents
• vector stores

This becomes especially important during system changes.

If a source system is retired, an interface changes, or a data field is modified, good lineage helps answer:

What else will this break?

That makes lineage useful for architecture, change management and operational resilience — not just data governance.

Quality and reconciliation belong in lineage

Knowing where data moves is only part of the picture.

For critical data, I also want to know how we confirm that the movement worked correctly.

Typical controls might include:

• record-count reconciliation
• financial-total reconciliation
• duplicate detection
• validation rules
• exception reporting
• completeness checks
• timeliness checks
• failed-interface monitoring

This links lineage directly to data quality.

Instead of saying only:

System A sends data to System B.

A stronger lineage record can explain:

System A sends the daily transaction dataset to System B through an API, and the receiving process reconciles record counts and financial totals before processing completes.

That is much more useful operationally.

Access and ownership should travel with the record

A lineage map should also help answer:

• Who owns this data?
• Who supports the platform?
• Who is allowed to access it?
• Which service accounts or integrations can read or modify it?
• Who approves changes?
• Who accepts an exception?

Ownership without decision rights becomes little more than a title.

Lineage becomes substantially more useful when ownership and access controls are visible alongside the technical flow.

Third-party lineage matters too

Data does not stop being our responsibility just because it leaves our network.

If data is provided to a third party, lineage should record:

• what data was provided
• why it was provided
• the classification
• the transfer method
• the receiving organisation
• applicable contractual controls
• retention expectations
• return or deletion requirements

This becomes particularly important during supplier transitions, contract termination, M&A and divestment activity.

AI extends the lineage boundary

AI introduces another set of dependencies.

A modern lineage path may look like:

Curated Dataset → Embedding / Indexing → Vector Store → RAG / Agent → Model → User Output

Or for model development:

Source → Preparation → Training Dataset → Model Version → Evaluation → Deployment

Questions then include:

• What data was used?
• Where did it originate?
• Was that use permitted?
• How was it transformed?
• Which model or agent consumed it?
• Can the source be updated or deleted?
• How do we know which model version used which dataset?

This is one reason I see lineage becoming increasingly important as organisations expand their AI use.

Different levels of lineage are useful

Not every audience needs the same level of detail.

Business lineage

Focuses on:

• business domain
• ownership
• purpose
• major sources
• major consumers
• critical dependencies

Useful for executives, risk teams and governance forums.

System lineage

Focuses on:

• applications
• interfaces
• integrations
• transformations
• downstream systems
• operational ownership

Useful for architecture, IT operations and change management.

Technical lineage

May extend into:

• tables
• fields
• pipelines
• transformations
• schemas
• model features
• detailed dependencies

Useful where engineering, regulatory requirements or data complexity justify the additional depth.

The important point is to use the level of detail that supports the decision being made.

Lineage also needs a lifecycle

A lineage diagram created during a project and never updated quickly loses value.

Useful review triggers include:

• new integrations
• system migrations
• major application changes
• third-party onboarding
• AI or analytics use
• significant transformation changes
• acquisitions or divestments
• incidents
• regulatory or contractual changes
• system retirement

The goal is not perfect documentation.

The goal is current enough information to support real decisions.

The practical test

For an important dataset, I should be able to ask:

1. Where did it come from?
2. Who owns it?
3. How is it classified?
4. How did it get here?
5. What happened to it along the way?
6. Who can access it?
7. How do we know it is correct?
8. Where does it go next?
9. Who or what depends on it?
10. When was this last reviewed?

If those questions cannot be answered, there is probably a governance gap worth addressing.

Final thought

Data lineage is sometimes presented as a specialist data-management activity.

I see it more broadly.

It supports:

• data governance
• architecture
• operational resilience
• incident response
• change management
• security
• privacy
• M&A and divestment
• AI governance

The core principle is simple:

You cannot govern data well if you cannot explain where it came from, what happened to it, and where it goes next.

Related resources

• Data Lineage & Dependency Mapping
• Interactive Data Governance Toolkit
• Data Governance Toolkit source on GitHub

In practice, it usually starts much earlier — with the data.

If an organisation cannot clearly explain who owns its data, how it is classified, who can access it, whether it is reliable, or where it flows, then adding AI on top of that environment creates more uncertainty rather than more value.

That is why I tend to think about AI readiness as an extension of data governance rather than as a completely separate discipline.

The progression

A simple way to frame it is:

Data Governance → Trusted Data → AI Readiness → Responsible AI → Measurement → Scale

Each stage builds on the one before it.

1. Data ownership comes first

One of the most common governance problems is that data exists everywhere but ownership is unclear.

A practical operating model should make it easy to answer:

• who owns the data;
• who is responsible for its quality;
• who can approve access;
• what classification applies;
• what retention or lifecycle rules apply;
• where the data is used downstream.

This is less about creating more governance documentation and more about making accountability visible.

2. Quality becomes more important with AI

Poor-quality data creates poor-quality reporting.

With AI, the problem can become harder to spot because the output may still look plausible.

Useful data-quality controls therefore need to cover things such as:

• completeness;
• accuracy;
• consistency;
• timeliness;
• duplication;
• validity.

The aim is not perfect data.

The aim is to understand whether the data is good enough for the purpose it is being used for.

3. Classification needs to connect to AI use

Organisations often already have information-classification models.

There is usually little value in inventing a completely separate classification scheme just for AI.

Instead, the useful question is:

Is this class of information permitted in this AI platform for this use case?

That connects existing privacy, security, DLP and information-governance controls to AI adoption.

4. Access and lineage matter

Responsible AI depends heavily on understanding what data the system can access.

That means understanding:

• source systems;
• permissions;
• data movement;
• integrations;
• transformations;
• downstream consumers.

This is where lineage becomes especially useful.

You do not need perfect enterprise-wide lineage before doing anything with AI, but you do need enough visibility to understand what is feeding a material use case.

5. AI readiness is not just technical readiness

An organisation may have modern infrastructure and still not be ready to scale AI safely.

AI readiness also includes:

• clear ownership;
• acceptable-use rules;
• approved platforms;
• human oversight;
• privacy and security controls;
• risk assessment;
• monitoring;
• incident handling;
• measurable value.

This is why AI readiness sits between trusted data and responsible AI.

6. Governance should enable adoption

Governance is most useful when it helps people make better decisions faster.

Too little governance creates uncontrolled risk.

Too much governance creates bureaucracy and pushes people toward Shadow AI.

The better model is risk-proportionate governance:

• low-risk use cases move quickly;
• moderate-risk use cases receive additional controls;
• high-impact use cases receive deeper assessment and stronger oversight.

The objective is controlled adoption, not zero risk.

A practical operating model

The Data Governance + Responsible AI Framework I built uses this cycle:

Discover → Define → Assign → Control → Measure → Improve

It then connects that model to the AI lifecycle:

Intake → Triage → Assess → Approve → Pilot → Monitor → Scale / Restrict / Retire

That creates a clearer path from raw data to governed AI use.

Why this matters

Responsible AI is difficult to achieve if the underlying data environment is poorly understood.

Strong data governance does not guarantee good AI outcomes, but it creates the conditions needed to make AI decisions more deliberate, explainable and controllable.

The practical goal is not to make every organisation perfectly governed before it adopts AI.

It is to know enough about the data, risk and ownership to make good decisions as adoption grows.

The full framework is available on GitHub:

Data Governance + Responsible AI Framework
https://github.com/rakeshranderia/data-governance-framework