Data Governance for Engineers: Building Compliance That Ships with the Code

Summary#

Most data governance programmes start with a policy document and end with a spreadsheet that nobody updates. This article, part of the data architecture series, covers the engineering side: implementing GDPR right-to-erasure across denormalised warehouse tables, automating PII detection at ingestion, enforcing access controls that scale beyond 200 analysts, and building audit trails that satisfy regulators without slowing down the team. The approach is governance as code, where compliance ships with the pipeline, not as a retrospective audit.

The Audit That Changes How You Think About Data#

The auditor's questions were simple. Who accessed this table in the last 90 days? When was the last PII scan on the payments schema? Can you demonstrate that this customer's data was deleted within the 30-day GDPR window? How do you know the deletion was complete across all downstream tables?

The team could answer the first question from the query logs. The remaining three required a week of manual investigation.

This is the pattern across every data platform I have seen in regulated industries. The infrastructure to process data is mature. The infrastructure to prove you are processing it correctly is an afterthought. The gap between what the regulation requires and what the engineering delivers is not a policy problem. It is a systems design problem.

Cumulative GDPR fines have reached EUR 5.88 billion since the regulation took effect (DLA Piper GDPR Fines and Data Breach Survey, January 2026). The average cost of a data breach hit $4.88 million in 2024, a 10% increase year-over-year (IBM Cost of a Data Breach Report, 2024). The 96% of organisations that report privacy investment returns exceed costs, with a median 1.6x ROI (Cisco Data Privacy Benchmark Study), are the ones that automated early. The rest are paying with incident response hours.

Governance as Code: Moving Compliance into the Pipeline#

Traditional data governance operates on a review cycle. Someone writes a policy. Someone else audits compliance quarterly. The findings go into a report. The report generates action items. The action items compete with feature work. The cycle repeats.

Governance as code inverts this. Access policies, PII classification rules, retention schedules, and audit configurations are version-controlled alongside the pipeline code. They are tested in CI. They deploy with the data platform. When a policy changes, the change is a pull request with a clear diff, not an email attachment.

# governance/access_policies/payments.yaml
policy:
  schema: payments
  classification: restricted
  rules:
    - role: payments_analyst
      columns: ["order_id", "amount", "currency", "status"]
      row_filter: "region = '${user.region}'"
    - role: compliance_officer
      columns: ["*"]
      row_filter: null
    - role: data_scientist
      columns: ["order_id", "amount", "currency", "status"]
      pii_columns_masked: ["customer_email", "billing_address"]
  retention:
    raw: 90 days
    aggregated: 7 years
  audit:
    query_logging: enabled
    access_review_frequency: monthly

The CI pipeline validates that every table with a restricted classification has an access policy defined, that every PII column is either masked or excluded for non-privileged roles, and that retention policies have corresponding deletion jobs scheduled in Airflow.

This is not aspirational. Uber's Databook catalog enforces mandatory ownership per table, and their data quality monitoring system executes roughly 100,000 test runs daily across 18,000 tests. The engineering investment is significant, but the alternative is the audit conversation described above.

PII Detection and Classification at Scale#

Manual PII tagging does not scale. A warehouse with 4,000 tables and 60,000 columns cannot be classified by humans reviewing column names. The names are often misleading: field_7 might contain email addresses, user_ref might be a hashed identifier with no PII exposure, and notes almost certainly contains unstructured text with names, phone numbers, and occasionally passport numbers embedded in free-form comments.

Automated PII detection during ingestion works in three layers. Pattern matching (regex for email addresses, credit card numbers, phone numbers, national ID formats) catches the obvious cases. Named entity recognition catches PII in unstructured text fields. Statistical profiling (cardinality analysis, value distribution) identifies columns that behave like identifiers even if they are not labelled as such.

The classification taxonomy matters. Direct identifiers (name, email, phone number, national ID) require different treatment than quasi-identifiers (age, postcode, job title) and sensitive attributes (health data, financial transactions, political affiliation). The combination of quasi-identifiers can re-identify individuals even after direct identifiers are removed. This is why k-anonymity and differential privacy exist, and why simple column masking is insufficient for research datasets.

# PII classification pipeline (runs during ingestion)

CLASSIFICATION_LEVELS = {
    "direct_identifier": {
        "action": "mask_or_tokenize",
        "examples": ["email", "phone", "ssn", "passport"],
    },
    "quasi_identifier": {
        "action": "generalize",
        "examples": ["age", "postcode", "job_title"],
    },
    "sensitive_attribute": {
        "action": "restrict_access",
        "examples": ["diagnosis", "salary", "political_view"],
    },
    "non_pii": {
        "action": "none",
        "examples": ["product_id", "timestamp", "count"],
    },
}

# Classification result feeds into:
# 1. Column-level access policies
# 2. Masking rules for non-production environments
# 3. Retention schedule assignment
# 4. Audit trail requirements

The engineering tradeoff between pseudonymisation and anonymisation is important. Pseudonymised data (where identifiers are replaced with tokens that can be reversed) remains personal data under GDPR. Anonymised data (where re-identification is not reasonably possible) falls outside GDPR scope entirely. Most teams default to pseudonymisation because anonymisation is harder to achieve and harder to prove. The regulators know this, which is why pseudonymised data still requires full GDPR compliance.

GDPR and PCI DSS in the Data Warehouse#

The right-to-erasure (Article 17) is straightforward in a normalised transactional database. Delete the row, cascade the foreign keys, confirm the deletion. In a denormalised data warehouse, it is an engineering project.

A customer record in a well-designed warehouse is spread across dimension tables, fact tables, materialised views, historical snapshots, and potentially replicated into downstream systems. A deletion request means identifying every location where that customer's data exists, executing deletions in the correct dependency order, and verifying completeness.

The practical architecture uses a three-layer approach. The Bronze layer stores raw data with PII intact, encrypted at rest, with restricted access. The Silver layer applies pseudonymisation: PII is separated from behavioural data, and identifiers are replaced with tokens. The Gold layer contains aggregated, anonymised data with no direct PII. GDPR deletion requests target the Bronze and Silver layers. The Gold layer, if properly anonymised, does not require per-customer deletion.

-- Erasure ledger: tracks deletion requests and execution status

CREATE TABLE compliance.erasure_ledger (
    request_id UUID PRIMARY KEY,
    customer_id VARCHAR NOT NULL,
    requested_at TIMESTAMP NOT NULL,
    regulation VARCHAR NOT NULL,  -- 'GDPR_ART17', 'CCPA', etc.
    status VARCHAR DEFAULT 'pending',
    tables_affected JSONB,
    completed_at TIMESTAMP,
    verified_at TIMESTAMP,
    verified_by VARCHAR
);

-- Deletion must cascade through:
-- 1. dim_customers (direct delete)
-- 2. fct_transactions (hash customer_id, retain aggregates)
-- 3. Materialised views (rebuild after deletion)
-- 4. Staging tables (purge matching records)
-- 5. Backup snapshots (flag for exclusion on restore)

Surgical deletion from backup snapshots is often technically impossible without a full restore, delete, and re-backup cycle. This is an area where the regulation and the engineering reality diverge. The pragmatic approach is designing backup retention periods that align with the maximum deletion timeline, so that backups containing the deleted data age out naturally within the compliance window.

PCI DSS adds a separate set of requirements for cardholder data. Card numbers must be encrypted at rest and in transit, masked in all display contexts (show only last four digits), and accessible only to users with a documented business need. In a warehouse, this means cardholder data should never exist in a general-purpose analytics schema. It belongs in an isolated, encrypted, access-controlled schema with separate audit logging.

Access Control That Scales Beyond Spreadsheets#

Role-based access control (RBAC) works until the organisation has 200 analysts who need different views of the same tables. At that point, the role explosion becomes unmanageable: payments_analyst_eu, payments_analyst_us, payments_analyst_eu_pii, payments_analyst_us_no_pii. Each new combination creates a new role, and each new role needs to be maintained.

Attribute-based access control (ABAC) resolves the explosion by evaluating policies against user attributes at query time. Instead of pre-defining a role for every combination, the policy engine checks the user's department, region, clearance level, and the data classification of the requested columns. Snowflake's row access policies and column masking policies support this pattern natively.

The most common mistake is over-permissioning. When an analyst requests access to a schema, the path of least resistance is granting SELECT on all tables. The path of compliance is granting SELECT on specific tables with PII columns masked. Most teams default to the former because the latter requires tooling investment. The tooling investment is always smaller than the cost of explaining to a regulator why 200 analysts had unrestricted access to customer email addresses.

Dynamic data masking for non-production environments solves a related problem. Development and staging environments need realistic data for testing, but they should not contain real PII. Masking functions that replace names with fake names, emails with hashed equivalents, and phone numbers with randomised digits produce test data that behaves like production data without the compliance exposure.

Audit Trails and Lineage for Compliance#

A compliance-grade audit trail answers four questions for any piece of data: where did it come from (lineage), who has accessed it (access logs), how has it been transformed (processing records), and when was it last validated (quality checks).

Most data platforms can answer the first two. Query logs capture who ran what query against which tables. Lineage tools (dbt's built-in lineage, OpenLineage, Marquez) trace data from source to dashboard. These are necessary but not sufficient.

The third question, processing records, maps to GDPR Article 30: records of processing activities. This is not a technical log. It is a document that describes the purpose of each data processing operation, the categories of data subjects, the categories of personal data, the recipients, the retention period, and the security measures. The engineering implementation is a metadata layer that attaches purpose and legal basis to each pipeline.

The fourth question, validation history, is where the data observability engineering guide intersects with compliance. Quality check results need to be retained, not just executed. When an auditor asks "was this table fresh on March 15?", the answer must be a timestamp from a check that ran on March 15, not a dashboard showing current freshness.

The Governance Maturity Curve#

After working on data platforms in regulated industries, the progression is consistent across organisations. It moves through four stages.

Reactive: the team has a governance policy document. It was written for a compliance certification. It lives in a shared drive. Nobody references it during development. When an audit happens, the team scrambles to demonstrate compliance manually.

Documented: data assets are catalogued with ownership, classification, and lineage. The catalogue exists and is maintained, but enforcement is manual. The catalogue tells you what should be happening. It does not verify that it is.

Automated: governance rules are codified in CI. Access policies are enforced at the platform level. PII detection runs during ingestion. Retention policies execute automatically. The catalogue is generated from the code, not maintained separately. This is the stage where governance stops being a cost centre and becomes a reliability feature.

Proactive: continuous compliance monitoring with automated remediation. Policy violations trigger alerts in the same system that monitors pipeline health. New tables are automatically classified and assigned default policies. Deletion requests execute within the compliance window without manual intervention.

Most organisations I have worked with are between stages one and two. The gap between stage two and stage three is not technology. It is the decision to treat governance as an engineering problem rather than a documentation problem.

Gartner estimates that 80% of data governance initiatives will fail by 2027 without a crisis catalyst. The crisis is usually an audit finding, a breach, or a fine. The organisations that reach stage three before the crisis are the ones that build governance into the same architecture decisions that shape their data platform, and apply the same database selection rigour to choosing governance tooling.

Compliance is not a feature you add later. It is a property of the system design.

Frequently Asked Questions#

How do you implement GDPR right-to-erasure in a denormalised data warehouse?#

Right-to-erasure (Article 17) is technically challenging in a warehouse where user data is denormalised across fact and dimension tables, aggregated in materialised views, and embedded in historical snapshots. The practical approach is maintaining a central erasure ledger, implementing soft deletes with a retention window, cascading deletions through lineage-aware scripts, and validating completeness with automated scans. Hard deletes in columnar storage are expensive; design your warehouse schema with erasure in mind from the start.

What is DataGovOps and how is it different from traditional data governance?#

DataGovOps treats governance rules as code: access policies, PII classification rules, retention schedules, and audit configurations are version-controlled, tested in CI, and deployed alongside the data platform. Traditional governance relies on policy documents and manual enforcement. DataGovOps embeds compliance into the engineering workflow so that governance is enforced automatically rather than audited retrospectively.

What are the minimum data governance requirements for a fintech data platform?#

At minimum, a fintech data platform must implement column-level access controls for PII and payment data, automated PII detection during data ingestion, retention policies that execute automatically, query-level audit logging, and documented data lineage from source to consumption. GDPR requires demonstrable data processing records (Article 30), and PCI DSS requires encryption of cardholder data at rest and in transit, with access restricted to business need.