Data Lake Security: Bucket Policies, KMS Envelopes, and the Six Assumptions That Break in Production

A London fintech data lead pulls up the S3 policy console on a Tuesday afternoon. Her team runs a data lake across seven accounts — production, staging, analytics, ML training, backup, archive, DR. The last full audit of the bucket policies happened eighteen months ago. Since then, three teams have merged, one has spun out, and a data-science partner engagement has left behind a cross-account trust relationship nobody on the current team remembers approving. She has ninety minutes before her one-on-one with the CISO. She spends the first ten scrolling through S3 buckets and stops counting the ones she cannot immediately explain.

The data lake is the least-audited surface in most organisations that have one. AI security gets the headlines; posture-management platforms cover the compute plane; the data plane sits underneath, holds the crown jewels, and drifts. This piece is the honest engineering read on the six misconfigurations that show up in almost every data-lake incident review, the three architectural primitives that stop each of them from happening, and what a defensible data-lake configuration actually looks like when a regulator asks.

Key takeaways

The data lake is the sum of four control layers — bucket policies, IAM policies, service control policies at the organisation, and KMS key policies. A misconfiguration on any layer punches through the other three. Defence-in-depth is the design commitment, not the diagram.
KMS envelope encryption is the load-bearing primitive. The data key encrypts the object; the customer-managed key encrypts the data key; access to decrypt requires both the S3 read permission and the KMS decrypt permission. Skipping either half leaves an attacker one permission short of the payload.
Six misconfigurations show up in almost every data-lake incident — overly permissive bucket policy, missing SCPs at the organisation, wildcard KMS grants, drifted cross-account trust, unencrypted staging buckets, mis-scoped Glacier restore permissions. Each has a specific fix.
Cross-account trust drift is the failure mode that hurts most. A partner engagement ends, the trust relationship stays, and the audit chain still says the external account can read the crown jewels. Trust relationships need review cadence, not just approval workflow.
The compliance question at the end of the piece — GDPR Article 32, NDPA Section 39, PCI DSS Requirement 3 — asks the same thing in different words: prove the data is protected against unauthorised access, prove you can demonstrate it, prove the access audit chain is intact. The six fixes below are what makes the proof possible.

A data lake spans multiple AWS accounts (production, staging, analytics, ML training, backup, archive). Four control layers protect it — the bucket policy at each S3 bucket, the IAM policies on the identities that access buckets, the service control policies at the organisation root that constrain what any policy can grant, and the KMS key policies on the customer-managed keys that encrypt the objects. A read request has to satisfy all four layers to succeed. Below the layer diagram, the six real misconfiguration patterns are listed with their fixes. — Figure 1 — The four control layers of a defensible data lake and where each of the six misconfigs breaks them

The Four Control Layers

Every read against an object in a data lake bucket has to satisfy four independent policy layers. Skipping the framing that they exist as independent layers is where most data-lake configurations quietly become one-layer configurations.

Bucket policy — attached to the S3 bucket, granting or denying access to principals across accounts. This is the most-visible layer and the most-mis-drafted; a single overly-permissive Principal: "*" on the wrong condition set is a public bucket.

IAM policy — attached to the identity making the request. Federated users, service roles, machine identities. Each carries its own scope of buckets and actions it is entitled to touch. IAM is the layer teams tend to over-scope for convenience — data-scientists get broad s3:* on the ML bucket "for now" and it stays that way.

Service control policy — attached at the organisation or organisational-unit level. SCPs cannot grant access, only constrain it. A well-drafted SCP set at the organisation root prevents any account, any policy, any principal from making a bucket public or from disabling encryption. Missing SCPs is the failure that scales — every account has to individually get its bucket policy right, forever.

KMS key policy — attached to the customer-managed key that encrypts the object. Read requires both S3 permission and KMS decrypt permission. This is the layer that turns a bucket-policy mistake into a still-recoverable situation — the attacker who got the object can't decrypt it without also getting the KMS grant.

Defence-in-depth means all four layers are drafted, reviewed, and audited independently. Getting one right and treating the others as backup is not defence in depth; it is a diagram of defence in depth.

KMS Envelope Encryption Is the Load-Bearing Primitive

The pattern that turns a bucket-policy incident into a survivable incident is envelope encryption. A data key encrypts the object contents. A customer-managed key (CMK) encrypts the data key. The CMK never touches the object; the data key never leaves KMS unencrypted. To read the object, a principal needs S3 read permission (retrieve the encrypted object) plus KMS Decrypt permission on the CMK (unwrap the data key) plus the runtime path to combine them.

The reason this matters is the failure-mode geometry. A misconfigured bucket policy exposes the encrypted object; without the KMS grant, the object is ciphertext. A misconfigured KMS policy exposes the ability to decrypt; without the S3 grant, there is nothing to decrypt. The attacker needs both to reach payload, and the two are on independently-drafted policies with independent review cadences.

Envelope encryption also enables key rotation without re-encrypting every object. The CMK rotates on schedule; the data keys stay bound to their objects; the objects stay put. This is not a security control on its own — it is what makes the security control operational at data-lake scale.

The Six Misconfigurations That Keep Showing Up

Almost every data-lake incident review names one or more of these six patterns. They are common enough that any organisation running a lake without a specific programme to check them has some subset in production.

Overly permissive bucket policy. Principal: "*" without a matching condition set, s3:* where s3:GetObject would do, Resource: "*" where a specific prefix would do. The fix is drafted-policy review by a second engineer plus an SCP at the organisation root that denies s3:PutBucketPolicy for policies matching the overly-permissive shape.
Missing SCPs at the organisation. Any account can make any bucket public. The fix is a small set of organisation-wide SCPs — deny public bucket policies, deny unencrypted object uploads, deny disabling of Block Public Access, deny cross-region replication to non-approved regions. SCPs are the layer that scales.
Wildcard KMS grants. kms:Decrypt on Resource: "*" in an IAM policy, or kms:* at the key policy without a matching principal-scope condition. The fix is per-key policies with explicit principal scoping and IAM policies that reference the specific key ARN, never *.
Drifted cross-account trust. A partner engagement ended six months ago; the trust relationship for their AWS account is still in the bucket policy and the KMS key policy. The fix is a review cadence on cross-account trust — quarterly for standard trust, monthly for high-value data — with an automated report of every external principal in every policy.
Unencrypted staging buckets. Production is encrypted; the staging bucket used for one-off analysis is not. The staging bucket contains a snapshot of production data taken last month for a project that ended. The fix is an SCP that denies unencrypted uploads globally plus a monthly sweep for buckets with encryption not enabled.
Mis-scoped Glacier restore permissions. The archive tier is Glacier; the restore permission is granted broadly; a compromised identity can restore an object from archive without triggering the same alerts as production reads. The fix is scoped restore permissions with alerting on restore requests that fall outside a documented pattern.

Each fix is engineering work of hours-to-days, not months. The reason they do not get shipped is not cost; it is that nobody owns the discipline until an incident review makes ownership explicit.

The envelope encryption flow for an S3 GetObject request. The client requests the object; S3 returns the encrypted object and the encrypted data key. The client sends the encrypted data key to KMS with a Decrypt request; KMS checks the CMK policy and the caller's IAM policy for kms:Decrypt permission; if both pass, KMS returns the plaintext data key. The client uses the plaintext data key to decrypt the object contents. The failure geometry is annotated — a bucket-policy leak exposes ciphertext only; a KMS-policy leak exposes decryption capability with nothing to decrypt; both are required for the attacker to reach payload. — Figure 2 — Envelope encryption on GetObject — why the attacker needs two independent grants to reach payload

Cross-Account Trust Drift

The failure mode that hurts most is the one that looks fine on the day it is set up and quietly rots afterward. A partner engagement starts, a cross-account trust relationship gets approved, the partner's AWS account is granted s3:GetObject on a specific prefix and kms:Decrypt on a specific key. Six months later the partner engagement ends. Twelve months later nobody on the current team remembers the partner. Eighteen months later the trust is still there and the audit chain still says the external principal can read the crown jewels.

The fix is discipline that survives team turnover. Every cross-account trust relationship has an owner recorded in the policy comment. Every trust relationship has an expiry date; if it needs to persist beyond the expiry, the extension goes through the same approval workflow as the original grant. Every review cycle produces a report of every external principal in every policy, with the owner and the expiry, and the report goes to a human whose job is to challenge each row. The tool support for this is uneven; the discipline works with or without it.

Under NDPA Section 39, GDPR Article 32, and PCI DSS Requirement 7, the regulator asks the same question in different words — prove access is limited to the principals who need it. A drifted cross-account trust relationship is the answer that does not survive that question.

What the Compliance Officer Actually Needs to See

The compliance question at audit is not "do you have policies" — every organisation has policies. The question is "prove that the access-control chain is intact." The evidence set for a defensible data lake is roughly this:

Bucket policies for every bucket in a version-controlled repository, reviewed on a documented cadence, with named owners
SCPs at the organisation root that make the worst-case bucket misconfigurations structurally impossible, tested with a policy-simulator run
KMS key policies with explicit principal scoping, no wildcards, and rotation cadence documented
Cross-account trust register with owner, expiry, and last-reviewed date for every external principal
Access-audit trail from CloudTrail or equivalent, retained per the regulatory floor for the data classification, queryable by DPO
Incident-response runbook for a bucket-exposure event, exercised on a documented schedule

The evidence set is not exciting. It is what turns "we take security seriously" into "here is the audit-chain query that answers your specific question." Under GDPR Article 32, the difference between the two is meaningful.

If your data lake is running across seven accounts and the last full audit of bucket policies happened eighteen months ago, which of the six misconfigurations above would the review find first — and would it be the one that turns into the headline?

FAQs

Is Block Public Access sufficient on its own, or do we still need SCPs?

Block Public Access is necessary and not sufficient. It stops the most-visible misconfiguration (a bucket policy that grants read to everyone), but a compromised administrative role can disable it. The SCP at the organisation root that denies s3:PutAccountPublicAccessBlock or its equivalent is what makes the Block Public Access setting stable against internal privilege abuse. Use both.

How often should cross-account trust relationships be reviewed?

Quarterly for standard partner engagements. Monthly for high-value data (personal data of children, financial records, credentials, unreleased financial results). Whenever a partner engagement ends, immediately, with the trust removed on the same day the engagement ends. The failure mode is not "we do not review often enough"; it is "we do not review at all", and any documented cadence with named owners is a meaningful improvement.

What is the cost of running the whole discipline versus doing the minimum?

The recurring cost is roughly 0.2-0.5 FTE of a platform-security engineer for a mid-sized data lake (single-digit accounts, tens of buckets). The one-time cost of getting the four control layers reviewed and the six misconfigurations checked is a two-to-four-week engagement. The cost of an incident that would have been prevented by the discipline is meaningful enough that the recurring cost is a rounding error against it.

Does this change if we run on GCP or Azure instead of AWS?

The primitives translate. GCS bucket policies + IAM + Organisation Policy + Cloud KMS is the GCP shape of the same four layers. Azure Storage account access policies + Azure RBAC + Azure Policy + Azure Key Vault is the Azure shape. The specific misconfiguration patterns show up in different forms — GCP's uniform vs fine-grained bucket access is a specific trap — but the underlying discipline is portable. If your lake spans multiple clouds, standardise the review cadence and the evidence set across them.

Companion content

DSPM Meets RAG: Data Classification for the Knowledge Base Era — the classification discipline that decides which bucket gets which policy
AWS Security Posture for AI Workloads — the AI-adjacent posture layer above the data lake
Streaming Data at PII Scale: Kafka, Schema Registry, and the Encryption Trade-Off — Part 2 of the data-security track, streaming rather than lake
The AI Act Enforcement Cliff: The August 2026 Deadline — regulatory pressure that sits above the data plane

How to engage

If your data lake has grown faster than the discipline that governs it, talk to us at creativeminds.dev/contact. The cmdev data-lake diagnostic runs the six misconfigurations against your specific configuration, produces a prioritised remediation list, and delivers the evidence set that survives a GDPR Article 32 or NDPA Section 39 audit.