A Bangalore data-platform team runs Kafka in production. Fifty-two topics, four hundred million events per day, three hundred consumers across six business units. The security team's message on Monday morning is direct — the regulator is asking for a field-level answer to which consumer read which PII field in the last quarter, and the current audit surface cannot answer it. The platform team has three weeks. This is not a policy problem. It is an engineering problem, and the engineering choices around Kafka topic layout, Schema Registry discipline, and encryption tier decide whether the answer arrives on time and whether the answer is credible.
This piece walks through the three questions a Kafka platform at PII scale has to answer — which topics carry personal data, how the personal data is encrypted between producer and consumer, and how the audit surface proves who read what field when. Each answer is an engineering decision with a specific cost. The Bangalore team's cost numbers are in the piece.
Key takeaways
- Topic-level classification is table stakes. Every topic has a data-classification tag (public / internal / confidential / restricted-PII) and the tag drives every downstream decision — encryption tier, consumer ACL scope, retention policy, audit-log verbosity.
- The Schema Registry is where the governance surface lives. Field-level PII annotations in the Avro or Protobuf schema flow into the consumer-side field masking, the encryption-tier selection, and the audit log. Missing this discipline is where every governance programme silently fails.
- Broker-level encryption at rest is necessary and cheap; field-level encryption is expensive and sometimes necessary. The choice between them is a cost-versus-risk trade-off that the compliance officer and the FinOps lead should make together, not a decision the platform team makes alone.
- Client-side field-level encryption (CSFLE) via Confluent adds roughly 15-25% latency and 30-50% platform cost at high throughput. It also produces an audit-log surface that field-level Broker encryption cannot match. On restricted-PII topics, the cost is usually worth it; on internal-classification topics, it usually is not.
- The audit surface that proves which consumer read which field is the deliverable the regulator asks for. It requires field-level access tracking that only CSFLE (or an equivalent client-side pattern) produces natively. Retrofitting it into an unencrypted-at-field-level topology is the six-month project most teams under-scope.
Which Topics Carry PII
The first question is a governance question dressed as an engineering question. Which of the fifty-two topics carry personal data? Which fields, on which topics, are the ones the regulator will ask about first? The answer cannot be a spreadsheet the security team maintains manually. It has to live in the pipeline, next to the schemas, updated as the data model evolves.
The pattern that works — every topic gets a data-classification tag in the topic-configuration metadata (public, internal, confidential, restricted-PII). Every schema in the Schema Registry gets field-level annotations for each PII field. Every producer references the schema by its subject name; the schema carries the annotations; the annotations flow into the field-level encryption selection at the producer, the field masking on the consumer, and the audit-log verbosity at the broker. The single source of truth is the Schema Registry, not a wiki page.
The failure mode teams hit here is drift — the schema in the registry says a field is restricted-PII; the producer has an older version of the schema cached from three sprints ago; the field flows unencrypted. The fix is schema compatibility enforcement at the producer (fail-closed on missing annotation) plus a nightly reconciliation between the registered schemas and the versions the running producers reference. This is not exciting engineering. It is the engineering that decides whether the classification lives.
Broker Encryption Versus Field-Level Encryption
The encryption question has two independent answers that get conflated more often than they should. Encryption-in-transit (TLS between producer/broker/consumer) is table stakes; every Kafka deployment should have it, and the cost is negligible. Encryption-at-rest at the broker (via the KMS-integrated storage) is inexpensive and covers the case where the broker's storage volume is compromised.
Neither of those two answers the field-level question. If a consumer with topic-read permission reads a PII field they should not have accessed, broker encryption did not prevent it — the broker was doing exactly what it was supposed to do, and the PII flowed to the consumer's memory in plaintext. The audit log shows the consumer read the topic; it does not show which field they consumed.
Field-level encryption solves the field-level question. In the Confluent CSFLE pattern, the producer encrypts specific fields (identified via the Schema Registry annotations) with per-field data keys before the message leaves the producer. The message is written to the broker with the sensitive fields already encrypted. The consumer needs both topic-read permission and KMS Decrypt on the specific field key to decrypt the field. Every decryption is a discrete KMS call, which produces a discrete audit-log entry, which is what the regulator was asking for.
The cost is real. CSFLE adds roughly 15-25% latency on high-throughput topics (the encryption path is on the hot producer thread; the decryption path is on the hot consumer thread) and 30-50% platform cost when the increased CPU, KMS-call volume, and Schema Registry load are all counted. On a restricted-PII topic with fifty million events per day, that cost is worth it. On an internal-classification topic with forty million events per day and no PII, it is not. The classification tier decides the encryption tier; the encryption tier decides the cost.
The Cost Math the Bangalore Team Ran
Rough numbers from the Bangalore team's classification and encryption decisions across their fifty-two topics:
- Public and internal topics (thirty-nine of the fifty-two) — TLS in transit, KMS-integrated storage at rest, no field-level encryption. Marginal cost over baseline: negligible.
- Confidential topics without PII (nine topics) — same as above, plus tighter consumer ACLs and audit-log retention extended from thirty days to a hundred and eighty. Marginal cost: roughly $800/month across nine topics for the storage.
- Restricted-PII topics (four topics carrying customer identity, payment card data, and health-adjacent flags) — CSFLE via Confluent on the specific PII fields, per-field KMS keys, audit-log retention extended to seven years for the regulator's request-history window. Marginal cost: roughly $11,500/month across four topics — the KMS call volume dominates.
The four restricted-PII topics represent 12% of event volume and 62% of the incremental security spend. This ratio is normal at PII scale; the regulator's questions are almost entirely about the restricted-PII slice, and the engineering investment mirrors the regulatory attention.
The number that decided the trade-off — the cost of the CSFLE build plus the ongoing $11,500/month against the cost of a data-subject-access-request the platform could not answer — was not close. The regulator's question, unanswered, was a €10,000,000 or 4% global-revenue penalty under GDPR Article 83. The €138,000 annual security investment cleared that threshold by three orders of magnitude.
Building the Audit Surface
The audit surface at PII scale is the deliverable. Broker read-audit-logs answer the topic-level question — which consumer read which topic when. That is not the question the regulator asks. The question the regulator asks is the field-level version — which consumer read which PII field about which data subject when.
CSFLE produces the answer natively. Every field decryption is a KMS call. Every KMS call is logged with the caller identity, the key ARN (which is per-field), the timestamp, and — with a small amount of context propagation — the trace ID of the originating request. Three months later, the compliance officer runs the query against the KMS audit log and gets a signed answer.
Retrofitting this into a non-CSFLE topic topology is the six-month project most teams under-scope. The options are either to move the topic to CSFLE (feasible for future events, difficult for historical events already in the log), or to build a field-level access-tracking overlay via consumer-side instrumentation (fragile; the consumer's telemetry is what produces the audit, and consumers can lie). Neither is as clean as making the decision correctly at classification time.
What This Doesn't Solve
The three-layer discipline above does not solve the case where an authorised consumer legitimately reads a PII field and then misuses the value downstream. That is a data-loss-prevention problem at the consumer, not a Kafka problem. It does not solve the case where the Schema Registry itself is compromised — the annotations that drive encryption become attacker-controlled and PII flows unencrypted; the mitigation is registry access controls and change auditing that match the production topology's security posture. It does not solve the cost math for organisations at pre-PII-scale volumes where the fixed CSFLE overhead is disproportionate to the incremental risk.
It does solve the question of whether the regulator's field-level data-subject-access-request can be answered, and it does solve the cost model that decides which topics carry which encryption tier.
If your Kafka platform runs restricted-PII topics without field-level encryption and the regulator's next question is field-level, which of the fifty-two topics is the one whose answer would take you six months to build?
FAQs
Is field-level encryption necessary if the broker is in a private VPC and the consumers are all inside the same trust boundary?
Broker isolation reduces the attack surface but does not answer the field-level access question. A consumer with legitimate topic-read permission that reads a PII field it did not need is not a network attack; it is an authorised-but-inappropriate access, and the audit surface has to name it. Field-level encryption makes the audit surface field-scoped rather than topic-scoped, which is what the regulator asks for. The private-VPC posture is complementary, not substitutive.
What is the runtime cost of CSFLE at real production throughput?
Roughly 15-25% latency added on the hot producer and consumer paths for topics where encryption is used. Roughly 30-50% platform cost added end-to-end when the CPU overhead, the additional KMS call volume, and the Schema Registry load are counted. These numbers scale with throughput and with the number of encrypted fields per event; a high-throughput topic with one encrypted field is on the lower end, a moderate-throughput topic with eight encrypted fields is on the higher end.
Does the pattern change if we use Kinesis or Google Pub/Sub instead of Kafka?
The primitives translate. Kinesis has server-side KMS encryption at the stream level and client-side encryption via the KCL. Google Pub/Sub has customer-managed encryption keys at the topic level and client-side encryption via Tink or an equivalent. The Schema Registry equivalents (AWS Glue Schema Registry, Google Cloud Schema Registry) offer field-level annotations in similar formats. The engineering decisions and the cost math are the same shape. The specific implementation surface differs.
Can we skip the Schema Registry and manage PII annotations directly in code?
You can, and the cost shows up eighteen months later when the annotations have drifted across services and the compliance officer's question does not have a single source of truth. The Schema Registry as the single source of truth pays for itself the first time it does. The engineering cost of adopting it is a two-to-three-week project for a mid-sized platform; the cost of not adopting it is bounded above by the €10M-or-4%-global-revenue GDPR Article 83 penalty. The ROI is not close.
Companion content
- Data Lake Security: Bucket Policies, KMS Envelopes, and the Six Assumptions That Break in Production — the lake-adjacent piece; the data-security track opens here
- DSPM Meets RAG: Data Classification for the Knowledge Base Era — the classification discipline this piece assumes
- LLM Resilience Patterns, Part 1: Design — the reliability layer that sits above the streaming layer
- Coordinating the Agent — The Working Vocabulary, Part 2 — the coordination vocabulary agents use over Kafka-shaped streams
How to engage
If your Kafka platform has grown past the point where the security team can answer the regulator's field-level questions, talk to us at creativeminds.dev/contact. The cmdev streaming-security diagnostic classifies your topics, models the CSFLE cost against your traffic, and produces the audit-surface plan that lands in the specific weeks before the regulator's next request.