A Hong Kong insurer has an AI claims agent ready for production. It reads claim forms, checks policy terms and drafts settlement letters. The pilot worked. Then the security review asked one question nobody could answer: once a customer’s HKID number enters the agent’s memory, how do you get it out?
That question is now holding back AI agents across the industry, and the answer most security teams are reaching for is data tokenization. This guide explains what tokenization is, how it works inside an AI workflow, what it means under Hong Kong’s Personal Data (Privacy) Ordinance (PDPO), and how to approve AI projects faster without loosening control.
What is data tokenization in AI?
Data tokenization replaces sensitive values, such as HKID numbers, account numbers or customer names, with harmless substitutes called tokens before the data reaches an AI model, vector store or agent memory. The real values stay in a separate, locked-down vault, and only authorised systems or people can swap a token back.
A tokenized record still looks and behaves like data. An account number becomes another string of the same length, so systems, joins and analytics keep working. The model can reason about “customer TKN-4471” without ever seeing who that customer is.
One clarification saves confusion in board discussions. This is not the same as the “tokens” that large language models count when they bill you. LLM tokens are fragments of text. Data tokenization is a security control, and it has been used in card payments for more than a decade under the PCI DSS standard.
Why are AI agents making tokenization urgent in 2026?
AI agents are making tokenization urgent because they copy data into places that are hard to clean: prompts, vector stores, logs and shared agent memory. Security teams know this, so they slow approvals. EMA research found 82.9% of organisations delayed AI production over security or compliance reviews.
The research, The State of AI Friction, was conducted by Enterprise Management Associates and sponsored by Protegrity, a tokenization vendor, so read it with that in mind. Its findings still match what most Hong Kong IT directors report: 82.9% saw production delays, and 81.6% shipped AI with reduced capabilities, stripped of the data that would make it useful.
Speaking at Cloudera’s EVOLVE26 conference, Protegrity’s Diego Loureda put the problem bluntly to SiliconANGLE on 1 November 2026: once sensitive data sits in a model, a vector store or an agent’s memory that propagates to other agents, “good luck removing that sensitive data.”
Autonomy raises the stakes. Deloitte’s State of AI in the Enterprise survey, released on 29 October 2026, found 40% of enterprises now let AI execute decisions without human approval at least to a moderate degree, while 23% say accountability is unclear when AI produces a bad outcome.
How is tokenization different from encryption, masking and redaction?
Tokenization swaps a value for a substitute and keeps the original in a vault, so data stays usable and reversible. Encryption scrambles data mathematically and must be decrypted to be used. Masking hides part of a value for display. Redaction deletes it permanently. Each control fits a different AI risk.
--- Encryption: strong protection at rest and in transit, but a model cannot reason over ciphertext, so data is decrypted before use and exposed at that moment.
--- Masking: shows “A123***(7)” to a human; useful for screens, weak for AI because the hidden part is often what links records.
--- Redaction: removes the value entirely; safest, but the AI loses the ability to connect a claim to a policy or a customer.
--- Tokenization: consistent substitutes let the AI link records, count, group and reason, while the real identity stays in the vault.
Because there is no mathematical key to break, vault-based tokenization is also described as resistant to future quantum attacks on today’s encryption. That claim depends on how well the vault itself is protected.
How does tokenization work inside an AI agent workflow?
In an AI agent workflow, tokenization sits in the data path. Sensitive fields are tokenized when data leaves the source system, before it enters prompts, embeddings, tool calls or logs. The agent works on tokens. Only at the final step, and only for authorised users, are selected tokens swapped back for real values.
A typical five-step pattern
--- 1. Classify: tag which fields are personal or confidential, such as HKID, phone, account, salary and medical notes.
--- 2. Tokenize at source: replace those fields as data is extracted from the CRM, core banking or HR system.
--- 3. Process on tokens: the model, the vector store and every tool call see only tokens.
--- 4. Detokenize by policy: a claims officer viewing the final letter sees the real name; the agent and its logs never do.
--- 5. Monitor the vault: rate-limit and alert on bulk detokenization requests, the most likely sign of misuse.
Detokenization rights should be tied to identity. UD’s guide to AI agent identity covers how to give each agent its own scoped credentials, which is what makes vault policies enforceable.
What does tokenization mean for PDPO compliance in Hong Kong?
Under Hong Kong’s PDPO, tokenized data is usually still personal data, because the organisation holding the vault can re-identify individuals. Tokenization is therefore a strong security measure under Data Protection Principle 4, not an exemption from the Ordinance. It does, however, sharply reduce what leaks if an AI system is compromised.
The Privacy Commissioner’s AI guidance materials, including the 2024 Model Personal Data Protection Framework, ask organisations to minimise the personal data used by AI systems and to apply safeguards proportionate to risk. Tokenization is one of the most practical ways to show that minimisation in an audit.
--- DPP1 (collection): tokenization does not change what you may collect; it limits who and what sees it afterwards.
--- DPP3 (use): using tokens for a new AI purpose may reduce, but does not remove, the need to check consistency with the original purpose.
--- DPP4 (security): tokenization is direct evidence of “all practicable steps” to protect data processed by AI.
--- Cross-border transfer: sending only tokens to an overseas model provider keeps the identifying values in Hong Kong.
For the governance steps the PCPD expects specifically for agentic AI, see UD’s summary of the PCPD agentic AI guidance.
Which enterprise AI use cases benefit most from tokenization?
Tokenization pays off most where AI needs to link records about real people but does not need to know who they are: claims triage, customer service summaries, credit and fraud analytics, HR case handling and shipment exception management. In each case the model reasons on patterns, and identity is needed only at the end.
Financial services
A bank’s complaint-summary agent reads thousands of emails a week. With names, account numbers and HKID tokenized, the agent can group complaints by customer and product, and only the relationship manager sees who is behind each token.
Professional services
A law firm’s research agent compares clauses across past matters. Client names and counterparties are tokenized, so the firm can use a hosted model without placing privileged identities in a third party’s logs.
Logistics and property management
Delivery addresses, tenant names and phone numbers are tokenized before an exception-handling agent drafts customer messages. Detokenization happens in the messaging system, not in the model.
How can leaders approve AI patterns instead of individual projects?
Leaders can speed up AI approval by pre-approving patterns: a standard combination of data class, tokenization rule, model location and detokenization right. Any project that fits an approved pattern goes through a light check instead of a full review. Loureda’s summary: companies approving projects one by one “will not scale in the age of AI.”
A four-gate pattern approval framework
--- Gate 1, data class: public, internal, confidential, or personal data under PDPO.
--- Gate 2, protection rule: which fields are tokenized, redacted or allowed in clear.
--- Gate 3, model location: in-house, private cloud in Hong Kong, or overseas provider, with tokens only for the last.
--- Gate 4, detokenization right: which roles may see real values, logged and rate-limited.
A security committee that approves five or six such patterns once can then clear most new agent projects in days. The time saved goes to the genuinely novel cases that deserve full review.
What are the common pitfalls when tokenizing data for AI?
The most common pitfalls are tokenizing too late, after data has already reached logs or embeddings; ignoring free text, where names hide in email bodies; tokenizing so aggressively that the AI loses useful context; treating the vault as an afterthought; and assuming tokenized data is anonymous under the PDPO.
--- Too late in the pipeline: tokenizing at the model gateway misses copies already sitting in staging tables and debug logs.
--- Structured fields only: customer names in email bodies and scanned PDFs need entity detection, not just column rules.
--- Over-tokenizing: if every date and amount becomes a token, the agent cannot judge whether a claim is late or large. Tokenize identity; keep the facts the task needs.
--- Weak vault controls: the vault becomes the crown jewel. It needs its own access reviews, monitoring and disaster recovery.
--- False sense of anonymity: reversible tokens remain personal data. Your retention, access and breach procedures still apply.
What should you ask before choosing a tokenization approach?
Before choosing a tokenization approach, ask five questions: where in the data path it runs, whether it handles unstructured text, whether tokens stay consistent across systems, where the vault is hosted, and how detokenization is logged. The answers matter more than the brand, because a misplaced control protects nothing.
--- Placement: can it tokenize at the source system and in API payloads and tool arguments, not only in the database?
--- Unstructured data: how accurately does it find names and IDs in Chinese and English free text, including mixed-language emails common in Hong Kong?
--- Consistency: does the same customer get the same token in the CRM, the data warehouse and the vector store, so the AI can still link records?
--- Vault location: is the vault in Hong Kong or in your own environment, and who at the provider can reach it?
--- Audit trail: is every detokenization logged with the requesting identity, and can you set per-role volume limits?
A short proof of concept on one real workflow, with your own data classes and your own security reviewers, will answer these faster than any vendor presentation.
What should enterprise leaders do this quarter?
Start with one agent that is stuck in security review. Map where its sensitive fields travel, tokenize them at the source, and use that project to write your first approved pattern. Measure how long the next similar project takes to clear.
Tokenization will not remove the hard parts of AI governance, but it turns the security team from the department of no into the team that makes production possible. We understand AI. We understand you. With UD by your side, AI never feels cold.
Not sure which of your AI projects are blocked by data risk? Start with a three-minute readiness check, and we'll walk you through every step, from data classification and tokenization patterns to model selection and deployment.
Reviewed by the UD enterprise AI team. Statistics are attributed to their named sources; vendor-sponsored research is identified as such.