Building an AI Privacy Router

A few months ago, NVIDIA CEO Jensen Huang announced their version of NanoClaw, which promised to offer better privacy, security and performance compared to the popular open-source tool OpenClaw. OpenClaw, an agent framework with easy tooling, was mired with numerous security and privacy problems involving loss of sensitive data like keys, wallets and likely numerous breaches yet to be disclosed.

As part of the announcement, Huang mentioned a "privacy router." As far as I know, I was the first person to use the term privacy router in the context of AI/ML workflows, so I was both flattered and a bit surprised that the concept had grown clearly beyond my sphere. 😅

Since I've been thinking about privacy routing for AI workflows for a while, I thought it'd be useful to clarify what I've discovered so far and begin outlining where the concept could go.

If you are an audio-visual learner, I have a YouTube video covering a code example and the high-level architecture of an AI Privacy router.

Purpose and potential workflow

What exactly is an AI privacy router?

For me, an AI privacy router directs AI model workflows by evaluating data sensitivity and altering the path using specific security and privacy protocols. The privacy router is able to reroute and manipulate traffic based on informed decisions around privacy requirements.

Here are a few examples to illustrate the concept:

• Shifting between local models and external third-party models based on prompt and context data sensitivity
• Evaluating privacy vs utility and performance tradeoffs to make prompt routing decisions
• Routing on service preference or rankings (i.e. routing to region A over region B or vendor X vs vendor Y based on vendor agreements or legal advice)
• Many more... read on!

To better understand, let's investigate what a privacy router could look like and what parts might be interesting to your workflows and setup.

Anatomy of an AI Privacy Router

Let's walk through components you could use to make a router for your AI-based workflows (whether they are agent-based, assistant-based or simple LLM or other multimodal input tasks).

To give you a visual example of how the entire system could work, here is an example. It is not meant to be overly instructive on implementation or even architecture choices, but instead to give us an initial rough draft to use for our design and understanding.

An image showing a high level data flow for an example AI Privacy Router.

In this rough draft, there is a software developer on their computer sending out a prompt/user input. That input goes through an initial gateway server, which could manage credentials and caching, moves to a data classifier and creates three paths based on predicted sensitivity.

The highly sensitive path (red) routes to internal models. The medium sensitive path (yellow) routes to a data minimization method and perhaps either back to the classifier or to a trusted model provider. The not sensitive path (green) routes to an external model provider. Once the response comes from any of the models, it gets routed back to the developer with transparent notices on routing decisions.

Let's investigate how each of these potential components could work.

Traffic Classification

In networking, traffic classification is used to identify particular flows and forward them (or monitor them) based on their attributes. For privacy, this means determining potential sensitivity levels or identifying problematic information in flows.

Data classifier

This is essentially the crux of your privacy support. Can you build (or use) a classifier that can determine if the data is sensitive?

A more difficult question than building the classifier itself will be figuring out how to measure sensitivity. Ideally this is based on both your definition and better clarified by the tasks you are building. Let's take a real example.

Say you are using a coding assistant at work. You are using an external service for the coding assistant most of the time (like Claude or something similar). You have gotten approval to send the code back and forth to the coding assistant, but with the caveat to please not leak confidential business documents to the service.

Since coding assistants tend to grep lots of files in your working directory and send them as part of context, you might want to make sure your directory and any other directories you might use do not have confidential documents, but let's say you also might have some coding documentation that might leak such documents or minimally confidential business decisions.

Well, now you have a pretty accurate definition of a few things:

• Your ideal workflow
• The type of data that is allowed
• The type of data that should be blocked

Which means you have what you need to start building a data classifier! You could start small and quite brittle, by catching all PDF documents and requiring explicit approval. You could check against a list of filenames that are not allowed or you could parse the text and look for things like 'INTERNAL', 'CONFIDENTIAL'. These all might be your first iteration of the data classifier.

Note: In a well-documented and governed setup, these documents would already have information classification metadata noted in a catalog or document store. This would mean you could programmatically sort out documents that aren't to be used for LLM.

Over time, you might add more use cases or expand the definition and these brittle solutions won't work anymore. Then, you'd turn to building better logic, probably using machine learning to help learn the rules and labeled data to guide that learning. You could also use a local-running LLM as a Judge (more later) to evaluate the data and collect it to train your classifier. Of course here it should be transparent what data is being used, and there should be options to opt-out.

The classifier then makes a decision: is this data sensitive or not? If it's sensitive, it doesn't automatically forward it to the external endpoint.

A classifier can be of great assistance, but you might not yet have a good idea of how to classify data flows. The good news is there are some guardrail models that can help you get started.

External Guardrails

You know about the different types of guardrails available (external deterministic and algorithmic and alignment methods) from previous blog posts, but in summary, we have:

• External deterministic (software- and data-structure-based)
• External algorithmic (ML-based)
• Alignment Training / Fine-Tuning (in the model itself)

The first and second categories are things you can use for your router. Let's take a closer look.

Software-based Guardrails

These guardrails could run filters on text directly. Similar to the initial workflow described above, you could look for regularized text that says "internal only" or "confidential". Additionally, there are a variety of regular-expression based parsers that look for common patterns like credit card numbers, email addresses or other semi-structured text data.

These could then "flag" if something is matched; which would trigger a router decision or shift in the traffic pathway. Note that deterministic filters are brittle and will need to be maintained and updated over time; however they are easy to test and usually fast to run.

Algorithmic Guardrails

LlamaGuard is an open-weight model that also has a category for finding potential prompts that violate privacy. Recently OpenAI released their own open-weight model focusing specifically on PII detection and removal.

Like all algorithmic systems, these will not catch everything and are not to be used as a "privacy cure" for all prompts; however, they are one step towards potentially catching more nuanced problems than the deterministic software.

If you already use "LLM as a Judge" workflows, you can leverage that as an external algorithmic guardrail. I have a video and a Jupyter Notebook with examples.

If you use LlamaGuard (or something similar) or LLM as a judge, I recommend you start writing tests that demonstrate the examples you want to catch and to regularly test them. Via this method, you can start building out a larger evaluation suite based on those tests. That way, when new guardrail models are released or you want to update your LLM-model or prompt for the judge, you can test it and compare. I'll be posting a longer article on designing privacy evaluations soon, so stay tuned!

These tests can help you build out a dataset which enables you to train your own privacy-evaluation-models. Often self-trained models fit your use case better than any third-party model would because they can fit your data and use cases better than a generalized model. Start out with a small model (like a simple text classifier or document classifier) and compare it with your LLM-based or other model-based flow.

As you build out your training data with more examples and information, you can train a bigger model and compare again. Your custom model can therefore become more expansive as you get more use cases or examples of what you'd like to block. It also will save you latency/processing time when compared with an LLM as a judge or LlamaGuard (presuming it performs as well).

For whatever type of guardrail you implement, if you find a match, you'll then change the routing behavior. This could mean:

• removing problematic text and continue forwarding (especially for deterministic finds)
• flag requests that contain sensitive data and reroute to internal models
• fail and ask for a new prompt
• tag the flow and log it for follow up and auditing
• many more options...

Note you'd want to test each of these behaviors and review it with affected teams long before deploying your router so that users are not confused or annoyed. Blocking flows by default will lead to annoyance and people deciding to simply use a VPN and go for a less safe third-party model, so use that option carefully. :)

Edits in the loop

So far you've explored how you can either quickly classify or flag input and change the response or forward based on potential privacy issues in that input. But what other protections can you offer in order to achieve the best privacy v. utility tradeoffs?

Let's review a few options that could help you reduce information while keeping response accuracy high.

Data minimizer

Minimizing data is a big part of what you work on in privacy, but it can be tricky with deep learning systems. There's a tension between the amount of information the model needs to make a decision and the amount of information you might want to give the model based on your privacy concerns. Ideally, you can find a sweet spot where both needs are met.

When applying data minimization, you are experimenting with this balance between privacy and utility. One easy initial minimization is to find potentially sensitive text or other input and pseudonymize (i.e. redact, mask or alter) that information, so it is somewhat protected, but not completely gone.

I have a YouTube explainer and some example code on using Microsoft Presidio for this, or also comparing that to using an LLM as a Pseudonymization engine (code).

But pseudonymization is a pretty basic option and will not catch everything (as shown in the code notebooks). Are there other ways we can minimize data that might be sensitive?

Another way for us to look at minimization and this tension is to experiment with redaction and summarization as an intermediary step. In this design, the minimizer would work closely with the classifier to redact and remove information until the classifier says that it is no longer sensitive. You could try this approach to both improve your classifier and your minimizer. Ideally, you'd get the system to a place where it would work in sync without having to iterate many turns (especially in a production setup).

You can also minimize-by-design by investigating task-specific models, let's take a look.

Task-specific models

Sometimes you don't need a general model when a task-specific model would be both less invasive and also offer higher performance. Task-specific models are models trained on a specific task. Even today's general models (like VLMs) are to some degree "trained to task"; that task being taking images and text and responding in text.

There's a wide variety of models that are much clearer about the task. Just taking a look through Hugging Face categories shows you a view into the model landscape:

Let's imagine a new router design. In this case, you want to transcribe a project's meeting audio and categorize meeting notes for an internal knowledge base. Yes, you could use a third-party "do everything" model, but you could also break down the task into several steps:

1. Audio-to-text: This task is fairly well-defined. The most performant audio-to-text models don't do anything else and are well within the size and space that they can be run locally or minimally on an internal server that belongs to the company.

2. Powerpoint or Slides to text: This can be done in many cases with just simple document parsing (no ML needed). You could combine this with something that takes a "screenshot" of each slide and saves it as an image if you want.

3. Categorization: You probably want to define project categories or aspects. There are many small NLP models that can perform this task. It's also useful to have a human approve categories, so this is a good chance to have a human-in-the-loop to properly document the categories and maybe add some global links to specific project outcomes or deliverables. A human would check the model outputs until the model is running appropriately.

4. Summarization and Indexing: These are well-known NLP tasks where models vary in size and purpose, but many of them can run on a single computer and perform just as well (if not better) than LLMs trained on a variety of tasks. This is because they are trained to be less verbose and not add unnecessary additonal information.

So, in this case, you might break down your workflow into those steps and eventually leverage a more general AI router to make such decisions. In this setup, you would be able to tag workflows with a specific task, telling the router, I would like a categorization model for this step. The router could then route the query to the appropriate model and leverage a task queue depending on if you want to chunk your requests.

Note: Many agentic frameworks attempt to do this, but usually just by shifting prompts and with rote skills. If you want to build real power into your workflows, it means also thinking deeper into what basic and task-specific models can add which you can expand and adapt as your router matures.

With each of these task-specific models, it would mean investing in and designing for standardized and repeatable workflows for the organization. Ideally these are optimized for product and engineering needs and business priorities. Chunking the work means you can test out new models or diagnose problems one at a time instead of wondering where the system is breaking (or why your token costs have exploded).

From a data minimization aspect, smaller self-hosted or internal-cloud-hosted models have a strong win for privacy; both because they use less information to make decisions and store less information, and because they are well under the organization's control and agency. Of course, when working on someone else's data this needs to be clarified with the appropriate responsible parties, just like any other data processing!

Now, let's say you are using an LLM or VLM workflow and want a router that is general purpose without having to break it down into smaller tasks, is there anything you can do for prompt minimization beyond pseudonymization?

LLM as rewriter

Your results and workflows may vary, but one design pattern you could use is determining how an LLM in the loop can help rewrite queries to maintain information but reduce specificity.

In this case, you might first perform more deterministic pseudonymization to redact sensitive text and then prompt an LLM to look for other potentially sensitive information and either redact or reword that information. This is where it can get tricky because it's important to remember that an LLM is not trained on that task and might perform erratically. Therefore, this is not recommended without thorough testing and tuning to your use cases and workflows (and the performance still can change after you test it!).

However, you can imagine a topic summarizer that can eventually be built based on taking samples of these types of flows and starting to cluster them. Eventually your LLM as a rewriter can perform in some ways like a prompt classification service, similar to Google query suggestions (e.g. "Did you mean X?").

If you can get your system to that point the LLM as a rewriter essentially becomes a known-prompt router that can not only cache results that have been partially designed by a human, but also provide much more privacy for each query, by turning free-text into known-categories.

Context summarizer

Context is on everyone's mind lately, since it's essential for token usage, model performance and for privacy and security! Context can leak extra information that influences model output in a way that is both unnecessary and a potential privacy violation.

One way of managing context for both token use and for better privacy is to minimize (or "compact") context every few iterations. This can mean setting a context limit, where once it is reached the context is minimized or using other methods (like deterministic guardrails) to trigger minimization. You could also ground minimization decisions on flow classification (see above "traffic classification" section).

Depending on the type of flow, you might choose different methods for minimization. One obvious choice is to use summarization as a way to minimize context. This can either use an LLM-in-the-loop or a summarization task-specific model, like XSum models or even one that you develop yourself based on traces that you save and summarize using a service or by hand.

I walk through comparing different summarization methods on Probably Private YouTube with code examples and comparisons if you want to learn more.

The privacy goal should align with other model and product goals, so you'll want to ensure your summarization doesn't remove too much information for a useful output. You can also align this with financial goals on token spend to find a happy medium between the different requirements.

Performance concerns

If you are inserting something like a router into an architecture. You want to ensure you're not adding unnecessary latency while still doing the job appropriately.

Let's review a few ways to ensure top performance while also enhancing privacy.

Start small and shadow

Initially you'll want to start small and fast. This means having a minimal example of a privacy router which can be released as something like a lambda function or a microservice. If you have multiple zones, you can test it during a known quiet period, in a particular zone, for a selected set of users so long as you are okay with impacting a small percentage of users.

If impacting users is unacceptable, use frameworks for dark traffic collection, which lets you record and replay real production traffic on your router prototype. With that setup, you can evaluate the new architecture behavior and performance outside of the critical path. You should observe the impact on flows and performance and adjust based on your learnings.

There's also a common architecture when introducing new models into production setups called shadow or canary model deployment.¹ This means that if you deploy an algorithmic guardrail as a part of your setup you first allow all traffic to pass through normally, but you record and observe how the model would have changed the flow. In a privacy router case, you would also observe introduction of latency and results (i.e. did it perform to your expectation?).

An example routing canary flow testing a new candidate model. The box around the canary model demonstrates that it should not impact real production decisions during testing.

Only after thoroughly testing in increasingly busy periods do you actually start rerouting traffic using your router. Depending on the maturity of your infrastructure and release, it'd be good to release with feature flags to observe changes in behavior and performance.²

You need to have a way for infrastructure teams to "degrade" or remove the router if it's introducing problems. This could mean having the smallest, fastest router always available as its own service and switching to that if you have a DDOS or other performance problem with your privacy router.

Tip from Sandy Strong, infrastructure expert and leader: Degradation options can range from easy-to-implement manual toggles (like switching a light on and off) to graceful mechanisms built into the software stack that guarantee deterministic, controlled degradation based on system metrics (e.g. when latency exceeds a certain threshold, for 95% of requests, fall back to the smaller, faster router, when latency is healthy again, promote traffic to the more expensive router). Sometimes these are called self-healing systems, using concepts from control flow theory.

Depending on your setup, this probably means introducing more CI/CD than you currently have. It's a necessary investment for maturing your privacy and security posture overall and the AI maturity of your organization.

Caching and Cache-Matching

Caching is a smart idea for busy setups, even without privacy concerns. There are already several LLM routing projects that offer caching to reduce token spend (with complete match or near match settings).

For your router, you might start by caching recent router decisions and accessing that first in the normal way you do with LLM-serving or whatever interface you use for external LLMs. Look into if whatever framework you are using already allows easy insertion and access to cache.

Aside from that, if you have a gateway, you can potentially insert directly from your router into gateway caching (i.e. static response for IP address X or region Y).

As you determine how caching works for your setup, you'll likely get into more complicated topics, like expiration, allowing cache bypassing and determining the exact parameters for caching that work for your different AI flows. At that point you're moving it into infrastructure engineering which can help the privacy router function better for your production environment.

Sandy's Tip: Take advantage of your infrastructure team's vast experience in building, maintaining, and operating cache-backed application stacks. They can provide options for technology choices and explain best practices, such as data structure (key and value structures), cache client design, data access patterns (reads/writes), cache sizing/geometry, replication factor, TTL, maintenance considerations, regional location choices, load and stress testing, and disaster recovery strategy.

You should let your infrastructure team lead on such decisions and involve them in every step, from initial design to final deployment. This makes it ready for actual production and ensures it fits their standards and specifications, follows best practices, is maintainable and has an operations plan.

Develop metrics and testing to guide decisions

If you don't already have platform and product metrics that help guide service architectures and decisions, now is a good time to start thinking through these and testing ways to measure and evaluate performance.

This involves looking at already collected model performance data and current error and failure rates. It also means thinking through new metrics for attributes like privacy. To begin, you could red team your systems for privacy and find baseline failure rates or establish trace sampling to observe today's privacy mistakes in your active system.

Once you establish a better understanding of your baseline and current requirements, you can set initial performance requirements for the router setup or determine reasonable performance bounds for router integration. Defer to your product and infrastructure teams needs to clarify and ensure you are not blocking other teams from delivering their services w.r.t. their own acceptance criteria.

Sandy's Tip: Clarify who will be on-call for the new stack and how they will get trained. Create a service runbook and ensure operating procedures are documented before launch.

Ideally you start thinking about privacy audit metrics you can track and integrate. Since integrating privacy will almost always involve tradeoffs, establishing both technical and legal understandings of your organization's privacy "minimum". This will guide what your first router (and iterations) look like and how they achieve an acceptable balance.

Building something like your router into AI platform management is a longer-term plan to allow for ease of use and ability to adequately test new iterations.³ Eventually you might serve several versions of the router depending on the sensitivity or location of the different services. As your architecture, AI and platform maturity grows, so will your privacy engineering.

Putting it all together

You've reviewed some fresh ideas for integrating better privacy into your AI/ML workflows, and I hope left with some new thinking around how to route for privacy. You could in many ways apply this thinking to other difficult problems in AI workflows, such as security engineering.

If you're just starting out, start small and iterate. Hold a workshop for multidisciplinary stakeholders to begin conversations. Inform yourself as to the understanding of the data flows and overall data governance. Meet your organization where they are at and go from there.

If you already have a quite mature setup, begin with some privacy testing and evaluations as part of your CI/CD. Find common sensitive data flows and start with a minimal viable router for those flows specifically. Run a privacy red teaming exercise to identify potential risks and failure states.

If you need any help with the above exercises (both starting out and mature), I offer advisory, workshops and trainings.

Remember that privacy engineering mirrors organizational and data maturity, so stretch yourself but be prepared for a long journey. Every step towards offering better protection and more transparency around privacy is a step in the right direction. Keep on your path!

If you end up building a router or posting anything about your work on this topic, I'd love to hear from you.

Acknowledgements: Many thanks to Aaron Glenn, Sandy Strong and Noah Swartz for their feedback, driving questions and expertise that helped me write and greatly improve this post. Any mistakes are my own.

---

If this post helped you, consider subscribing to my newsletter or my YouTube and sharing my work. I also offer advisory and workshops and a new Practical AI Privacy course on topics like security and privacy in AI/ML and personal AI.