Security in model-based systems

Prompt injection

Any text the system reads — email, a website, a document, a comment — may contain instructions. The model doesn't distinguish your instruction from text that looks like an instruction.

The rule: external content is data, never a command. Actions with impact derived from it require human approval or a hard rule in code.

Output as input

If the system runs, sends or stores what the model produced, treat the output as untrusted input: schema validation, character escaping, and an allowed-actions list. A model that generates a query shouldn't run it directly against a database with broad permissions.

A leak in the other direction

Sensitive information sent in the prompt, and retrieved chunks the user isn't allowed to see. Permission filtering must happen at the retrieval stage, and a system prompt isn't a secret — you can get the model to reveal it.