OpenAI
The following syntax is appropriate for a secret that authenticates to an OpenAI-compatible inference endpoint.
type: openai
secret:
token: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
allowed-base-urls:
- https://api.openai.com/v1
The openai secret type stores an API token together with the endpoints
Fuzzball may send that token to. Both fields are required.
fuzzball secret update NAME --edit prepopulates the existing
allowed-base-urls so that changing only the token leaves them as they were.
Whatever the editor saves replaces the secret, so an entry deleted there is
deleted from the secret.
Unlike a token you send yourself, this credential is spent by Fuzzball on your behalf: the agent’s inference calls are proxied through Orchestrate. The Fuzzball CLI and web UI never show the token, and a workflow cannot read it.
allowed-base-urls bounds where Fuzzball itself sends the token: a registered
model whose base URL the list does not cover is refused before any request is
made, so a mistyped or malicious model registration cannot redirect the token
to somebody else’s endpoint. It does not limit anyone who can read the secret:
everyone its scope makes it available to and, for a group or
organization secret, the organization’s owners. On an operator deployment, any
of them can retrieve the token from inside the cluster. Scope the secret to
people you would trust with the token itself.
What is compared is the base URL of the model that uses the secret, not the
full endpoint Fuzzball then calls beneath it. A base URL is allowed when its
scheme, host and port match an entry exactly, and its path sits at or below the
entry’s path at a segment boundary. So write each entry as the base URL itself,
or a prefix of it, such as https://api.openai.com/v1, and not as a full
endpoint such as https://api.openai.com/v1/chat/completions: a model whose
base URL is https://api.openai.com/v1 does not sit below that.
Given an entry of https://api.openai.com/v1:
| Base URL | Allowed | Why |
|---|---|---|
https://api.openai.com/v1/chat/completions | yes | below the allowed path |
https://api.openai.com/v1 | yes | exact match |
http://api.openai.com/v1 | no | the scheme differs, and the token would be sent in clear text |
https://api.openai.com:8443/v1 | no | the port differs |
https://api.openai.com/v2 | no | a sibling path, not below /v1 |
https://api.openai.com/v1beta/x | no | /v1beta is a different path, not a segment below /v1 |
https://api.example.com/v1 | no | a different host |
An entry with no path, such as https://api.example.com, covers every path on
that host. Use one only when you intend the token to be usable anywhere on it.
A requested URL containing . or .. path segments is refused outright. Those
are not resolved before matching, so https://api.example.com/v1/../admin
would otherwise satisfy an entry of /v1 and then arrive at /admin on any
server that normalizes it, defeating the containment the path is there to
provide. A ; parameter does not hide one either: /v1/..;/..;/admin is read
as .. here, because some gateways strip ;params before normalizing. Write
the path you mean.
Also refused in the requested URL: a backslash anywhere in the path, a percent
sign still present after one round of decoding, and any byte outside printable
ASCII. Most of these are ways of spelling a dot segment that only becomes one
further along — \..\ on a server treating \ as a path separator,
%252e%252e on one that decodes twice, and %C0%AE%C0%AE, an overlong UTF-8
encoding of .., or %EF%BC%8E, its fullwidth form, on one that normalizes
Unicode. A control byte is the exception: it never becomes a dot segment, it
truncates or reframes the path for whatever mishandles it, NUL being the common
case. None of them is reasoned about — a base URL path Fuzzball will accept
is printable ASCII.
A model whose base URL has such a path is refused when you register it, so this
surfaces at fuzzball model add, not as a failed turn later.
An allow-list entry must itself be printable ASCII, with no percent sign. Write
an internationalized host in its punycode (xn--) form, and the path
unencoded. A Unicode host is compared with case folding but reached through
IDNA, and the two disagree for a few characters, so a Unicode entry such as
straße.example could also let the token reach strasse.example. A
percent-encoded entry is compared after decoding, so %2F in it would split
one path segment into two. Both are refused when the secret is saved, and a
non-ASCII host never matches, so write a model’s base URL host in punycode
too. A requested URL whose path contains an encoded slash (%2F) never
matches either, and neither does one whose path contains * or an empty
segment (//). This leaves three things an allow-list cannot express: a path
segment that needs percent-encoding, such as one containing a space, a path
containing * or //, and an IPv6 zone. An entry with no path still covers a
base URL whose path needs percent-encoding.
The port a URL addresses is what is compared, not the text of it, so writing a
scheme’s default port explicitly makes no difference — https://api.openai.com/v1
and https://api.openai.com:443/v1 are the same endpoint and match each other,
in either direction. Any other port is a different endpoint, which is the point
of comparing ports at all. A port is compared as a number, so :0443 is the same
as :443. The same holds for IPv6 literals, with or without
brackets: https://[::1]/v1 matches https://[::1]:443/v1.
In a YAML secret file, allowed-base-urls is a list. In the web interface it is
a text box with one URL per line, and entries are trimmed. A comma followed by
a space also separates entries, so
https://api.openai.com/v1, https://eu.api.openai.com/v1 is two entries, while
a comma with nothing after it stays part of its URL. A value consisting only of
separators is rejected on the form rather than accepted and refused later by
the API. fuzzball secret show prints the entries separated by a comma and a
space, the same way, so its output can be pasted back into the form.
An entry is refused if it carries a username or password, a query string, a
fragment, whitespace anywhere in it (including at either end), a percent sign,
any character outside printable ASCII, or a scheme other than http or
https. Its host must be a hostname or an IP address: there are no wildcards,
so list each host. Its path may not contain *, an empty segment (//), a
. or .. segment, or a backslash, none of which any request could match.
Hosts are compared exactly as written, apart from letter case, so list each spelling you use: a host with and without a trailing dot, or two forms of the same IPv6 address, are different entries.
An entry is stored exactly as given, and the web interface both trims entries and treats a comma followed by a space as a separator, so an entry containing whitespace would change the next time the secret is edited, even when only the token is rotated.
The allow-list is the non-sensitive half of the secret: unlike the token, it is
returned by fuzzball secret show and the web detail panel to everyone the
secret is shared with, which at group or organization scope is not only the
person who created it. And matching compares scheme, host, port and path
only — so none of those three components could ever widen what an entry
permits, while every reader could see them. That is why they are refused, and
an entry carrying one that was stored anyway matches nothing.
This is the one place an entry differs from a model’s --base-url, where a
query is allowed because Azure OpenAI needs ?api-version=. A base URL is a
URL Fuzzball sends; an entry here is a URL Fuzzball matches. Write the
Azure endpoint without its query:
allowed-base-urls:
- https://x.openai.azure.com/openai
That entry permits the request your model actually makes, query and all, because
the query is not part of what gets compared. A trailing bare ? or # with
nothing after it is accepted — there is nothing in either to disclose.
Fuzzball sends the token as Authorization: Bearer <token>, and never
refreshes it. Azure OpenAI expects an API key in an api-key header instead,
which Fuzzball does not send, so the endpoint rejects it; and a Microsoft Entra
ID token, which Azure does accept as a bearer token, expires within hours. Reach
Azure OpenAI through a gateway or proxy that accepts a long-lived bearer token
and forwards the request with Azure’s own credential.
Like other secrets, an openai secret may be created at user, group or
organization scope, and a cluster administrator can also create one at cluster
scope. Only a cluster-scoped model can use a cluster-scoped openai secret,
and Fuzzball reads it for every user of such a model, so it is spent on behalf
of everyone who can use the model. Scope it as narrowly as the people who
should be able to spend it: inference can be billed against the token by anyone
who can use the secret (for a group or organization secret, the organization’s
owners included), and, as above, anyone who can read it may be able to retrieve
the token itself.
Orchestrate sends inference over https:// only. A model endpoint with an
http:// base URL is refused when a turn uses it, and the turn fails with a
message saying that plaintext inference is disabled on the cluster. The token
would otherwise cross the network unencrypted.
For an in-cluster endpoint on a trusted network, a cluster administrator can allow plaintext on the Fuzzball custom resource:
spec:
fuzzball:
agent:
model:
allowInsecureInference: true
Outside an operator deployment, setting CONVERSE_ALLOW_INSECURE_INFERENCE=true
in Orchestrate’s environment does the same. Either way the setting covers the
whole cluster, and Orchestrate logs a warning at every startup while it is on.
Inference verifies https endpoints against the system’s certificate
authorities, plus one more CA when Orchestrate is told about it, so the agent
can reach a model served at one of the cluster’s own endpoints when those are
signed by a private CA:
- Operator deployments need nothing: the operator adds the cluster’s internal and ingress CAs, including a custom CA, to the trust bundle every Fuzzball pod mounts.
- Docker-compose deployments serve endpoints with nginx’s certificate. The
stack points
CONVERSE_TOOL_CA_FILEat that CA, and inference uses the same one. - Anywhere else, set
CONVERSE_INFERENCE_CA_FILEin Orchestrate’s environment to a PEM file holding the CA. When it is unset, inference usesCONVERSE_TOOL_CA_FILE; when it is set, it is used instead of that one, not as well, so it must hold every private CA inference needs.
The CA is added to the system’s, so hosted providers keep working. A file that cannot be read, or holds no certificate, is logged at startup and ignored; verification is never turned off.
Inference never follows a redirect. Its errors are fixed phrases: of what the provider sends back, only the HTTP status and its error type or code, when that is one of a fixed list of known values, reach a user or a log, never its text. When an inference call fails, the log names the model and the endpoint’s scheme and host, with the token removed. Other messages that name a base URL drop its userinfo, query and fragment but keep its path, so never put a credential in a base URL.