As of now, you can only set the policies on a per-request basis. We are planning to add settings to the Model Settings page at some point in the future.
Policies
Max cost (for the whole request)
This feature is not yet available!
Maximum number of times to retry a particular model
0
, the system won’t retry the request to this model. In general, there will be up to N+1
LLM calls performed to the same model (the original, and the retries.)
To be used in combination with max_switch_model_retries
The maximum number of requests (worst-case) will be
(max_same_model_retries+1) * (max_switch_model_retries+1)
For requests that require multiple LLM calls, the settings are applied to each of the intermediate requests! This can result in a high number of tokens used.
Maximum number of different models to try
0
, the system won’t use any other models. In general, Pulze will try the request with N+1
different LLM models (the best model, and one for each “retry”.)
To be used in combination with max_same_model_retries
The maximum number of requests (worst-case) will be
(max_same_model_retries+1) * (max_switch_model_retries+1)
For requests that require multiple LLM calls, the settings are applied to each of the intermediate requests! This can result in a high number of tokens used.
Privacy level
- (default) Store the
prompt
,response
, and all the metadata associated with it (labels, weights, costs…) - Store all the metadata, but the
prompt
and the LLM’sresponse
will not be logged in any way. The log is still visible, labelled, and searcheable. - (stealth mode) The log is not stored[*], not visible, not searchable, and no
prompt
,response
or labels are stored.
[*] Internally we must log the datetime and costs incurred, which we require for billing.
Prompt
Example
You can set any, some, or all the policies on every request
Pulze-Labels
header, the policies will be stored as part of the Labels in this format:
This will only happen if
Pulze-Labels
are defined.Send {}
if you don't want to send any particular Labels, but want the Pulze-Weights
and Pulze-Policies
to be auto-saved.
Ignoring the Pulze-Labels
header will not store the other header information as part of it -- essentially, ignoring the Pulze-Labels
header is like saying "I don't want labels".Note that
privacy_level
isn’t stored at all