The KNN Router is an open-sourced intent-tuned LLM router designed to efficiently select the best language model (LLM) for a user’s query. This documentation provides an overview of how to deploy and utilize the KNN Router, as well as insights into its underlying technology.
Semantic Query Routing:
Weighted Scoring:
Minimal Latency:
Integration Flexibility:
Open Source:
The KNN Router leverages the pulze-intent-v0.1 model, which is trained to select the most appropriate LLM based on user queries. The model and dataset can be accessed via the following links:
The KNN Router can work with a variety of models, including:
To deploy the KNN Router locally, follow these steps:
Fetch Artifacts from Huggingface:
Start the Services:
Query the Router: Use curl to send a query:
The expected output will include a ranked list of hits and scores for each target, similar to the following:
For Kubernetes deployment, refer to the provided example configuration.
Before deploying, you need to generate the required artifacts:
points.jsonl
: Contains points and their respective categories and embeddings.targets.jsonl
: Contains targets and their respective scores for each point.Generate Artifacts: Use the following script to generate the required artifacts:
We encourage developers and enthusiasts to contribute to the KNN Router project. Your feedback and contributions help us improve the tool for everyone.
For more information and to stay updated, visit our GitHub repository: KNN Router on GitHub.
We have plans for additional features and enhancements, including:
Stay tuned for updates!
For questions, feedback, or support, please reach out through our community channels or directly on GitHub.
The KNN Router is an open-sourced intent-tuned LLM router designed to efficiently select the best language model (LLM) for a user’s query. This documentation provides an overview of how to deploy and utilize the KNN Router, as well as insights into its underlying technology.
Semantic Query Routing:
Weighted Scoring:
Minimal Latency:
Integration Flexibility:
Open Source:
The KNN Router leverages the pulze-intent-v0.1 model, which is trained to select the most appropriate LLM based on user queries. The model and dataset can be accessed via the following links:
The KNN Router can work with a variety of models, including:
To deploy the KNN Router locally, follow these steps:
Fetch Artifacts from Huggingface:
Start the Services:
Query the Router: Use curl to send a query:
The expected output will include a ranked list of hits and scores for each target, similar to the following:
For Kubernetes deployment, refer to the provided example configuration.
Before deploying, you need to generate the required artifacts:
points.jsonl
: Contains points and their respective categories and embeddings.targets.jsonl
: Contains targets and their respective scores for each point.Generate Artifacts: Use the following script to generate the required artifacts:
We encourage developers and enthusiasts to contribute to the KNN Router project. Your feedback and contributions help us improve the tool for everyone.
For more information and to stay updated, visit our GitHub repository: KNN Router on GitHub.
We have plans for additional features and enhancements, including:
Stay tuned for updates!
For questions, feedback, or support, please reach out through our community channels or directly on GitHub.