Setting up Language Models in config.json
To use various Language Models (LLMs) in your project, you need to configure them in the config.json file. This file should be located in the root directory of your project. Here's how you can set up different LLMs in the config.json file.
Basic Setup
The config.json file should have the following structure:
{
"language_model": {
"provider": "litellm",
"enable_observability_logging": true,
"redis_enabled": true,
"models": [
// Model configurations go here
]
},
"github_app": {
"check_signature": false,
"auto_pr_review": true,
"edit_pr_desc": true,
"process_on_push": true,
"auto_unit_test_generation": false
}
}The language_model object contains the configurations for the LLMs you want to use. The models array inside it is where you define the configurations for each LLM.
Example: Open AI models
First, you need to add OPENAI_API_KEY in the .env file.
Here's an example of how to configure a LitELLM model:
{
"model_name": "default",
"litellm_params": {
"model": "gpt-4o-mini",
"input_cost_per_token": 0.000000015,
"output_cost_per_token": 0.0000006
}
}In this example, we're defining a model named default that uses the gpt-4o-mini model from LitELLM. The litellm_params object specifies the cost per token for input and output.
Example: Azure OpenAI Model
For this example, you need to add the AZURE_API_BASE and AZURE_API_KEY in the .env file.
To configure an Azure OpenAI model, you can use the following structure:
{
"model_name": "CUSTOM_MODEL",
"litellm_params": {
"model": "azure_ai/MODEL_NAME",
"api_key": "os.environ['AZURE_API_KEY']",
"api_base": "os.environ['AZURE_API_BASE']"
},
"model_info": {
"max_tokens": 4096,
"input_cost_per_token": 0.000015,
"output_cost_per_token": 0.000015,
"max_input_tokens": 128000,
"max_output_tokens": 4096,
"litellm_provider": "openai",
"mode": "chat"
}
}Example: Groq models
For GroQ, you need to add the GROQ_API_KEY in the .env file.
To configure a GroQ model, you can use the following structure:
{
"model_name": "default",
"litellm_params": {
"model": "groq/llama3-8b-8192"
}
}