DEFAULT_NUM_CTX additions

adding further changes for DEFAULT_NUM_CTX, including docs
This commit is contained in:
Aaron Bolton
2024-11-19 07:46:51 +00:00
parent bfaaf86c69
commit 5359423481
4 changed files with 26 additions and 3 deletions

View File

@@ -1,4 +1,7 @@
# Contributing to Bolt.new Fork
## DEFAULT_NUM_CTX
The `DEFAULT_NUM_CTX` environment variable can be used to limit the maximum number of context values used by the qwen2.5-coder model. For example, to limit the context to 24576 values (which uses 32GB of VRAM), set `DEFAULT_NUM_CTX=24576` in your `.env.local` file.
First off, thank you for considering contributing to Bolt.new! This fork aims to expand the capabilities of the original project by integrating multiple LLM providers and enhancing functionality. Every contribution helps make Bolt.new a better tool for developers worldwide.
@@ -80,6 +83,19 @@ ANTHROPIC_API_KEY=XXX
```bash
VITE_LOG_LEVEL=debug
```
- Optionally set context size:
```bash
DEFAULT_NUM_CTX=32768
```
Some Example Context Values for the qwen2.5-coder:32b models are.
* DEFAULT_NUM_CTX=32768 - Consumes 36GB of VRAM
* DEFAULT_NUM_CTX=24576 - Consumes 32GB of VRAM
* DEFAULT_NUM_CTX=12288 - Consumes 26GB of VRAM
* DEFAULT_NUM_CTX=6144 - Consumes 24GB of VRAM
**Important**: Never commit your `.env.local` file to version control. It's already included in .gitignore.
### 🚀 Running the Development Server