hello maintainers 👋
keeping it short to not waste a lot of ur time. I am a contributor to cncf kgateway and I wanted to benchmark its inference routing capabilites, for which I was checking out your benchmarking tests.
I saw that for nightly tests, we are using same env (dataset, vllm configurations, etc.) compared to there corresponding benchmarking scenarios. e.g. nightly Prefill Heavy has same routing aim (stress EPP under KV-cache / prefill saturation), same dataset type as the main benchmarking prefill heavy scenario.
but for multi LoRA - Regression multi lora uses Infinity-Instruct_conversations.json while the similar Nightly Multi-LoRA uses billsum_conversations.json (prefill-heavy), I want to know why was this choice made? if we want manual regression testing or automated nightly testing for both the cases, then we could have done both in regression tests or in nightly tests, why one in manual regression and another in nightly?
please help, I am not able to get this.
hello maintainers 👋
keeping it short to not waste a lot of ur time. I am a contributor to cncf kgateway and I wanted to benchmark its inference routing capabilites, for which I was checking out your benchmarking tests.
I saw that for nightly tests, we are using same env (dataset, vllm configurations, etc.) compared to there corresponding benchmarking scenarios. e.g. nightly Prefill Heavy has same routing aim (stress EPP under KV-cache / prefill saturation), same dataset type as the main benchmarking prefill heavy scenario.
but for multi LoRA - Regression multi lora uses Infinity-Instruct_conversations.json while the similar Nightly Multi-LoRA uses billsum_conversations.json (prefill-heavy), I want to know why was this choice made? if we want manual regression testing or automated nightly testing for both the cases, then we could have done both in regression tests or in nightly tests, why one in manual regression and another in nightly?
please help, I am not able to get this.