Skip to content

🧠 两小时从 0 训出你的 64M 大模型。Train your own 64M LLM from scratch in 2 hours.

[fix] serve_openai_api 非流式端点的 max_tokens 不生效,与流式端点行为不一致

Linxiushend4b67f1783e0
diff --git a/scripts/serve_openai_api.py b/scripts/serve_openai_api.py
index adf2212..e9cf871 100644
--- a/scripts/serve_openai_api.py
+++ b/scripts/serve_openai_api.py
@@ -202,7 +202,7 @@ async def chat_completions(request: ChatRequest):
with torch.no_grad():
generated_ids = model.generate(
inputs["input_ids"],
- max_length=inputs["input_ids"].shape[1] + request.max_tokens,
+ max_new_tokens=request.max_tokens,
do_sample=True,
attention_mask=inputs["attention_mask"],
pad_token_id=tokenizer.pad_token_id,