OpenAI api seems to not be working

[benbot]

I'm trying to run this script, but I keep getting nonsense output from the openai api

import requests
import json
from typing import Optional

# Function to call the OpenAI API with optional logit_bias
def test_logit_bias(api_key: str, prompt: str, max_tokens: int, logit_bias: Optional[dict] = None):
    # Endpoint URL for OpenAI API
    openai_api_base = "http://0.0.0.0:65530/api/oai"
    api_url = f"{openai_api_base}/v1/completions"
    
    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Bearer {api_key}'
    }
    
    # Payload with and without logit_bias
    payload = {
        "prompt": prompt,
        "max_tokens": 3, 
        "model": "rwkv",
        "temperature": 0
    }
    
    if logit_bias:
        payload["logit_bias"] = logit_bias
    
    response = requests.post(api_url, headers=headers, json=payload)
    
    if response.status_code == 200:
        print(json.dumps(response.json(), indent=2))
    else:
        print(f"Error {response.status_code}: {response.text}")

# Usage
API_KEY = "your-api-key"  # Replace with your actual OpenAI API key

# Test with logit_bias
test_logit_bias(
    api_key=API_KEY,
    prompt="A profression that involves going to space is an ",
    max_tokens=100,
    logit_bias={15311: -100}  # Replace with actual token values
)

# Test without logit_bias
print("Without")
test_logit_bias(
    api_key=API_KEY,
    prompt="A profression that involves going to space is an ",
    max_tokens=100
)

This is the response I get.

{
  "object": "text_completion",
  "model": "assets/models/rwkv-4.safetensors",
  "choices": [
    {
      "text": "\ud83d\ude80",
      "index": 0,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 3,
    "total_tokens": 14
  }
}
Without
{
  "object": "text_completion",
  "model": "assets/models/rwkv-4.safetensors",
  "choices": [
    {
      "text": "\ud83d\ude80",
      "index": 0,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 11,
    "completion_tokens": 3,
    "total_tokens": 14
  }
}

The weird thing is, i'm able to use the chat page in the UI just fine.

[benbot]

okay, seems like this happens when temperature is 0.

after removing that param I'm seeing text being generated, but it contains some weird tokens. Any idea what's up with that

      "text": "\u00b4amangert'\nmeaning it involves the use of astronauts to\nperform certain activities?\nOr is Amagert a management wizard who writes a\nspell to reverse the age of wizards who are\nover 400 years old?\nAnd these are specific examples. There are\nmany many others. But yet it has not been taken\nto court to see what can be argued in this.\n1. Pepsi does not believe that its position is\na legal one. Applying full compensation is a\nrelief and not an apology). It could have argued\nto its conclusion using equal measures, but\noffered an apology to quash all worries caused.\nIt did not.\nThats the difference between different types of\napologies. Different shows and different apology\nmethods. Advertising as well as politics use the\nchannel to deliver the apology to its target\nsure tackle the question i will just choose the\napproach which will please people. Your question\nis wrong. Diet Pepsi which is Ig2 with carbohydrates\n3 per cent gave the nation its first national\nIcy Capital Pepsi.\nso how can you say it is a full compensation?\na lot of young little guys... during",

[cryscan]

You may change your prompt into "A profression that involves going to space is an", i.e., without the space at the end. With/without a space at the end is totally different thing for the model.

[cgisky1980]

openai_api_base = "http://0.0.0.0:65530/api/oai" 0.0.0.0 means any local ip
use 127.0.0.1 maybe

[cgisky1980]

the response told you ==> "finish_reason": "length"

payload = {
"prompt": prompt,
"max_tokens": 3, <=======
"model": "rwkv",
"temperature": 0
}

  "max_tokens": 3,  ===>      "max_tokens": max_tokens, 

[benbot]

openai_api_base = "http://0.0.0.0:65530/api/oai" 0.0.0.0 means any local ip use 127.0.0.1 maybe

It's not an issue with connecting to the api. The output contains weird tokens

[benbot]

@cryscan I'll give that a shot next time I'm messing around with rwkv.

From what I remember though, it'd happen even if i only put the word "Hello" as the prompt.