Skip to content

fix: Calling the model using non stream cannot obtain the token #3826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 6, 2025
Merged

fix: Calling the model using non stream cannot obtain the token #3826

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions apps/models_provider/impl/base_chat_open_ai.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ def get_num_tokens_from_messages(
except Exception as e:
tokenizer = TokenizerManage.get_tokenizer()
return sum([len(tokenizer.encode(get_buffer_string([m]))) for m in messages])
return self.usage_metadata.get('input_tokens', 0)
return self.usage_metadata.get('input_tokens', self.usage_metadata.get('prompt_tokens', 0))

def get_num_tokens(self, text: str) -> int:
if self.usage_metadata is None or self.usage_metadata == {}:
Expand All @@ -108,7 +108,8 @@ def get_num_tokens(self, text: str) -> int:
except Exception as e:
tokenizer = TokenizerManage.get_tokenizer()
return len(tokenizer.encode(text))
return self.get_last_generation_info().get('output_tokens', 0)
return self.get_last_generation_info().get('output_tokens',
self.get_last_generation_info().get('completion_tokens', 0))

def _stream(self, *args: Any, **kwargs: Any) -> Iterator[ChatGenerationChunk]:
kwargs['stream_usage'] = True
Expand All @@ -133,7 +134,7 @@ def _convert_chunk_to_generation_chunk(
)

usage_metadata: Optional[UsageMetadata] = (
_create_usage_metadata(token_usage) if token_usage else None
_create_usage_metadata(token_usage) if token_usage and token_usage.get("prompt_tokens") else None
)
if len(choices) == 0:
# logprobs is implicitly None
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some suggestions to improve the readability and error handling of the provided code, along with addressing any logical errors:

  1. Remove Redundant Error Handling: The exception block in both get_num_tokens_from_messages and get_num_tokens seems unnecessary given that _create_usage_metadata function would handle exceptions.

    def get_sum_of_tokens_in_chat_history(messages):
    try:
        tokenizer = TokenizerManage.get_tokenizer()
        return sum([len(tokenizer.encode(s)) for s in messages])
    except Exception as e:
        raise RuntimeError(f"Failed to count tokens: {e}")

def get_number_of_tokens(input_text):
    try:
        tokenizer = TokenizerManage.get_tokenizer()
        return len(tokenizer.encode(input_text))
    except Exception as e:
        raise RuntimeError(f"Failed to count tokens: {e}")

def set_usage_metadata(input_message: Union[str, List], output_message=None) -> UsageMetadata:
    try:
        if isinstance(input_message, list):
            input_usage = []
            for message in input_message:
                msg_obj = {"prompt": message}
                if output_message and hasattr(output_message.message.choices[0].logprobs, 'tokens'):
                    result = {
                        "prompt_tokens": len(message),
                        "completion_tokens": sum((prob['token']) for prob in output_message.message.choices[0].logprobs.tokens)
                    }
                    print(result)
                  }                    
                  
                  # If no log probs available for completion tokens
                  else:
                      if not isinstance(input_message, List):  
                          input_tokens = [input_message]
                          input_usage.append({"content": input_message})
                          
      except ValueError as ve:
          print("Incorrect usage format:", ve)

2. **Optimize String Concatenation**: In Python versions before 3.6, string concatenation using the `+` operator can be inefficient because strings are immutable and each new concatenated operation creates a new string object. Use an f-string or the `join()` method instead to avoid this inefficiency.

3. **Error Handling in Method Calls**: Ensure that you have adequate error handling when calling methods like `TokenizerManage.get_tokenizer()`.

4. **Return Type Annotations**: Consider adding type annotations to the parameters and variables where possible to improve clarity and maintainability.

5. **Code Readability**: Refactor complex expressions into smaller functions for better organization.

```python
def get_summary_statistics():
   stats = {}
   
   def calculate_average(numbers):
       if not numbers:
           return 0
       return sum(nums) / len(nums)

   all_scores = [...your data here...]

   average_score = calculate_average(all_scores)
   max_score = max(scores)
   min_score = min(score)


calculate_stats()  # Call your refactored summary_statistics function

This approach helps organize the code more effectively and makes it easier to understand the logic behind different parts.

Expand Down
Loading