Skip to content

Commit

Permalink
增加utf-8格式转换,减少在不同平台上生成的文件中出现中文编码而不是中文文字本体的情况。已在macbook和windows测试
Browse files Browse the repository at this point in the history
  • Loading branch information
靳博 committed Apr 3, 2023
1 parent 504af07 commit e2b58e5
Show file tree
Hide file tree
Showing 9 changed files with 195 additions and 4 deletions.
8 changes: 8 additions & 0 deletions .idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 17 additions & 0 deletions .idea/BELLE.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

70 changes: 70 additions & 0 deletions .idea/deployment.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

72 changes: 72 additions & 0 deletions .idea/inspectionProfiles/Project_Default.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/inspectionProfiles/profiles_settings.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions .idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions 1.5M/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,9 @@ def openai_completion(

if return_text:
if api == "completion":
completions = [completion.text for completion in completions]
completions = [completion.text.encode('utf-8').decode('utf-8') for completion in completions]
elif api == "chat":
completions = [completion.message.content for completion in completions]
completions = [completion.message.content.encode('utf-8').decode('utf-8') for completion in completions]
if decoding_args.n > 1:
# make completions a nested list, where each entry is a consecutive decoding_args.n of original entries.
completions = [completions[i : i + decoding_args.n] for i in range(0, len(completions), decoding_args.n)]
Expand All @@ -161,7 +161,7 @@ def _make_w_io_base(f, mode: str):
f_dirname = os.path.dirname(f)
if f_dirname != "":
os.makedirs(f_dirname, exist_ok=True)
f = open(f, mode=mode)
f = open(f, mode=mode, encoding="utf-8")
return f


Expand All @@ -183,7 +183,7 @@ def jdump(obj, f, mode="w", indent=4, default=str):
"""
f = _make_w_io_base(f, mode)
if isinstance(obj, (dict, list)):
json.dump(obj, f, indent=indent, default=default)
json.dump(obj, f, indent=indent, default=default, ensure_ascii=False)
elif isinstance(obj, str):
f.write(obj)
else:
Expand Down

0 comments on commit e2b58e5

Please sign in to comment.