Skip to content

Commit

Permalink
fix encoding issue when building on Windows
Browse files Browse the repository at this point in the history
Summary:
On Windows, the writing operation would fail with:

```
Traceback (most recent call last):
  File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
    sys.exit(main())
  File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
    return args.func(args)
  File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
    change_status = fetcher.update()
  File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
    return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
  File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
    f.write(name + "\n")
  File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
```

and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.

Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)

Reviewed By: wez

Differential Revision: D15281521

fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
  • Loading branch information
fanzeyi authored and facebook-github-bot committed May 10, 2019
1 parent 4cc8480 commit dd8cb1b
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions build/fbcode_builder/getdeps/fetcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -388,17 +388,17 @@ def mirror(self, fbsource_root, dest_root):
# may legitimately need to keep some state in the source tree :-/
installed_name = os.path.join(dest_root, ".shipit_shipped")
if os.path.exists(installed_name):
with open(installed_name, "r") as f:
for name in f.readlines():
with open(installed_name, "rb") as f:
for name in f.read().decode("utf-8").splitlines():
name = name.strip()
if name not in full_file_list:
print("Remove %s" % name)
os.unlink(name)
change_status.record_change(name)

with open(installed_name, "w") as f:
with open(installed_name, "wb") as f:
for name in sorted(list(full_file_list)):
f.write("%s\n" % name)
f.write(("%s\n" % name).encode("utf-8"))

return change_status

Expand Down

0 comments on commit dd8cb1b

Please sign in to comment.