Nemcio/Fast-Export
1
0
Fork 0
mirror of https://github.com/frej/fast-export.git synced 2026-05-07 04:37:40 +02:00
Code Issues Packages Projects Releases Activity

Compare commits

base: Nemcio:v220921
Branches Tags
Nemcio:master
Nemcio:v260405 Nemcio:v250330 Nemcio:v231118 Nemcio:v221024 Nemcio:v220921 Nemcio:v210917 Nemcio:v201029 Nemcio:v200213 Nemcio:v190913 Nemcio:v190107 Nemcio:v180610 Nemcio:v180317 Nemcio:v180126 Nemcio:v171002 Nemcio:v170826 Nemcio:v170818 Nemcio:v170624 Nemcio:v170617 Nemcio:v170604 Nemcio:v170101 Nemcio:v160914 Nemcio:v160415
...
compare: Nemcio:v221024
Branches Tags
Nemcio:master
Nemcio:v260405 Nemcio:v250330 Nemcio:v231118 Nemcio:v221024 Nemcio:v220921 Nemcio:v210917 Nemcio:v201029 Nemcio:v200213 Nemcio:v190913 Nemcio:v190107 Nemcio:v180610 Nemcio:v180317 Nemcio:v180126 Nemcio:v171002 Nemcio:v170826 Nemcio:v170818 Nemcio:v170624 Nemcio:v170617 Nemcio:v170604 Nemcio:v170101 Nemcio:v160914 Nemcio:v160415

2 Commits
v220921 ... v221024

Author SHA1 Message Date
Frej Drejhammar
6700b164d0 Merge branch 'PR/293' 
Closes #292
 2022-10-23 14:47:04 +02:00
chrisjbillington
13c273f10c Resolve unicode escape sequences not being processed correctly 
In `process_unicode_escape_sequences()`, any backslash escape sequences
in the original string are escaped upon the first
`.encode('unicode-escape')` and therefore round-trip the sequence of
`.encode('unicode-escape').decode('unicode-escape')`.

That is not what we want - we want these sequences to be passed-through
the `.encode` unchanged, so that they will be converted to the
character they represent upon `.decode()`.

This patch changes the `.encode()` step to pass through any ascii
characters unchanged, only escaping non-ascii characters. This ensures
any existing backslash escape sequences will be interpreted as the
character they represent upon `.decode()`.
 2022-10-23 11:51:33 +11:00
1 changed files with 9 additions and 3 deletions
Expand all files Collapse all files

12
hg-fast-export.py
View File

@@ -434,9 +434,15 @@ def load_mapping(name, filename, mapping_is_raw):
def process_unicode_escape_sequences(s): def process_unicode_escape_sequences(s):
# Replace unicode escape sequences in the otherwise UTF8-encoded bytestring s with # Replace unicode escape sequences in the otherwise UTF8-encoded bytestring s with
# the UTF8-encoded characters they represent. We need to do an additional # the UTF8-encoded characters they represent. We need to do an additional
# .decode('utf8').encode('unicode-escape') to convert any non-ascii characters into # .decode('utf8').encode('ascii', 'backslashreplace') to convert any non-ascii
# their escape sequences so that the subsequent .decode('unicode-escape') succeeds: # characters into their escape sequences so that the subsequent
return s.decode('utf8').encode('unicode-escape').decode('unicode-escape').encode('utf8') # .decode('unicode-escape') succeeds:
return (
s.decode('utf8')
.encode('ascii', 'backslashreplace')
.decode('unicode-escape')
.encode('utf8')
)
def parse_quoted_line(line): def parse_quoted_line(line):
m=quoted_regexp.match(line) m=quoted_regexp.match(line)