b961f146df
Port hg-fast-import to Python 2/3 polyglot code.
Since mercurial accepts and returns bytestrings for all repository data,
the approach I've taken here is to use bytestrings throughout the
hg-fast-import code. All strings pertaining to repository data are
bytestrings. This means the code is using the same string datatype for
this data on Python 3 as it did (and still does) on Python 2.
Repository data coming from subprocess calls to git, or read from files,
is also left as the bytestrings either returned from
subprocess.check_output or as read from the file in 'rb' mode.
Regexes and string literals that are used with repository data have
all had a b'' prefix added.
When repository data is used in error/warning messages, it is decoded
with the UTF8 codec for printing.
With this patch, hg-fast-export.py writes binary output to
sys.stdout.buffer on Python 3 - on Python 2 this doesn't exist and it
still uses sys.stdout.
The only strings that are left as "native" strings and not coerced to
bytestrings are filepaths passed in on the command line, and dictionary
keys for internal data structures used by hg-fast-import.py, that do
not originate in repository data.
Mapping files are read in 'rb' mode, and thus bytestrings are read from
them. When an encoding is given, their contents are decoded with that
encoding, but then immediately encoded again with UTF8 and they are
returned as the resulting bytestrings
Other necessary changes were:
- indexing byestrings with a single index returns an integer on Python.
These indexing operations have been replaced with a one-element
slice: x[0] -> x[0:1] or x[-1] -> [-1:] so at to return a bytestring.
- raw_hash.encode('hex_codec') replaced with binascii.hexlify(raw_hash)
- str(integer) -> b'%d' % integer
- 'string_escape' codec replaced with 'unicode_escape' (which was
backported to python 2.7). Strings decoded with this codec were then
immediately re-encoded with UTF8.
- Calls to map() intended to execute their contents immediately were
unwrapped or converted to list comprehensions, since map() is an
iterator and does not execute until iterated over.
hg-fast-export.sh has been modified to not require Python 2. Instead, if
PYTHON has not been defined, it checks python2, python, then python3,
and uses the first one that exists and can import the mercurial module.
hg-fast-export.(sh|py) - mercurial to git converter using git-fast-import
Legal
Most hg-* scripts are licensed under the MIT license and were written by Rocco Rutte pdmef@gmx.net with hints and help from the git list and #mercurial on freenode. hg-reset.py is licensed under GPLv2 since it copies some code from the mercurial sources.
The current maintainer is Frej Drejhammar frej.drejhammar@gmail.com.
Support
If you have problems with hg-fast-export or have found a bug, please create an issue at the github issue tracker. Before creating a new issue, check that your problem has not already been addressed in an already closed issue. Do not contact the maintainer directly unless you want to report a security bug. That way the next person having the same problem can benefit from the time spent solving the problem the first time.
System Requirements
This project depends on Python 2.7 or 3.5+, and the Mercurial >= 4.6
package (>= 5.2, if Python 3.5+). If Python is not installed, install
it before proceeding. TheMercurial package can be installed with
pip install mercurial.
On windows the bash that comes with "Git for Windows" is known to work well.
Usage
Using hg-fast-export is quite simple for a mercurial repository :
mkdir repo-git # or whatever
cd repo-git
git init
hg-fast-export.sh -r <local-repo>
git checkout HEAD
Please note that hg-fast-export does not automatically check out the
newly imported repository. You probably want to follow up the import
with a git checkout-command.
Incremental imports to track hg repos is supported, too.
Using hg-reset it is quite simple within a git repository that is hg-fast-export'ed from mercurial:
hg-reset.sh -R <revision>
will give hints on which branches need adjustment for starting over again.
When a mercurial repository does not use utf-8 for encoding author
strings and commit messages the -e <encoding> command line option
can be used to force fast-export to convert incoming meta data from
to utf-8. This encoding option is also applied to file names.
In some locales Mercurial uses different encodings for commit messages
and file names. In that case, you can use --fe <encoding> command line
option which overrides the -e option for file names.
As mercurial appears to be much less picky about the syntax of the
author information than git, an author mapping file can be given to
hg-fast-export to fix up malformed author strings. The file is
specified using the -A option. The file should contain lines of the
form "<key>"="<value>". Inside the key and value strings, all escape
sequences understood by the python string_escape encoding are
supported. (Versions of fast-export prior to v171002 had a different
syntax, the old syntax can be enabled by the flag
--mappings-are-raw.)
The example authors.map below will translate User <garbage<tab><user@example.com> to User <user@example.com>.
-- Start of authors.map --
"User <garbage\t<user@example.com>"="User <user@example.com>"
-- End of authors.map --
Tag and Branch Naming
As Git and Mercurial have differ in what is a valid branch and tag name the -B and -T options allow a mapping file to be specified to rename branches and tags (respectively). The syntax of the mapping file is the same as for the author mapping.
When the -B and -T flags are used, you will probably want to use the -n flag to disable the built-in (broken in many cases) sanitizing of branch/tag names. In the future -n will become the default, but in order to not break existing incremental conversions, the default remains with the old behavior.
By default, the default mercurial branch is renamed to the master
branch on git. If your mercurial repo contains both default and
master branches, you'll need to override this behavior. Use
-M <newName> to specify what name to give the default branch.
Content filtering
hg-fast-export supports filtering the content of exported files. The filter is supplied to the --filter-contents option. hg-fast-export runs the filter for each exported file, pipes its content to the filter's standard input, and uses the filter's standard output in place of the file's original content. The prototypical use of this feature is to convert line endings in text files from CRLF to git's preferred LF:
-- Start of crlf-filter.sh --
#!/bin/sh
# $1 = pathname of exported file relative to the root of the repo
# $2 = Mercurial's hash of the file
# $3 = "1" if Mercurial reports the file as binary, otherwise "0"
if [ "$3" == "1" ]; then cat; else dos2unix; fi
-- End of crlf-filter.sh --
Plugins
hg-fast-export supports plugins to manipulate the file data and commit metadata. The plugins are enabled with the --plugin option. The value of said option is a plugin name (by folder in the plugins directory), and optionally, and equals-sign followed by an initialization string.
There is a readme accompanying each of the bundled plugins, with a
description of the usage. To create a new plugin, one must simply
add a new folder under the plugins directory, with the name of the
new plugin. Inside, there must be an __init__.py file, which contains
at a minimum:
def build_filter(args):
return Filter(args)
class Filter:
def __init__(self, args):
pass
#Or don't pass, if you want to do some init code here
Beyond the boilerplate initialization, you can see the two different defined filter methods in the dos2unix and branch_name_in_commit plugins.
commit_data = {'branch': branch, 'parents': parents, 'author': author, 'desc': desc}
def commit_message_filter(self,commit_data):
The commit_message_filter method is called for each commit, after parsing
from hg, but before outputting to git. The dictionary commit_data contains the
above attributes about the commit, and can be modified by any filter. The
values in the dictionary after filters have been run are used to create the git
commit.
file_data = {'filename':filename,'file_ctx':file_ctx,'d':d}
def file_data_filter(self,file_data):
The file_data_filter method is called for each file within each commit.
The dictionary file_data contains the above attributes about the file, and
can be modified by any filter. file_ctx is the filecontext from the
mercurial python library. After all filters have been run, the values
are used to add the file to the git commit.
Submodules
See README-SUBMODULES.md for how to convert subrepositories into git submodules.
Notes/Limitations
hg-fast-export supports multiple branches but only named branches with exactly one head each. Otherwise commits to the tip of these heads within the branch will get flattened into merge commits.
hg-fast-export will ignore any files or directories tracked by mercurial
called .git, and will print a warning if it encounters one. Git cannot
track such files or directories. This is not to be confused with submodules,
which are described in README-SUBMODULES.md.
As each git-fast-import run creates a new pack file, it may be required to repack the repository quite often for incremental imports (especially when importing a small number of changesets per incremental import).
The way the hg API and remote access protocol is designed it is not possible to use hg-fast-export on remote repositories (http/ssh). First clone the repository, then convert it.
Design
hg-fast-export.py was designed in a way that doesn't require a 2-pass mechanism or any prior repository analysis: if just feeds what it finds into git-fast-import. This also implies that it heavily relies on strictly linear ordering of changesets from hg, i.e. its append-only storage model so that changesets hg-fast-export already saw never get modified.
Submitting Patches
Please use the issue-tracker at github to report bugs and submit patches.
Please read https://chris.beams.io/posts/git-commit/ on how to write a good commit message before submitting a pull request for review. Although the article recommends at most 50 characters for the subject, up to 72 characters are frequently accepted for fast-export.
Frequent Problems
-
git fast-import crashes with:
error: cannot lock ref 'refs/heads/...Branch names in git behave as file names (as they are just files and sub-directories under
refs/heads/, and a path cannot name both a file and a directory, i.e. the branchesaanda/bcan never exist at the same time in a git repo.Use a mapping file to rename the troublesome branch names.
-
Branch [<branch-name>] modified outside hg-fast-exportbut I have not touched the repo!If you are running fast-export on a case-preserving but case-insensitive file system (Windows and OSX), this will make git treat
Aandaas the same branch. The solution is to use a mapping file to rename branches which only differ in case. -
My mapping file does not seem to work when I rename the branch
git fast-importcrashes on!fast-export (imperfectly) mangles branch names it thinks won't be valid. The mechanism cannot be removed as it would break already existing incremental imports that expects it. When fast export mangles a name, it prints out a warning of the form
Warning: sanitized branch [<unmangled>] to [<mangled>]. Ifgit fast-importcrashes on<mangled>, you need to put<unmangled>into the mapping file. -
fast-import mangles valid git branch names which I have remapped!
Use the
-nflag to hg-fast-export.sh. -
git statusreports that all files are scheduled for deletion after the initial conversion.By design fast export does not touch your working directory, so to git it looks like you have deleted all files, when in fact they have never been checked out. Just do a checkout of the branch you want.