Merge branch 'PR/197'

Closes #197, #185, #196

README-SUBMODULES.md

@@ -2,11 +2,21 @@
 
 ## Introduction
 
-Subrepositories must first be converted in order for the conversion of
+hg-fast-export supports migrating mercurial subrepositories in the
-the super repository to know how hg commits map to git commits in the
+repository being converted into git submodules in the converted repository.
-sub repositories.  When all subrepositories have been converted, a
+
-mapping file that maps the mercurial subrepository path to a converted
+Git submodules must be git repositories while mercurial's subrepositories can
-git submodule path must be created. The format for this file is:
+be git, mercurial or subversion repositories. hg-fast-export will handle any
+git subrepositories automatically, any other kinds must first be converted
+to git repositories. Currently hg-fast-export does not support the conversion
+of subversion subrepositories. The rest of this page covers the conversion of
+mercurial subrepositories which require some manual steps:
+
+The first step for mercurial subrepositories involves converting the
+subrepository into a git repository using hg-fast-export.  When all
+subrepositories have been converted, a mapping file that maps the mercurial
+subrepository path to a converted git submodule path must be created. The
+format for this file is:
 
 "<mercurial subrepo path>"="<git submodule path>"
 "<mercurial subrepo path2>"="<git submodule path2>"

README.md

@@ -4,35 +4,37 @@ hg-fast-export.(sh|py) - mercurial to git converter using git-fast-import
 Legal
 -----
 
-Most hg-* scripts are licensed under the [MIT license]
+Most hg-* scripts are licensed under the [MIT license] and were written
-(http://www.opensource.org/licenses/mit-license.php) and were written
 by Rocco Rutte <pdmef@gmx.net> with hints and help from the git list and
 \#mercurial on freenode. hg-reset.py is licensed under GPLv2 since it
 copies some code from the mercurial sources.
 
 The current maintainer is Frej Drejhammar <frej.drejhammar@gmail.com>.
 
+[MIT license]: http://www.opensource.org/licenses/mit-license.php
+
 Support
 -------
 
 If you have problems with hg-fast-export or have found a bug, please
-create an issue at the [github issue tracker]
+create an issue at the [github issue tracker]. Before creating a new
-(https://github.com/frej/fast-export/issues). Before creating a new
 issue, check that your problem has not already been addressed in an
 already closed issue. Do not contact the maintainer directly unless
 you want to report a security bug. That way the next person having the
 same problem can benefit from the time spent solving the problem the
 first time.
 
+[github issue tracker]: https://github.com/frej/fast-export/issues
+
 System Requirements
 -------------------
 
-This project depends on Python 2.7 and the Mercurial 4.6 package. If
+This project depends on Python 2.7 and the Mercurial >= 4.6
-Python is not installed, install it before proceeding. The Mercurial
+package. If Python is not installed, install it before proceeding. The
-package can be installed with `pip install mercurial`.
+Mercurial package can be installed with `pip install mercurial`.
 
-If you're on Windows, run the following commands in git bash (Git for
+On windows the bash that comes with "Git for Windows" is known to work
-Windows).
+well.
 
 Usage
 -----
@@ -99,6 +101,17 @@ name the -B and -T options allow a mapping file to be specified to
 rename branches and tags (respectively). The syntax of the mapping
 file is the same as for the author mapping.
 
+When the -B and -T flags are used, you will probably want to use the
+-n flag to disable the built-in (broken in many cases) sanitizing of
+branch/tag names. In the future -n will become the default, but in
+order to not break existing incremental conversions, the default
+remains with the old behavior.
+
+By default, the `default` mercurial branch is renamed to the `master` 
+branch on git. If your mercurial repo contains both `default` and 
+`master` branches, you'll need to override this behavior. Use
+`-M <newName>` to specify what name to give the `default` branch.
+
 Content filtering
 -----------------
 
@@ -183,6 +196,11 @@ hg-fast-export supports multiple branches but only named branches with
 exactly one head each. Otherwise commits to the tip of these heads
 within the branch will get flattened into merge commits.
 
+hg-fast-export will ignore any files or directories tracked by mercurial
+called `.git`, and will print a warning if it encounters one. Git cannot
+track such files or directories. This is not to be confused with submodules,
+which are described in README-SUBMODULES.md.
+
 As each git-fast-import run creates a new pack file, it may be
 required to repack the repository quite often for incremental imports
 (especially when importing a small number of changesets per
@@ -205,6 +223,54 @@ saw never get modified.
 Submitting Patches
 ------------------
 
-Please use the issue-tracker at github
+Please use the [issue-tracker](https://github.com/frej/fast-export) at
-https://github.com/frej/fast-export to report bugs and submit
+github to report bugs and submit patches.
-patches.
+
+Please read
+[https://chris.beams.io/posts/git-commit/](https://chris.beams.io/posts/git-commit/)
+on how to write a good commit message before submitting a pull request
+for review. Although the article recommends at most 50 characters for
+the subject, up to 72 characters are frequently accepted for
+fast-export.
+
+Frequent Problems
+=================
+
+* git fast-import crashes with: `error: cannot lock ref 'refs/heads/...`
+
+  Branch names in git behave as file names (as they are just files and
+  sub-directories under `refs/heads/`, and a path cannot name both a
+  file and a directory, i.e. the branches `a` and `a/b` can never
+  exist at the same time in a git repo.
+
+  Use a mapping file to rename the troublesome branch names.
+
+* `Branch [<branch-name>] modified outside hg-fast-export` but I have
+  not touched the repo!
+
+  If you are running fast-export on a case-preserving but
+  case-insensitive file system (Windows and OSX), this will make git
+  treat `A` and `a` as the same branch. The solution is to use a
+  mapping file to rename branches which only differ in case.
+
+* My mapping file does not seem to work when I rename the branch `git
+  fast-import` crashes on!
+
+  fast-export (imperfectly) mangles branch names it thinks won't be
+  valid. The mechanism cannot be removed as it would break already
+  existing incremental imports that expects it. When fast export
+  mangles a name, it prints out a warning of the form `Warning:
+  sanitized branch [<unmangled>] to [<mangled>]`. If `git fast-import`
+  crashes on `<mangled>`, you need to put `<unmangled>` into the
+  mapping file.
+
+* fast-import mangles valid git branch names which I have remapped!
+
+  Use the `-n` flag to hg-fast-export.sh.
+
+* `git status` reports that all files are scheduled for deletion after
+  the initial conversion.
+
+  By design fast export does not touch your working directory, so to
+  git it looks like you have deleted all files, when in fact they have
+  never been checked out. Just do a checkout of the branch you want.

hg-fast-export.py

@@ -31,6 +31,10 @@ cfg_export_boundary=1000
 subrepo_cache={}
 submodule_mappings=None
 
+# True if fast export should automatically try to sanitize
+# author/branch/tag names.
+auto_sanitize = None
+
 def gitmode(flags):
   return 'l' in flags and '120000' or 'x' in flags and '100755' or '100644'
 
@@ -127,52 +131,71 @@ def get_author(logmessage,committer,authors):
       return r
   return committer
 
+def remove_gitmodules(ctx):
+  """Removes all submodules of ctx parents"""
+  # Removing all submoduies coming from all parents is safe, as the submodules
+  # of the current commit will be re-added below. A possible optimization would
+  # be to only remove the submodules of the first parent.
+  for parent_ctx in ctx.parents():
+    for submodule in parent_ctx.substate.keys():
+      wr('D %s' % submodule)
+  wr('D .gitmodules')
+
+def refresh_git_submodule(name,subrepo_info):
+  wr('M 160000 %s %s' % (subrepo_info[1],name))
+  sys.stderr.write("Adding/updating submodule %s, revision %s\n"
+                   % (name,subrepo_info[1]))
+  return '[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name,name,
+    subrepo_info[0])
+
+def refresh_hg_submodule(name,subrepo_info):
+  gitRepoLocation=submodule_mappings[name] + "/.git"
+
+  # Populate the cache to map mercurial revision to git revision
+  if not name in subrepo_cache:
+    subrepo_cache[name]=(load_cache(gitRepoLocation+"/hg2git-mapping"),
+                         load_cache(gitRepoLocation+"/hg2git-marks",
+                                    lambda s: int(s)-1))
+
+  (mapping_cache,marks_cache)=subrepo_cache[name]
+  subrepo_hash=subrepo_info[1]
+  if subrepo_hash in mapping_cache:
+    revnum=mapping_cache[subrepo_hash]
+    gitSha=marks_cache[int(revnum)]
+    wr('M 160000 %s %s' % (gitSha,name))
+    sys.stderr.write("Adding/updating submodule %s, revision %s->%s\n"
+                     % (name,subrepo_hash,gitSha))
+    return '[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name,name,
+      submodule_mappings[name])
+  else:
+    sys.stderr.write("Warning: Could not find hg revision %s for %s in git %s\n" %
+      (subrepo_hash,name,gitRepoLocation))
+    return ''
+
+def refresh_gitmodules(ctx):
+  """Updates list of ctx submodules according to .hgsubstate file"""
+  remove_gitmodules(ctx)
+  gitmodules=""
+  # Create the .gitmodules file and all submodules
+  for name,subrepo_info in ctx.substate.items():
+    if subrepo_info[2]=='git':
+      gitmodules+=refresh_git_submodule(name,subrepo_info)
+    elif submodule_mappings and name in submodule_mappings:
+      gitmodules+=refresh_hg_submodule(name,subrepo_info)
+
+  if len(gitmodules):
+    wr('M 100644 inline .gitmodules')
+    wr('data %d' % (len(gitmodules)+1))
+    wr(gitmodules)
+
 def export_file_contents(ctx,manifest,files,hgtags,encoding='',plugins={}):
   count=0
   max=len(files)
+  is_submodules_refreshed=False
   for file in files:
-    if submodule_mappings and ctx.substate and file==".hgsubstate":
+    if not is_submodules_refreshed and (file=='.hgsub' or file=='.hgsubstate'):
-      # Remove all submodules as we don't detect deleted submodules properly
+      is_submodules_refreshed=True
-      # in any other way. We will add the ones not deleted back again below.
+      refresh_gitmodules(ctx)
-      for module in submodule_mappings.keys():
-        wr('D %s' % module)
-
-      # Read .hgsubstate file in order to find the revision of each subrepo
-      data=ctx.filectx(file).data()
-      subHashes={}
-      for line in data.split('\n'):
-        if line.strip()=="":
-          continue
-        cols=line.split(' ')
-        subHashes[cols[1]]=cols[0]
-
-      gitmodules=""
-      # Create the .gitmodules file and all submodules
-      for name in ctx.substate:
-        gitRepoLocation=submodule_mappings[name] + "/.git"
-
-        # Populate the cache to map mercurial revision to git revision
-        if not name in subrepo_cache:
-          subrepo_cache[name]=(load_cache(gitRepoLocation+"/hg2git-mapping"),
-                               load_cache(gitRepoLocation+"/hg2git-marks",
-                                          lambda s: int(s)-1))
-
-        (mapping_cache, marks_cache)=subrepo_cache[name]
-        if subHashes[name] in mapping_cache:
-          revnum=mapping_cache[subHashes[name]]
-          gitSha=marks_cache[int(revnum)]
-          wr('M 160000 %s %s' % (gitSha, name))
-          sys.stderr.write("Adding submodule %s, revision %s->%s\n"
-                           % (name,subHashes[name],gitSha))
-          gitmodules+='[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name, name, submodule_mappings[name])
-        else:
-          sys.stderr.write("Warning: Could not find hg revision %s for %s in git %s\n" % (subHashes[name],name,gitRepoLocation))
-
-      if len(gitmodules):
-        wr('M 100644 inline .gitmodules')
-        wr('data %d' % (len(gitmodules)+1))
-        wr(gitmodules)
-
     # Skip .hgtags files. They only get us in trouble.
     if not hgtags and file == ".hgtags":
       sys.stderr.write('Skip %s\n' % (file))
@@ -181,6 +204,9 @@ def export_file_contents(ctx,manifest,files,hgtags,encoding='',plugins={}):
       filename=file.decode(encoding).encode('utf8')
     else:
       filename=file
+    if '.git' in filename.split(os.path.sep):
+      sys.stderr.write('Ignoring file %s which cannot be tracked by git\n' % filename)
+      continue
     file_ctx=ctx.filectx(file)
     d=file_ctx.data()
 
@@ -223,6 +249,8 @@ def sanitize_name(name,what="branch", mapping={}):
     if name[0] == '.': return '_'+name[1:]
     return name
 
+  if not auto_sanitize:
+    return mapping.get(name,name)
   n=mapping.get(name,name)
   p=re.compile('([[ ~^:?\\\\*]|\.\.)')
   n=p.sub('_', n)
@@ -251,6 +279,8 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
     return n
 
   (revnode,_,user,(time,timezone),files,desc,branch,_)=get_changeset(ui,repo,revision,authors,encoding)
+  if repo[revnode].hidden():
+    return count
 
   branch=get_branchname(branch)
 
@@ -293,8 +323,8 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
       # later non-merge revision: feed in changed manifest
       # if we have exactly one parent, just take the changes from the
       # manifest without expensively comparing checksums
-      f=repo.status(parents[0],revnode)[:3]
+      f=repo.status(parents[0],revnode)
-      added,changed,removed=f[1],f[0],f[2]
+      added,changed,removed=f.added,f.modified,f.removed
       type='simple delta'
     else: # a merge with two parents
       wr('merge %s' % revnum_to_revref(parents[1], old_marks))
@@ -307,12 +337,14 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
   sys.stderr.write('%s: Exporting %s revision %d/%d with %d/%d/%d added/changed/removed files\n' %
       (branch,type,revision+1,max,len(added),len(changed),len(removed)))
 
-  if fn_encoding:
+  for filename in removed:
-    removed=[r.decode(fn_encoding).encode('utf8') for r in removed]
+    if fn_encoding:
+      filename=filename.decode(fn_encoding).encode('utf8')
+    filename=strip_leading_slash(filename)
+    if filename=='.hgsub':
+      remove_gitmodules(ctx)
+    wr('D %s' % filename)
 
-  removed=[strip_leading_slash(x) for x in removed]
-
-  map(lambda r: wr('D %s' % r),removed)
   export_file_contents(ctx,man,added,hgtags,fn_encoding,plugins)
   export_file_contents(ctx,man,changed,hgtags,fn_encoding,plugins)
   wr()
@@ -321,6 +353,8 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
 
 def export_note(ui,repo,revision,count,authors,encoding,is_first):
   (revnode,_,user,(time,timezone),_,_,_,_)=get_changeset(ui,repo,revision,authors,encoding)
+  if repo[revnode].hidden():
+    return count
 
   parents = [p for p in repo.changelog.parentrevs(revision) if p >= 0]
 
@@ -438,7 +472,7 @@ def verify_heads(ui,repo,cache,force,branchesmap):
 
   # verify that branch has exactly one head
   t={}
-  for h in repo.heads():
+  for h in repo.filtered('visible').heads():
     (_,_,_,_,_,_,branch,_)=get_changeset(ui,repo,h)
     if t.get(branch,False):
       sys.stderr.write('Error: repository has at least one unnamed head: hg r%s\n' %
@@ -486,12 +520,16 @@ def hg2git(repourl,m,marksfile,mappingfile,headsfile,tipfile,
 
   for rev in range(0,max):
   	(revnode,_,_,_,_,_,_,_)=get_changeset(ui,repo,rev,authors)
+  	if repo[revnode].hidden():
+  		continue
   	mapping_cache[revnode.encode('hex_codec')] = str(rev)
 
   if submodule_mappings:
     # Make sure that all submodules are registered in the submodule-mappings file
     for rev in range(0,max):
       ctx=revsymbol(repo,str(rev))
+      if ctx.hidden():
+        continue
       if ctx.substate:
         for key in ctx.substate:
           if key not in submodule_mappings:
@@ -527,6 +565,9 @@ if __name__=='__main__':
 
   parser=OptionParser()
 
+  parser.add_option("-n", "--no-auto-sanitize",action="store_false",
+      dest="auto_sanitize",default=True,
+      help="Do not perform built-in (broken in many cases) sanitizing of names")
   parser.add_option("-m","--max",type="int",dest="max",
       help="Maximum hg revision to import")
   parser.add_option("--mapping",dest="mappingfile",
@@ -575,6 +616,7 @@ if __name__=='__main__':
   (options,args)=parser.parse_args()
 
   m=-1
+  auto_sanitize = options.auto_sanitize
   if options.max!=None: m=options.max
 
   if options.marksfile==None: bail(parser,'--marks')
@@ -601,7 +643,7 @@ if __name__=='__main__':
 
   t={}
   if options.tagsfile!=None:
-    t=load_mapping('tags', options.tagsfile, True)
+    t=load_mapping('tags', options.tagsfile, options.raw_mappings)
 
   if options.default_branch!=None:
     set_default_branch(options.default_branch)

hg-fast-export.sh

@@ -26,7 +26,29 @@ SFX_MARKS="marks"
 SFX_HEADS="heads"
 SFX_STATE="state"
 GFI_OPTS=""
-PYTHON=${PYTHON:-python2}
+
+if [ -z "${PYTHON}" ]; then
+    # $PYTHON is not set, so we try to find a working python 2.7 to
+    # use. PEP 394 tells us to use 'python2', otherwise try plain
+    # 'python'.
+    if command -v python2 > /dev/null; then
+	PYTHON="python2"
+    elif command -v python > /dev/null; then
+	PYTHON="python"
+    else
+        echo "Could not find any python interpreter, please use the 'PYTHON'" \
+	     "environment variable to specify the interpreter to use."
+        exit 1
+    fi
+fi
+
+# Check that the python specified by the user or autodetected above is
+# >= 2.7 and < 3.
+if ! ${PYTHON} -c 'import sys; v=sys.version_info; exit(0 if v.major == 2 and v.minor >= 7 else 1)' > /dev/null 2>&1 ; then
+    echo "${PYTHON} is not a working python 2.7 interpreter, please use the" \
+	 "'PYTHON' environment variable to specify the interpreter to use."
+    exit 1
+fi
 
 USAGE="[--quiet] [-r <repo>] [--force] [-m <max>] [-s] [--hgtags] [-A <file>] [-B <file>] [-T <file>] [-M <name>] [-o <name>] [--hg-hash] [-e <encoding>]"
 LONG_USAGE="Import hg repository <repo> up to either tip or <max>
@@ -48,6 +70,8 @@ Options:
 	-B <file> Read branch map from file
 	-T <file> Read tags map from file
 	-M <name> Set the default branch name (defaults to 'master')
+	-n        Do not perform built-in (broken in many cases) sanitizing
+                  of branch/tag names.
 	-o <name> Use <name> as branch namespace to track upstream (eg 'origin')
 	--hg-hash Annotate commits with the hg hash as git notes in the
                   hg namespace.

hg2git.py

@@ -35,7 +35,9 @@ def setup_repo(url):
   except TypeError:
     myui=ui.ui()
     myui.setconfig('ui', 'interactive', 'off')
-  return myui,hg.repository(myui,url)
+    # Avoids a warning when the repository has obsolete markers
+    myui.setconfig('experimental', 'evolution.createmarkers', True)
+  return myui,hg.repository(myui,url).unfiltered()
 
 def fixup_user(user,authors):
   user=user.strip("\"")

plugins/branch_name_in_commit/README.md

@@ -6,5 +6,15 @@ during the migration to Git. You can use this plugin to either
 prepend or append the branch name from the mercurial
 commit into the commit message in Git.
 
+Valid arguments are:
+
+- `start`: write the branch name at the start of the commit
+- `end`: write the branch name at the end of the commit
+- `sameline`: if `start` specified, put a colon and a space
+  after the branch name, such that the commit message reads
+  `branch_name: first line of commit message`. Otherwise, the
+  branch name is on the first line of the commit message by itself.
+- `skipmaster`: Don't write the branch name if the branch is `master`.
+
 To use the plugin, add
-`--plugin branch_name_in_commit=(start|end)`.
+`--plugin branch_name_in_commit=<comma_separated_list_of_args>`.

plugins/branch_name_in_commit/__init__.py

@@ -3,12 +3,21 @@ def build_filter(args):
 
 class Filter:
     def __init__(self, args):
-        if not args in ['start','end']:
+        args = {arg: True for arg in args.split(',')}
-            raise Exception('Cannot have branch name anywhere but start and end')
+        self.start = args.pop('start', False)
-        self.pos = args
+        self.end = args.pop('end', False)
+        self.sameline = args.pop('sameline', False)
+        self.skip_master = args.pop('skipmaster', False)
 
-    def commit_message_filter(self,commit_data):
+        if self.sameline and not self.start:
-        if self.pos == 'start':
+            raise ValueError("sameline option only allowed if 'start' given")
-            commit_data['desc'] = commit_data['branch'] + '\n' + commit_data['desc']
+        if args:
-        if self.pos == 'end':
+            raise ValueError("Unknown args: " + ','.join(args))
-            commit_data['desc'] = commit_data['desc'] + '\n' + commit_data['branch']
+
+    def commit_message_filter(self, commit_data):
+        if not (self.skip_master and commit_data['branch'] == 'master'):
+            if self.start:
+                sep = ': ' if self.sameline else '\n' 
+                commit_data['desc'] = commit_data['branch'] + sep + commit_data['desc']
+            if self.end:
+                commit_data['desc'] = commit_data['desc'] + '\n' + commit_data['branch']

plugins/issue_prefix/README.md

@@ -0,0 +1,19 @@
+## Issue Prefix
+
+When migrating to other source code hosting sites, there are cases where a
+project maintainer might want to reset their issue tracker and not have old
+issue numbers in commit messages referring to the wrong issue.  One way around
+this is to prefix issue numbers with some other string.
+
+If migrating to GitHub, this issue prefixing can be paired with GitHub's
+autolinking capabilitiy to link back to a different issue tracker:
+https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
+
+To use this plugin, add:
+`--plugin=issue_prefix=<some_prefix>`
+
+Example:
+`--plugin=issue_prefix=BB-`
+
+This will prefix issue numbers with the string `BB-`.  Example: `#123` will
+change to `#BB-123`.

plugins/issue_prefix/__init__.py

@@ -0,0 +1,15 @@
+# encoding=UTF-8
+"""__init__.py"""
+import re
+
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        self.prefix = args
+
+    def commit_message_filter(self, commit_data):
+        for match in re.findall('#[1-9][0-9]+', commit_data['desc']):
+            commit_data['desc'] = commit_data['desc'].replace(
+                match, '#%s%s' % (self.prefix, match[1:]))

plugins/overwrite_null_messages/README.md

@@ -0,0 +1,23 @@
+## Overwrite Null Commit Messages
+
+There are cases (such as when creating a new, empty snippet on bitbucket
+before they deprecated mercurial repositories) where you could create a
+new repo with a single commit in it, but the message would be null.  Then,
+when attempting to convert this repository to a git repo and pushing to 
+a new host, the git push would fail with an error like this:
+
+    error: a NUL byte in commit log message not allowed
+
+To get around this, you may provide a string that will be used in place of
+a null byte in commit messages.
+
+To use the plugin, add
+
+    --plugin overwrite_null_messages=""
+
+This will use the default commit message `"<empty commit message>"`.
+
+Or to specify a different commit message, you may pass this in at the
+command line like so:
+
+    --plugin overwrite_null_messages="use this message instead"

plugins/overwrite_null_messages/__init__.py

@@ -0,0 +1,16 @@
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        if args == '':
+            message = '<empty commit message>'
+        else:
+            message = args
+        self.message = message
+
+    def commit_message_filter(self,commit_data):
+        # Only write the commit message if the recorded commit
+        # message is null.
+        if commit_data['desc'] == '\x00':
+            commit_data['desc'] = self.message