Merge branch 'PR/197'

Closes #197, #185, #196

README-SUBMODULES.md

@@ -2,11 +2,21 @@
 
 ## Introduction
 
-Subrepositories must first be converted in order for the conversion of
+hg-fast-export supports migrating mercurial subrepositories in the
-the super repository to know how hg commits map to git commits in the
+repository being converted into git submodules in the converted repository.
-sub repositories.  When all subrepositories have been converted, a
+
-mapping file that maps the mercurial subrepository path to a converted
+Git submodules must be git repositories while mercurial's subrepositories can
-git submodule path must be created. The format for this file is:
+be git, mercurial or subversion repositories. hg-fast-export will handle any
+git subrepositories automatically, any other kinds must first be converted
+to git repositories. Currently hg-fast-export does not support the conversion
+of subversion subrepositories. The rest of this page covers the conversion of
+mercurial subrepositories which require some manual steps:
+
+The first step for mercurial subrepositories involves converting the
+subrepository into a git repository using hg-fast-export.  When all
+subrepositories have been converted, a mapping file that maps the mercurial
+subrepository path to a converted git submodule path must be created. The
+format for this file is:
 
 "<mercurial subrepo path>"="<git submodule path>"
 "<mercurial subrepo path2>"="<git submodule path2>"

README.md

@@ -29,12 +29,12 @@ first time.
 System Requirements
 -------------------
 
-This project depends on Python 2.7 and the Mercurial 4.6 package. If
+This project depends on Python 2.7 and the Mercurial >= 4.6
-Python is not installed, install it before proceeding. The Mercurial
+package. If Python is not installed, install it before proceeding. The
-package can be installed with `pip install mercurial`.
+Mercurial package can be installed with `pip install mercurial`.
 
-If you're on Windows, run the following commands in git bash (Git for
+On windows the bash that comes with "Git for Windows" is known to work
-Windows).
+well.
 
 Usage
 -----
@@ -107,6 +107,11 @@ branch/tag names. In the future -n will become the default, but in
 order to not break existing incremental conversions, the default
 remains with the old behavior.
 
+By default, the `default` mercurial branch is renamed to the `master` 
+branch on git. If your mercurial repo contains both `default` and 
+`master` branches, you'll need to override this behavior. Use
+`-M <newName>` to specify what name to give the `default` branch.
+
 Content filtering
 -----------------
 
@@ -191,6 +196,11 @@ hg-fast-export supports multiple branches but only named branches with
 exactly one head each. Otherwise commits to the tip of these heads
 within the branch will get flattened into merge commits.
 
+hg-fast-export will ignore any files or directories tracked by mercurial
+called `.git`, and will print a warning if it encounters one. Git cannot
+track such files or directories. This is not to be confused with submodules,
+which are described in README-SUBMODULES.md.
+
 As each git-fast-import run creates a new pack file, it may be
 required to repack the repository quite often for incremental imports
 (especially when importing a small number of changesets per
@@ -213,6 +223,54 @@ saw never get modified.
 Submitting Patches
 ------------------
 
-Please use the issue-tracker at github
+Please use the [issue-tracker](https://github.com/frej/fast-export) at
-https://github.com/frej/fast-export to report bugs and submit
+github to report bugs and submit patches.
-patches.
+
+Please read
+[https://chris.beams.io/posts/git-commit/](https://chris.beams.io/posts/git-commit/)
+on how to write a good commit message before submitting a pull request
+for review. Although the article recommends at most 50 characters for
+the subject, up to 72 characters are frequently accepted for
+fast-export.
+
+Frequent Problems
+=================
+
+* git fast-import crashes with: `error: cannot lock ref 'refs/heads/...`
+
+  Branch names in git behave as file names (as they are just files and
+  sub-directories under `refs/heads/`, and a path cannot name both a
+  file and a directory, i.e. the branches `a` and `a/b` can never
+  exist at the same time in a git repo.
+
+  Use a mapping file to rename the troublesome branch names.
+
+* `Branch [<branch-name>] modified outside hg-fast-export` but I have
+  not touched the repo!
+
+  If you are running fast-export on a case-preserving but
+  case-insensitive file system (Windows and OSX), this will make git
+  treat `A` and `a` as the same branch. The solution is to use a
+  mapping file to rename branches which only differ in case.
+
+* My mapping file does not seem to work when I rename the branch `git
+  fast-import` crashes on!
+
+  fast-export (imperfectly) mangles branch names it thinks won't be
+  valid. The mechanism cannot be removed as it would break already
+  existing incremental imports that expects it. When fast export
+  mangles a name, it prints out a warning of the form `Warning:
+  sanitized branch [<unmangled>] to [<mangled>]`. If `git fast-import`
+  crashes on `<mangled>`, you need to put `<unmangled>` into the
+  mapping file.
+
+* fast-import mangles valid git branch names which I have remapped!
+
+  Use the `-n` flag to hg-fast-export.sh.
+
+* `git status` reports that all files are scheduled for deletion after
+  the initial conversion.
+
+  By design fast export does not touch your working directory, so to
+  git it looks like you have deleted all files, when in fact they have
+  never been checked out. Just do a checkout of the branch you want.

hg-fast-export.py

@@ -141,33 +141,47 @@ def remove_gitmodules(ctx):
       wr('D %s' % submodule)
   wr('D .gitmodules')
 
+def refresh_git_submodule(name,subrepo_info):
+  wr('M 160000 %s %s' % (subrepo_info[1],name))
+  sys.stderr.write("Adding/updating submodule %s, revision %s\n"
+                   % (name,subrepo_info[1]))
+  return '[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name,name,
+    subrepo_info[0])
+
+def refresh_hg_submodule(name,subrepo_info):
+  gitRepoLocation=submodule_mappings[name] + "/.git"
+
+  # Populate the cache to map mercurial revision to git revision
+  if not name in subrepo_cache:
+    subrepo_cache[name]=(load_cache(gitRepoLocation+"/hg2git-mapping"),
+                         load_cache(gitRepoLocation+"/hg2git-marks",
+                                    lambda s: int(s)-1))
+
+  (mapping_cache,marks_cache)=subrepo_cache[name]
+  subrepo_hash=subrepo_info[1]
+  if subrepo_hash in mapping_cache:
+    revnum=mapping_cache[subrepo_hash]
+    gitSha=marks_cache[int(revnum)]
+    wr('M 160000 %s %s' % (gitSha,name))
+    sys.stderr.write("Adding/updating submodule %s, revision %s->%s\n"
+                     % (name,subrepo_hash,gitSha))
+    return '[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name,name,
+      submodule_mappings[name])
+  else:
+    sys.stderr.write("Warning: Could not find hg revision %s for %s in git %s\n" %
+      (subrepo_hash,name,gitRepoLocation))
+    return ''
+
 def refresh_gitmodules(ctx):
   """Updates list of ctx submodules according to .hgsubstate file"""
   remove_gitmodules(ctx)
   gitmodules=""
   # Create the .gitmodules file and all submodules
   for name,subrepo_info in ctx.substate.items():
-    gitRepoLocation=submodule_mappings[name] + "/.git"
+    if subrepo_info[2]=='git':
-
+      gitmodules+=refresh_git_submodule(name,subrepo_info)
-    # Populate the cache to map mercurial revision to git revision
+    elif submodule_mappings and name in submodule_mappings:
-    if not name in subrepo_cache:
+      gitmodules+=refresh_hg_submodule(name,subrepo_info)
-      subrepo_cache[name]=(load_cache(gitRepoLocation+"/hg2git-mapping"),
-                           load_cache(gitRepoLocation+"/hg2git-marks",
-                                      lambda s: int(s)-1))
-
-    (mapping_cache,marks_cache)=subrepo_cache[name]
-    subrepo_hash=subrepo_info[1]
-    if subrepo_hash in mapping_cache:
-      revnum=mapping_cache[subrepo_hash]
-      gitSha=marks_cache[int(revnum)]
-      wr('M 160000 %s %s' % (gitSha,name))
-      sys.stderr.write("Adding/updating submodule %s, revision %s->%s\n"
-                       % (name,subrepo_hash,gitSha))
-      gitmodules+='[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name,name,
-        submodule_mappings[name])
-    else:
-      sys.stderr.write("Warning: Could not find hg revision %s for %s in git %s\n" %
-        (subrepo_hash,name,gitRepoLocation))
 
   if len(gitmodules):
     wr('M 100644 inline .gitmodules')
@@ -177,8 +191,10 @@ def refresh_gitmodules(ctx):
 def export_file_contents(ctx,manifest,files,hgtags,encoding='',plugins={}):
   count=0
   max=len(files)
+  is_submodules_refreshed=False
   for file in files:
-    if submodule_mappings and file==".hgsubstate":
+    if not is_submodules_refreshed and (file=='.hgsub' or file=='.hgsubstate'):
+      is_submodules_refreshed=True
       refresh_gitmodules(ctx)
     # Skip .hgtags files. They only get us in trouble.
     if not hgtags and file == ".hgtags":
@@ -188,6 +204,9 @@ def export_file_contents(ctx,manifest,files,hgtags,encoding='',plugins={}):
       filename=file.decode(encoding).encode('utf8')
     else:
       filename=file
+    if '.git' in filename.split(os.path.sep):
+      sys.stderr.write('Ignoring file %s which cannot be tracked by git\n' % filename)
+      continue
     file_ctx=ctx.filectx(file)
     d=file_ctx.data()
 
@@ -260,6 +279,8 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
     return n
 
   (revnode,_,user,(time,timezone),files,desc,branch,_)=get_changeset(ui,repo,revision,authors,encoding)
+  if repo[revnode].hidden():
+    return count
 
   branch=get_branchname(branch)
 
@@ -302,8 +323,8 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
       # later non-merge revision: feed in changed manifest
       # if we have exactly one parent, just take the changes from the
       # manifest without expensively comparing checksums
-      f=repo.status(parents[0],revnode)[:3]
+      f=repo.status(parents[0],revnode)
-      added,changed,removed=f[1],f[0],f[2]
+      added,changed,removed=f.added,f.modified,f.removed
       type='simple delta'
     else: # a merge with two parents
       wr('merge %s' % revnum_to_revref(parents[1], old_marks))
@@ -320,7 +341,7 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
     if fn_encoding:
       filename=filename.decode(fn_encoding).encode('utf8')
     filename=strip_leading_slash(filename)
-    if filename=='.hgsubstate':
+    if filename=='.hgsub':
       remove_gitmodules(ctx)
     wr('D %s' % filename)
 
@@ -332,6 +353,8 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
 
 def export_note(ui,repo,revision,count,authors,encoding,is_first):
   (revnode,_,user,(time,timezone),_,_,_,_)=get_changeset(ui,repo,revision,authors,encoding)
+  if repo[revnode].hidden():
+    return count
 
   parents = [p for p in repo.changelog.parentrevs(revision) if p >= 0]
 
@@ -449,7 +472,7 @@ def verify_heads(ui,repo,cache,force,branchesmap):
 
   # verify that branch has exactly one head
   t={}
-  for h in repo.heads():
+  for h in repo.filtered('visible').heads():
     (_,_,_,_,_,_,branch,_)=get_changeset(ui,repo,h)
     if t.get(branch,False):
       sys.stderr.write('Error: repository has at least one unnamed head: hg r%s\n' %
@@ -497,12 +520,16 @@ def hg2git(repourl,m,marksfile,mappingfile,headsfile,tipfile,
 
   for rev in range(0,max):
   	(revnode,_,_,_,_,_,_,_)=get_changeset(ui,repo,rev,authors)
+  	if repo[revnode].hidden():
+  		continue
   	mapping_cache[revnode.encode('hex_codec')] = str(rev)
 
   if submodule_mappings:
     # Make sure that all submodules are registered in the submodule-mappings file
     for rev in range(0,max):
       ctx=revsymbol(repo,str(rev))
+      if ctx.hidden():
+        continue
       if ctx.substate:
         for key in ctx.substate:
           if key not in submodule_mappings:
@@ -616,7 +643,7 @@ if __name__=='__main__':
 
   t={}
   if options.tagsfile!=None:
-    t=load_mapping('tags', options.tagsfile, True)
+    t=load_mapping('tags', options.tagsfile, options.raw_mappings)
 
   if options.default_branch!=None:
     set_default_branch(options.default_branch)

hg2git.py

@@ -35,7 +35,9 @@ def setup_repo(url):
   except TypeError:
     myui=ui.ui()
     myui.setconfig('ui', 'interactive', 'off')
-  return myui,hg.repository(myui,url)
+    # Avoids a warning when the repository has obsolete markers
+    myui.setconfig('experimental', 'evolution.createmarkers', True)
+  return myui,hg.repository(myui,url).unfiltered()
 
 def fixup_user(user,authors):
   user=user.strip("\"")

plugins/branch_name_in_commit/README.md

@@ -6,5 +6,15 @@ during the migration to Git. You can use this plugin to either
 prepend or append the branch name from the mercurial
 commit into the commit message in Git.
 
+Valid arguments are:
+
+- `start`: write the branch name at the start of the commit
+- `end`: write the branch name at the end of the commit
+- `sameline`: if `start` specified, put a colon and a space
+  after the branch name, such that the commit message reads
+  `branch_name: first line of commit message`. Otherwise, the
+  branch name is on the first line of the commit message by itself.
+- `skipmaster`: Don't write the branch name if the branch is `master`.
+
 To use the plugin, add
-`--plugin branch_name_in_commit=(start|end)`.
+`--plugin branch_name_in_commit=<comma_separated_list_of_args>`.

plugins/branch_name_in_commit/__init__.py

@@ -3,12 +3,21 @@ def build_filter(args):
 
 class Filter:
     def __init__(self, args):
-        if not args in ['start','end']:
+        args = {arg: True for arg in args.split(',')}
-            raise Exception('Cannot have branch name anywhere but start and end')
+        self.start = args.pop('start', False)
-        self.pos = args
+        self.end = args.pop('end', False)
+        self.sameline = args.pop('sameline', False)
+        self.skip_master = args.pop('skipmaster', False)
 
-    def commit_message_filter(self,commit_data):
+        if self.sameline and not self.start:
-        if self.pos == 'start':
+            raise ValueError("sameline option only allowed if 'start' given")
-            commit_data['desc'] = commit_data['branch'] + '\n' + commit_data['desc']
+        if args:
-        if self.pos == 'end':
+            raise ValueError("Unknown args: " + ','.join(args))
-            commit_data['desc'] = commit_data['desc'] + '\n' + commit_data['branch']
+
+    def commit_message_filter(self, commit_data):
+        if not (self.skip_master and commit_data['branch'] == 'master'):
+            if self.start:
+                sep = ': ' if self.sameline else '\n' 
+                commit_data['desc'] = commit_data['branch'] + sep + commit_data['desc']
+            if self.end:
+                commit_data['desc'] = commit_data['desc'] + '\n' + commit_data['branch']

plugins/issue_prefix/README.md

@@ -0,0 +1,19 @@
+## Issue Prefix
+
+When migrating to other source code hosting sites, there are cases where a
+project maintainer might want to reset their issue tracker and not have old
+issue numbers in commit messages referring to the wrong issue.  One way around
+this is to prefix issue numbers with some other string.
+
+If migrating to GitHub, this issue prefixing can be paired with GitHub's
+autolinking capabilitiy to link back to a different issue tracker:
+https://help.github.com/en/github/administering-a-repository/configuring-autolinks-to-reference-external-resources
+
+To use this plugin, add:
+`--plugin=issue_prefix=<some_prefix>`
+
+Example:
+`--plugin=issue_prefix=BB-`
+
+This will prefix issue numbers with the string `BB-`.  Example: `#123` will
+change to `#BB-123`.

plugins/issue_prefix/__init__.py

@@ -0,0 +1,15 @@
+# encoding=UTF-8
+"""__init__.py"""
+import re
+
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        self.prefix = args
+
+    def commit_message_filter(self, commit_data):
+        for match in re.findall('#[1-9][0-9]+', commit_data['desc']):
+            commit_data['desc'] = commit_data['desc'].replace(
+                match, '#%s%s' % (self.prefix, match[1:]))

plugins/overwrite_null_messages/README.md

@@ -0,0 +1,23 @@
+## Overwrite Null Commit Messages
+
+There are cases (such as when creating a new, empty snippet on bitbucket
+before they deprecated mercurial repositories) where you could create a
+new repo with a single commit in it, but the message would be null.  Then,
+when attempting to convert this repository to a git repo and pushing to 
+a new host, the git push would fail with an error like this:
+
+    error: a NUL byte in commit log message not allowed
+
+To get around this, you may provide a string that will be used in place of
+a null byte in commit messages.
+
+To use the plugin, add
+
+    --plugin overwrite_null_messages=""
+
+This will use the default commit message `"<empty commit message>"`.
+
+Or to specify a different commit message, you may pass this in at the
+command line like so:
+
+    --plugin overwrite_null_messages="use this message instead"

plugins/overwrite_null_messages/__init__.py

@@ -0,0 +1,16 @@
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        if args == '':
+            message = '<empty commit message>'
+        else:
+            message = args
+        self.message = message
+
+    def commit_message_filter(self,commit_data):
+        # Only write the commit message if the recorded commit
+        # message is null.
+        if commit_data['desc'] == '\x00':
+            commit_data['desc'] = self.message