Allow name sanitizer to be disabled with --no-auto-sanitize

Make it possible to completely disable the name sanitizer by the
--no-auto-sanitize flag. Previously the sanitizer was run on user
remapped names. As the sanitizer rewrites perfectly legal git
names (such as __.*) this is probably not what the user wants.

Closes #155.

.gitattributes

@@ -0,0 +1,2 @@
+# Set the default behavior, in case people don't have core.autocrlf set.
+* text=auto

.gitignore

@@ -1,2 +1,4 @@
+*.orig
 *.pyc
 .dotest
+.idea/

README-SUBMODULES.md

@@ -0,0 +1,65 @@
+# How to convert Mercurial Repositories with subrepos
+
+## Introduction
+
+Subrepositories must first be converted in order for the conversion of
+the super repository to know how hg commits map to git commits in the
+sub repositories.  When all subrepositories have been converted, a
+mapping file that maps the mercurial subrepository path to a converted
+git submodule path must be created. The format for this file is:
+
+"<mercurial subrepo path>"="<git submodule path>"
+"<mercurial subrepo path2>"="<git submodule path2>"
+...
+
+The path of this mapping file is then provided with the --subrepo-map
+command line option.
+
+## Example
+
+Example mercurial repo folder structure (~/mercurial):
+    src/...
+    subrepo/subrepo1
+    subrepo/subrepo2
+
+### Setup
+Create an empty new folder where all the converted git modules will be imported:
+    mkdir ~/imported-gits
+    cd ~/imported-gits
+
+### Convert all submodules to git:
+    mkdir submodule1
+    cd submodule1
+    git init
+    hg-fast-export.sh -r ~/mercurial/subrepo1
+    cd ..
+    mkdir submodule2
+    cd submodule2
+    git init
+    hg-fast-export.sh -r ~/mercurial/subrepo2
+
+### Create mapping file
+    cd ~/imported-gits
+    cat > submodule-mappings << EOF
+    "subrepo/subrepo1"="../submodule1"
+    "subrepo/subrepo2"="../submodule2"
+    EOF
+
+### Convert main repository
+    cd ~/imported-gits
+    mkdir git-main-repo
+    cd git-main-repo
+    git init
+    hg-fast-export.sh -r ~/mercurial --subrepo-map=../submodule-mappings
+
+### Result
+The resulting repository will now contain the subrepo/subrepo1 and
+subrepo/subrepo1 submodules. The created .gitmodules file will look
+like:
+
+    [submodule "subrepo/subrepo1"]
+          path = subrepo/subrepo1
+          url = ../submodule1
+    [submodule "subrepo/subrepo2"]
+          path = subrepo/subrepo2
+          url = ../submodule2

README.md

@@ -4,26 +4,38 @@ hg-fast-export.(sh|py) - mercurial to git converter using git-fast-import
 Legal
 -----
 
-Most hg-* scripts are licensed under the [MIT license]
-(http://www.opensource.org/licenses/mit-license.php) and were written
+Most hg-* scripts are licensed under the [MIT license] and were written
 by Rocco Rutte <pdmef@gmx.net> with hints and help from the git list and
 \#mercurial on freenode. hg-reset.py is licensed under GPLv2 since it
 copies some code from the mercurial sources.
 
 The current maintainer is Frej Drejhammar <frej.drejhammar@gmail.com>.
 
+[MIT license]: http://www.opensource.org/licenses/mit-license.php
+
 Support
 -------
 
 If you have problems with hg-fast-export or have found a bug, please
-create an issue at the [github issue tracker]
-(https://github.com/frej/fast-export/issues). Before creating a new
+create an issue at the [github issue tracker]. Before creating a new
 issue, check that your problem has not already been addressed in an
 already closed issue. Do not contact the maintainer directly unless
 you want to report a security bug. That way the next person having the
 same problem can benefit from the time spent solving the problem the
 first time.
 
+[github issue tracker]: https://github.com/frej/fast-export/issues
+
+System Requirements
+-------------------
+
+This project depends on Python 2.7 and the Mercurial 4.6 package. If
+Python is not installed, install it before proceeding. The Mercurial
+package can be installed with `pip install mercurial`.
+
+If you're on Windows, run the following commands in git bash (Git for
+Windows).
+
 Usage
 -----
 
@@ -33,7 +45,8 @@ Using hg-fast-export is quite simple for a mercurial repository <repo>:
 mkdir repo-git # or whatever
 cd repo-git
 git init
-hg-fast-export.sh -r <repo>
+hg-fast-export.sh -r <local-repo>
+git checkout HEAD
 ```
 
 Please note that hg-fast-export does not automatically check out the
@@ -88,6 +101,89 @@ name the -B and -T options allow a mapping file to be specified to
 rename branches and tags (respectively). The syntax of the mapping
 file is the same as for the author mapping.
 
+When the -B and -T flags are used, you will probably want to use the
+-n flag to disable the built-in (broken in many cases) sanitizing of
+branch/tag names. In the future -n will become the default, but in
+order to not break existing incremental conversions, the default
+remains with the old behavior.
+
+Content filtering
+-----------------
+
+hg-fast-export supports filtering the content of exported files.
+The filter is supplied to the --filter-contents option. hg-fast-export
+runs the filter for each exported file, pipes its content to the filter's
+standard input, and uses the filter's standard output in place
+of the file's original content. The prototypical use of this feature
+is to convert line endings in text files from CRLF to git's preferred LF:
+
+```
+-- Start of crlf-filter.sh --
+#!/bin/sh
+# $1 = pathname of exported file relative to the root of the repo
+# $2 = Mercurial's hash of the file
+# $3 = "1" if Mercurial reports the file as binary, otherwise "0"
+
+if [ "$3" == "1" ]; then cat; else dos2unix; fi
+-- End of crlf-filter.sh --
+```
+
+
+Plugins
+-----------------
+
+hg-fast-export supports plugins to manipulate the file data and commit
+metadata. The plugins are enabled with the --plugin option. The value
+of said option is a plugin name (by folder in the plugins directory),
+and optionally, and equals-sign followed by an initialization string.
+
+There is a readme accompanying each of the bundled plugins, with a
+description of the usage. To create a new plugin, one must simply
+add a new folder under the `plugins` directory, with the name of the
+new plugin. Inside, there must be an `__init__.py` file, which contains
+at a minimum:
+
+```
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        pass
+        #Or don't pass, if you want to do some init code here
+```
+
+Beyond the boilerplate initialization, you can see the two different
+defined filter methods in the [dos2unix](./plugins/dos2unix) and
+[branch_name_in_commit](./plugins/branch_name_in_commit) plugins.
+
+```
+commit_data = {'branch': branch, 'parents': parents, 'author': author, 'desc': desc}
+
+def commit_message_filter(self,commit_data):
+```
+The `commit_message_filter` method is called for each commit, after parsing
+from hg, but before outputting to git. The dictionary `commit_data` contains the
+above attributes about the commit, and can be modified by any filter. The
+values in the dictionary after filters have been run are used to create the git
+commit.
+
+```
+file_data = {'filename':filename,'file_ctx':file_ctx,'d':d}
+
+def file_data_filter(self,file_data):
+```
+The `file_data_filter` method is called for each file within each commit.
+The dictionary `file_data` contains the above attributes about the file, and
+can be modified by any filter. `file_ctx` is the filecontext from the
+mercurial python library.  After all filters have been run, the values
+are used to add the file to the git commit.
+
+Submodules
+----------
+See README-SUBMODULES.md for how to convert subrepositories into git
+submodules.
+
 Notes/Limitations
 -----------------
 

hg-fast-export.py

@@ -1,15 +1,17 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 # Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.
 # License: MIT <http://www.opensource.org/licenses/mit-license.php>
 
 from mercurial import node
+from mercurial.scmutil import revsymbol
 from hg2git import setup_repo,fixup_user,get_branch,get_changeset
 from hg2git import load_cache,save_cache,get_git_sha1,set_default_branch,set_origin_name
 from optparse import OptionParser
 import re
 import sys
 import os
+import pluginloader
 
 if sys.platform == "win32":
   # On Windows, sys.stdout is initially opened in text mode, which means that
@@ -26,6 +28,13 @@ cfg_checkpoint_count=0
 # write some progress message every this many file contents written
 cfg_export_boundary=1000
 
+subrepo_cache={}
+submodule_mappings=None
+
+# True if fast export should automatically try to sanitize
+# author/branch/tag names.
+auto_sanitize = None
+
 def gitmode(flags):
   return 'l' in flags and '120000' or 'x' in flags and '100755' or '100644'
 
@@ -78,7 +87,7 @@ def get_filechanges(repo,revision,parents,mleft):
   l,c,r=[],[],[]
   for p in parents:
     if p<0: continue
-    mright=repo.changectx(p).manifest()
+    mright=revsymbol(repo,str(p)).manifest()
     l,c,r=split_dict(mleft,mright,l,c,r)
   l.sort()
   c.sort()
@@ -122,19 +131,74 @@ def get_author(logmessage,committer,authors):
       return r
   return committer
 
-def export_file_contents(ctx,manifest,files,hgtags,encoding=''):
+def remove_gitmodules(ctx):
+  """Removes all submodules of ctx parents"""
+  # Removing all submoduies coming from all parents is safe, as the submodules
+  # of the current commit will be re-added below. A possible optimization would
+  # be to only remove the submodules of the first parent.
+  for parent_ctx in ctx.parents():
+    for submodule in parent_ctx.substate.keys():
+      wr('D %s' % submodule)
+  wr('D .gitmodules')
+
+def refresh_gitmodules(ctx):
+  """Updates list of ctx submodules according to .hgsubstate file"""
+  remove_gitmodules(ctx)
+  gitmodules=""
+  # Create the .gitmodules file and all submodules
+  for name,subrepo_info in ctx.substate.items():
+    gitRepoLocation=submodule_mappings[name] + "/.git"
+
+    # Populate the cache to map mercurial revision to git revision
+    if not name in subrepo_cache:
+      subrepo_cache[name]=(load_cache(gitRepoLocation+"/hg2git-mapping"),
+                           load_cache(gitRepoLocation+"/hg2git-marks",
+                                      lambda s: int(s)-1))
+
+    (mapping_cache,marks_cache)=subrepo_cache[name]
+    subrepo_hash=subrepo_info[1]
+    if subrepo_hash in mapping_cache:
+      revnum=mapping_cache[subrepo_hash]
+      gitSha=marks_cache[int(revnum)]
+      wr('M 160000 %s %s' % (gitSha,name))
+      sys.stderr.write("Adding/updating submodule %s, revision %s->%s\n"
+                       % (name,subrepo_hash,gitSha))
+      gitmodules+='[submodule "%s"]\n\tpath = %s\n\turl = %s\n' % (name,name,
+        submodule_mappings[name])
+    else:
+      sys.stderr.write("Warning: Could not find hg revision %s for %s in git %s\n" %
+        (subrepo_hash,name,gitRepoLocation))
+
+  if len(gitmodules):
+    wr('M 100644 inline .gitmodules')
+    wr('data %d' % (len(gitmodules)+1))
+    wr(gitmodules)
+
+def export_file_contents(ctx,manifest,files,hgtags,encoding='',plugins={}):
   count=0
   max=len(files)
   for file in files:
+    if submodule_mappings and file==".hgsubstate":
+      refresh_gitmodules(ctx)
     # Skip .hgtags files. They only get us in trouble.
     if not hgtags and file == ".hgtags":
       sys.stderr.write('Skip %s\n' % (file))
       continue
-    d=ctx.filectx(file).data()
     if encoding:
       filename=file.decode(encoding).encode('utf8')
     else:
       filename=file
+    file_ctx=ctx.filectx(file)
+    d=file_ctx.data()
+
+    if plugins and plugins['file_data_filters']:
+      file_data = {'filename':filename,'file_ctx':file_ctx,'data':d}
+      for filter in plugins['file_data_filters']:
+        filter(file_data)
+      d=file_data['data']
+      filename=file_data['filename']
+      file_ctx=file_data['file_ctx']
+
     wr('M %s inline %s' % (gitmode(manifest.flags(file)),
                            strip_leading_slash(filename)))
     wr('data %d' % len(d)) # had some trouble with size()
@@ -153,15 +217,21 @@ def sanitize_name(name,what="branch", mapping={}):
   # modifying names which previously were not touched it will break
   # preexisting setups which are doing incremental imports.
   #
-  # Use the -B and -T options to mangle branch and tag names
-  # instead. If you have a source repository where this is too much
-  # work to do manually, write a tool that does it for you.
+  # Fast-export tries to not inflict arbitrary naming policy on the
+  # user, instead it aims to provide mechanisms allowing the user to
+  # apply their own policy. Therefore do not add a transform which can
+  # already be implemented with the -B and -T options to mangle branch
+  # and tag names. If you have a source repository where this is too
+  # much work to do manually, write a tool that does it for you.
+  #
 
   def dot(name):
     if not name: return name
     if name[0] == '.': return '_'+name[1:]
     return name
 
+  if not auto_sanitize:
+    return mapping.get(name,name)
   n=mapping.get(name,name)
   p=re.compile('([[ ~^:?\\\\*]|\.\.)')
   n=p.sub('_', n)
@@ -180,7 +250,8 @@ def strip_leading_slash(filename):
   return filename
 
 def export_commit(ui,repo,revision,old_marks,max,count,authors,
-                  branchesmap,sob,brmap,hgtags,encoding='',fn_encoding=''):
+                  branchesmap,sob,brmap,hgtags,encoding='',fn_encoding='',
+                  plugins={}):
   def get_branchname(name):
     if brmap.has_key(name):
       return brmap[name]
@@ -193,6 +264,16 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
   branch=get_branchname(branch)
 
   parents = [p for p in repo.changelog.parentrevs(revision) if p >= 0]
+  author = get_author(desc,user,authors)
+
+  if plugins and plugins['commit_message_filters']:
+    commit_data = {'branch': branch, 'parents': parents, 'author': author, 'desc': desc}
+    for filter in plugins['commit_message_filters']:
+      filter(commit_data)
+    branch = commit_data['branch']
+    parents = commit_data['parents']
+    author = commit_data['author']
+    desc = commit_data['desc']
 
   if len(parents)==0 and revision != 0:
     wr('reset refs/heads/%s' % branch)
@@ -200,13 +281,13 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
   wr('commit refs/heads/%s' % branch)
   wr('mark :%d' % (revision+1))
   if sob:
-    wr('author %s %d %s' % (get_author(desc,user,authors),time,timezone))
+    wr('author %s %d %s' % (author,time,timezone))
   wr('committer %s %d %s' % (user,time,timezone))
   wr('data %d' % (len(desc)+1)) # wtf?
   wr(desc)
   wr()
 
-  ctx=repo.changectx(str(revision))
+  ctx=revsymbol(repo,str(revision))
   man=ctx.manifest()
   added,changed,removed,type=[],[],[],''
 
@@ -221,7 +302,7 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
       # later non-merge revision: feed in changed manifest
       # if we have exactly one parent, just take the changes from the
       # manifest without expensively comparing checksums
-      f=repo.status(repo.lookup(parents[0]),revnode)[:3]
+      f=repo.status(parents[0],revnode)[:3]
       added,changed,removed=f[1],f[0],f[2]
       type='simple delta'
     else: # a merge with two parents
@@ -235,14 +316,16 @@ def export_commit(ui,repo,revision,old_marks,max,count,authors,
   sys.stderr.write('%s: Exporting %s revision %d/%d with %d/%d/%d added/changed/removed files\n' %
       (branch,type,revision+1,max,len(added),len(changed),len(removed)))
 
-  if fn_encoding:
-    removed=[r.decode(fn_encoding).encode('utf8') for r in removed]
+  for filename in removed:
+    if fn_encoding:
+      filename=filename.decode(fn_encoding).encode('utf8')
+    filename=strip_leading_slash(filename)
+    if filename=='.hgsubstate':
+      remove_gitmodules(ctx)
+    wr('D %s' % filename)
 
-  removed=[strip_leading_slash(x) for x in removed]
-
-  map(lambda r: wr('D %s' % r),removed)
-  export_file_contents(ctx,man,added,hgtags,fn_encoding)
-  export_file_contents(ctx,man,changed,hgtags,fn_encoding)
+  export_file_contents(ctx,man,added,hgtags,fn_encoding,plugins)
+  export_file_contents(ctx,man,changed,hgtags,fn_encoding,plugins)
   wr()
 
   return checkpoint(count)
@@ -258,7 +341,7 @@ def export_note(ui,repo,revision,count,authors,encoding,is_first):
   if is_first:
     wr('from refs/notes/hg^0')
   wr('N inline :%d' % (revision+1))
-  hg_hash=repo.changectx(str(revision)).hex()
+  hg_hash=revsymbol(repo,str(revision)).hex()
   wr('data %d' % (len(hg_hash)))
   wr_no_nl(hg_hash)
   wr()
@@ -378,7 +461,8 @@ def verify_heads(ui,repo,cache,force,branchesmap):
 
 def hg2git(repourl,m,marksfile,mappingfile,headsfile,tipfile,
            authors={},branchesmap={},tagsmap={},
-           sob=False,force=False,hgtags=False,notes=False,encoding='',fn_encoding=''):
+           sob=False,force=False,hgtags=False,notes=False,encoding='',fn_encoding='',
+           plugins={}):
   def check_cache(filename, contents):
     if len(contents) == 0:
       sys.stderr.write('Warning: %s does not contain any data, this will probably make an incremental import fail\n' % filename)
@@ -415,12 +499,22 @@ def hg2git(repourl,m,marksfile,mappingfile,headsfile,tipfile,
   	(revnode,_,_,_,_,_,_,_)=get_changeset(ui,repo,rev,authors)
   	mapping_cache[revnode.encode('hex_codec')] = str(rev)
 
+  if submodule_mappings:
+    # Make sure that all submodules are registered in the submodule-mappings file
+    for rev in range(0,max):
+      ctx=revsymbol(repo,str(rev))
+      if ctx.substate:
+        for key in ctx.substate:
+          if key not in submodule_mappings:
+            sys.stderr.write("Error: %s not found in submodule-mappings\n" % (key))
+            return 1
 
   c=0
   brmap={}
   for rev in range(min,max):
     c=export_commit(ui,repo,rev,old_marks,max,c,authors,branchesmap,
-                    sob,brmap,hgtags,encoding,fn_encoding)
+                    sob,brmap,hgtags,encoding,fn_encoding,
+                    plugins)
   if notes:
     for rev in range(min,max):
       c=export_note(ui,repo,rev,c,authors, encoding, rev == min and min != 0)
@@ -444,6 +538,9 @@ if __name__=='__main__':
 
   parser=OptionParser()
 
+  parser.add_option("-n", "--no-auto-sanitize",action="store_false",
+      dest="auto_sanitize",default=True,
+      help="Do not perform built-in (broken in many cases) sanitizing of names")
   parser.add_option("-m","--max",type="int",dest="max",
       help="Maximum hg revision to import")
   parser.add_option("--mapping",dest="mappingfile",
@@ -480,10 +577,19 @@ if __name__=='__main__':
       help="Assume file names from Mercurial are encoded in <filename_encoding>")
   parser.add_option("--mappings-are-raw",dest="raw_mappings", default=False,
       help="Assume mappings are raw <key>=<value> lines")
+  parser.add_option("--filter-contents",dest="filter_contents",
+      help="Pipe contents of each exported file through FILTER_CONTENTS <file-path> <hg-hash> <is-binary>")
+  parser.add_option("--plugin-path", type="string", dest="pluginpath",
+      help="Additional search path for plugins ")
+  parser.add_option("--plugin", action="append", type="string", dest="plugins",
+      help="Add a plugin with the given init string <name=init>")
+  parser.add_option("--subrepo-map", type="string", dest="subrepo_map",
+      help="Provide a mapping file between the subrepository name and the submodule name")
 
   (options,args)=parser.parse_args()
 
   m=-1
+  auto_sanitize = options.auto_sanitize
   if options.max!=None: m=options.max
 
   if options.marksfile==None: bail(parser,'--marks')
@@ -492,6 +598,14 @@ if __name__=='__main__':
   if options.statusfile==None: bail(parser,'--status')
   if options.repourl==None: bail(parser,'--repo')
 
+  if options.subrepo_map:
+      if not os.path.exists(options.subrepo_map):
+        sys.stderr.write('Subrepo mapping file not found %s\n'
+                         % options.subrepo_map)
+        sys.exit(1)
+      submodule_mappings=load_mapping('subrepo mappings',
+                                      options.subrepo_map,False)
+
   a={}
   if options.authorfile!=None:
     a=load_mapping('authors', options.authorfile, options.raw_mappings)
@@ -518,8 +632,34 @@ if __name__=='__main__':
   if options.fn_encoding!=None:
     fn_encoding=options.fn_encoding
 
+  plugins=[]
+  if options.plugins!=None:
+    plugins+=options.plugins
+
+  if options.filter_contents!=None:
+    plugins+=['shell_filter_file_contents='+options.filter_contents]
+
+  plugins_dict={}
+  plugins_dict['commit_message_filters']=[]
+  plugins_dict['file_data_filters']=[]
+
+  if plugins and options.pluginpath:
+    sys.stderr.write('Using additional plugin path: ' + options.pluginpath + '\n')
+
+  for plugin in plugins:
+    split = plugin.split('=')
+    name, opts = split[0], '='.join(split[1:])
+    i = pluginloader.get_plugin(name,options.pluginpath)
+    sys.stderr.write('Loaded plugin ' + i['name'] + ' from path: ' + i['path'] +' with opts: ' + opts + '\n')
+    plugin = pluginloader.load_plugin(i).build_filter(opts)
+    if hasattr(plugin,'file_data_filter') and callable(plugin.file_data_filter):
+      plugins_dict['file_data_filters'].append(plugin.file_data_filter)
+    if hasattr(plugin, 'commit_message_filter') and callable(plugin.commit_message_filter):
+      plugins_dict['commit_message_filters'].append(plugin.commit_message_filter)
+
   sys.exit(hg2git(options.repourl,m,options.marksfile,options.mappingfile,
                   options.headsfile, options.statusfile,
                   authors=a,branchesmap=b,tagsmap=t,
                   sob=options.sob,force=options.force,hgtags=options.hgtags,
-                  notes=options.notes,encoding=encoding,fn_encoding=fn_encoding))
+                  notes=options.notes,encoding=encoding,fn_encoding=fn_encoding,
+                  plugins=plugins_dict))

hg-fast-export.sh

@@ -26,7 +26,29 @@ SFX_MARKS="marks"
 SFX_HEADS="heads"
 SFX_STATE="state"
 GFI_OPTS=""
-PYTHON=${PYTHON:-python}
+
+if [ -z "${PYTHON}" ]; then
+    # $PYTHON is not set, so we try to find a working python 2.7 to
+    # use. PEP 394 tells us to use 'python2', otherwise try plain
+    # 'python'.
+    if command -v python2 > /dev/null; then
+	PYTHON="python2"
+    elif command -v python > /dev/null; then
+	PYTHON="python"
+    else
+        echo "Could not find any python interpreter, please use the 'PYTHON'" \
+	     "environment variable to specify the interpreter to use."
+        exit 1
+    fi
+fi
+
+# Check that the python specified by the user or autodetected above is
+# >= 2.7 and < 3.
+if ! ${PYTHON} -c 'import sys; v=sys.version_info; exit(0 if v.major == 2 and v.minor >= 7 else 1)' > /dev/null 2>&1 ; then
+    echo "${PYTHON} is not a working python 2.7 interpreter, please use the" \
+	 "'PYTHON' environment variable to specify the interpreter to use."
+    exit 1
+fi
 
 USAGE="[--quiet] [-r <repo>] [--force] [-m <max>] [-s] [--hgtags] [-A <file>] [-B <file>] [-T <file>] [-M <name>] [-o <name>] [--hg-hash] [-e <encoding>]"
 LONG_USAGE="Import hg repository <repo> up to either tip or <max>
@@ -48,6 +70,8 @@ Options:
 	-B <file> Read branch map from file
 	-T <file> Read tags map from file
 	-M <name> Set the default branch name (defaults to 'master')
+	-n        Do not perform built-in (broken in many cases) sanitizing
+                  of branch/tag names.
 	-o <name> Use <name> as branch namespace to track upstream (eg 'origin')
 	--hg-hash Annotate commits with the hg hash as git notes in the
                   hg namespace.
@@ -56,6 +80,10 @@ Options:
 	--fe <filename_encoding> Assume filenames from Mercurial are encoded 
 	                         in <filename_encoding>
 	--mappings-are-raw Assume mappings are raw <key>=<value> lines
+	--filter-contents <cmd>  Pipe contents of each exported file through <cmd>
+	                         with <file-path> <hg-hash> <is-binary> as arguments
+	--plugin <plugin=init>  Add a plugin with the given init string (repeatable)
+	--plugin-path <plugin-path> Add an additional plugin lookup path
 "
 case "$1" in
     -h|--help)

hg2git.py

@@ -1,9 +1,12 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 
 # Copyright (c) 2007, 2008 Rocco Rutte <pdmef@gmx.net> and others.
 # License: MIT <http://www.opensource.org/licenses/mit-license.php>
 
 from mercurial import hg,util,ui,templatefilters
+from mercurial import error as hgerror
+from mercurial.scmutil import revsymbol,binnode
+
 import re
 import os
 import sys
@@ -46,7 +49,7 @@ def fixup_user(user,authors):
     # and mail from hg helpers. this seems to work pretty well.
     # if email doesn't contain @, replace it with devnull@localhost
     name=templatefilters.person(user)
-    mail='<%s>' % util.email(user)
+    mail='<%s>' % templatefilters.email(user)
     if '@' not in mail:
       mail = '<devnull@localhost>'
   else:
@@ -69,7 +72,15 @@ def get_branch(name):
   return name
 
 def get_changeset(ui,repo,revision,authors={},encoding=''):
-  node=repo.lookup(revision)
+  # Starting with Mercurial 4.6 lookup no longer accepts raw hashes
+  # for lookups. Work around it by changing our behaviour depending on
+  # how it fails
+  try:
+    node=repo.lookup(revision)
+  except hgerror.ProgrammingError:
+    node=binnode(revsymbol(repo,str(revision))) # We were given a numeric rev
+  except hgerror.RepoLookupError:
+    node=revision # We got a raw hash
   (manifest,user,(time,timezone),files,desc,extra)=repo.changelog.read(node)
   if encoding:
     user=user.decode(encoding).encode('utf8')

pluginloader/__init__.py

@@ -0,0 +1,19 @@
+import os
+import imp
+PluginFolder = os.path.join(os.path.dirname(os.path.realpath(__file__)),"..","plugins")
+MainModule = "__init__"
+
+def get_plugin(name, plugin_path):
+    search_dirs = [PluginFolder]
+    if plugin_path:
+        search_dirs = [plugin_path] + search_dirs
+    for dir in search_dirs:
+        location = os.path.join(dir, name)
+        if not os.path.isdir(location) or not MainModule + ".py" in os.listdir(location):
+            continue
+        info = imp.find_module(MainModule, [location])
+        return {"name": name, "info": info, "path": location}
+    raise Exception("Could not find plugin with name " + name)
+
+def load_plugin(plugin):
+    return imp.load_module(MainModule, *plugin["info"])

plugins/branch_name_in_commit/README.md

@@ -0,0 +1,10 @@
+## Branch Name in Commit Message
+
+Mercurial has a much stronger notion of branches than Git,
+and some parties may not wish to lose the branch information
+during the migration to Git. You can use this plugin to either
+prepend or append the branch name from the mercurial
+commit into the commit message in Git.
+
+To use the plugin, add
+`--plugin branch_name_in_commit=(start|end)`.

plugins/branch_name_in_commit/__init__.py

@@ -0,0 +1,14 @@
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        if not args in ['start','end']:
+            raise Exception('Cannot have branch name anywhere but start and end')
+        self.pos = args
+
+    def commit_message_filter(self,commit_data):
+        if self.pos == 'start':
+            commit_data['desc'] = commit_data['branch'] + '\n' + commit_data['desc']
+        if self.pos == 'end':
+            commit_data['desc'] = commit_data['desc'] + '\n' + commit_data['branch']

plugins/dos2unix/README.md

@@ -0,0 +1,9 @@
+## Dos2unix filter
+
+This plugin converts CRLF line ending to LF in text files in the repo.
+It is recommended that you have a .gitattributes file that maintains
+the usage of LF endings going forward, for after you have converted your
+repository.
+
+To use the plugin, add
+`--plugin dos2unix`.

plugins/dos2unix/__init__.py

@@ -0,0 +1,11 @@
+def build_filter(args):
+    return Filter(args)
+
+class Filter():
+    def __init__(self, args):
+        pass
+
+    def file_data_filter(self,file_data):
+        file_ctx = file_data['file_ctx']
+        if not file_ctx.isbinary():
+            file_data['data'] = file_data['data'].replace('\r\n', '\n')

plugins/shell_filter_file_contents/README.md

@@ -0,0 +1,30 @@
+## Shell Script File Filter
+
+This plugin uses shell scripts in order to perform filtering of files.
+If your preferred scripting is done via shell, this tool is for you.
+Be noted, though, that this method can cause an order of magnitude slow
+down. For small repositories, this wont be an issue.
+
+To use the plugin, add
+`--plugin shell_filter_file_contents=path/to/shell/script.sh`.
+The filter script is supplied to the plugin option after the plugin name,
+which is in turned passed to the plugin initialization. hg-fast-export
+runs the filter for each exported file, pipes its content to the filter's
+standard input, and uses the filter's standard output in place
+of the file's original content. An example use of this feature
+is to convert line endings in text files from CRLF to git's preferred LF,
+although this task is faster performed using the native plugin.
+
+The script is called with the following syntax:
+`FILTER_CONTENTS <file-path> <hg-hash> <is-binary>`
+
+```
+-- Start of crlf-filter.sh --
+#!/bin/sh
+# $1 = pathname of exported file relative to the root of the repo
+# $2 = Mercurial's hash of the file
+# $3 = "1" if Mercurial reports the file as binary, otherwise "0"
+
+if [ "$3" == "1" ]; then cat; else dos2unix; fi
+-- End of crlf-filter.sh --
+```

plugins/shell_filter_file_contents/__init__.py

@@ -0,0 +1,28 @@
+#Pipe contents of each exported file through FILTER_CONTENTS <file-path> <hg-hash> <is-binary>"
+import subprocess
+import shlex
+import sys
+from mercurial import node
+
+def build_filter(args):
+    return Filter(args)
+
+class Filter:
+    def __init__(self, args):
+        self.filter_contents = shlex.split(args)
+
+    def file_data_filter(self,file_data):
+        d = file_data['data']
+        file_ctx = file_data['file_ctx']
+        filename = file_data['filename']
+        filter_cmd = self.filter_contents + [filename, node.hex(file_ctx.filenode()), '1' if file_ctx.isbinary() else '0']
+        try:
+            filter_proc = subprocess.Popen(filter_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+            d, _ = filter_proc.communicate(d)
+        except:
+            sys.stderr.write('Running filter-contents %s:\n' % filter_cmd)
+            raise
+        filter_ret = filter_proc.poll()
+        if filter_ret:
+            raise subprocess.CalledProcessError(filter_ret, filter_cmd)
+        file_data['data'] = d