Cloning locks up when more than 3 at the same time.  #364

Activity

1. johnhaley81 commented on Jan 22, 2015

   

   johnhaley81

   on Jan 22, 2015

   Collaborator

   More actions

   I actually get the same thing with the first test (i.e. no remoteCallbacks). It makes the 0-3 directories and then just dies. Waited like 20 minutes and nothing happened. I even get the result when I use a super small repo.

   It seems like sending out those clone commands in parallel is causing them to go into some sort of thread locked state but even when I disable threading by removing this line I still get the same behavior.

   I'd say run this in series for right now. Something is definitely wrong with parallel cloning.
2. maxkorp commented on Jan 30, 2015

   

   maxkorp

   on Jan 30, 2015

   Collaborator

   More actions

   Hrm, 3 is the magic good number on my machine. Even with very large repos, I can do 3 concurrent clones with no issue.

   With 4 repos, even with very small repos, it hangs regardless. Like @johnhaley81, it doesn't seem to matter regarding the remoteCallbacks.
3. maxkorp commented on Jan 30, 2015

   

   maxkorp

   on Jan 30, 2015

   Collaborator

   More actions

   I reworked your example into a bit of a testbed, in case anyone else gets a chance to look at it.

   var size = "small"; // large or small
   var protocol = "https"; // https or ssh
   var howManyAsync = 3; // <=3 works, >=4 breaks
   
   var remotes = {
     small: {
       https: "https://github.com/nodegit/test2.git",
       ssh: "git@github.com:nodegit/test2.git"
     },
     large: {
       https: "https://github.com/nodegit/nodegit.git",
       ssh: "git@github.com:nodegit/nodegit.git"
     }
   };
   
   
   var nodegit = require("../..");
   var fse = require("fs-extra");
   var path = require("path");
   var Promise = require("nodegit-promise");
   
   var remote = remotes[size][protocol];
   
   function clone(to) {
     var opts = { ignoreCertErrors: 1 };
     if (protocol == "ssh") {
       opts.remoteCallbacks = {
         credentials: function (url, userName) {
           return git.Cred.sshKeyFromAgent(userName); // userName should be "git"
         }
       }
     }
   //    var opts = undefined;
     to = "/tmp/nodegit-test" + to;
     console.log("start ", to);
     fse.removeSync(to);
     return nodegit.Clone.clone(remote, to, opts).then(function () {
         console.log("ok!", to);
         return Promise.resolve();
     }, function (err) {
         console.log("failed!", to, err);
         return Promise.reject(err);
     });
   }
   
   console.log("start");
   
   clone("A")
     .then(function() {
       return clone("B")
     })
     .then(function() {
       return clone("C")
     })
     .then(function() {
       return clone("D")
     })
     .then(function() {
       return clone("E")
     })
     .then(function() {
       var promises = []
       for (var i = 0; i < howManyAsync; i++) {
         promises.push(clone(i));
       }
   
       return Promise.all(promises);
     })
     .then(function() {
       console.log("end GOOD");
     }, function() {
       console.log("end BAD");
     })
     .done(function() {
       console.log("end REGARDLESS");
     });
4. drieks commented on Jan 30, 2015

   

   drieks

   on Jan 30, 2015

   Author

   More actions

   Are you sure that this is required? At last for SSH this seems to be ignored...

   return git.Cred.sshKeyFromAgent(userName); // userName should be "git"
   
5. maxkorp commented on Jan 30, 2015

   

   maxkorp

   on Jan 30, 2015

   Collaborator

   More actions

   libgit2 docs don't specify, but I'd venture that if the values your passing to sshKeyFromAgent don't line up and it still works, it's hitting some default somehow.
6. maxkorp

   added

   Bug

   on Feb 4, 2015
7. maxkorp

   changed the title [-]ssh agent key and multiple checkouts[/-] [+]Cloning locks up when more than 3 at the same time. [/+] on Feb 4, 2015
8. tbranyen commented on Feb 28, 2015

   

   tbranyen

   on Feb 28, 2015

   Member

   More actions

   This has been resolved via: #425
9. tbranyen

   closed this as completedon Feb 28, 2015