Is it possible to use gitgitgadget with other projects? #690

Activity

1. dscho commented on Sep 4, 2021

   

   dscho

   on Sep 4, 2021

   Member

   More actions

   @lucasdemarchi Hi! It should be doable (but probably not very easy) to adjust GitGitGadget to use with projects other than Git. Since this has not been done yet, there is no documentation, and there is not even a comprehensive list of what needs to be done to that end (mainly because nobody had the need so far).

   The basic overview is probably still best described in this document: https://github.com/gitgitgadget/gitgitgadget/blob/main/DESIGN.md

   Essentially, GitGitGadget is registered as a GitHub App, backed by an Azure Function (which is defined here: https://github.com/gitgitgadget/gitgitgadget/tree/main/azure-function). Most importantly, this Azure Function needs two secrets to be added to its environment:

   • GITHUB_WEBHOOK_SECRET
   • GITGITGADGET_TRIGGER_TOKEN

   The former is the (optional) secret specified while registering the GitHub App, and the latter is an Azure Pipelines token that is used to trigger the Azure Pipeline that needs to be installed into an Azure DevOps project that corresponds to your main project. See e.g. here for details, you will also have to adjust this line to target the correct Azure DevOps org/project.

   The Azure Pipeline in question runs on a dedicated Linux VM (using a single self-hosted runner, to serialize the requests) and consists of four steps:

   

   The second step runs this shell scriptlet:

   test -d .git || git init || exit 1
   
   git fetch https://github.com/gitgitgadget/gitgitgadget main ||
   exit 1
   
   git reset --hard FETCH_HEAD ||
   exit 1
   
   # If nothing was pulled, we already built it
   test "$(git rev-parse HEAD)" != "$(cat .git/PRE_FETCH_HEAD)" ||
   exit 0
   
   npm install &&
   npm run build ||
   exit 1
   
   git rev-parse HEAD >.git/PRE_FETCH_HEAD

   The second step runs this shell scriptlet:

   GIT_CONFIG_PARAMETERS="$GIT_CONFIG_PARAMETERS $EXTRA_CONFIG"
   node build/script/misc-helper.js set-app-token &&
   git config gitgitgadget.publishRemote \
       https://x-access-token:$(git config gitgitgadget.githubToken)@github.com/gitgitgadget/git

   It is backed by the set-app-token subcommand of misc-helper.ts and requires the GitHub App's private key, which is stored as a secret variable in the Pipeline, and which is provided via the environment variable configured specifically for this step: EXTRA_CONFIG is set to 'gitgitgadget.privateKey=$(gitgitgadget.privateKey)' (including the single-quotes, which are used by Git's GIT_CONFIG_PARAMETERS parser, i.e. they are not optional).

   The second step ensures that the third step has a token to work, using authenticated operations. The third step, then, runs this shell scriptlet:

   GIT_CONFIG_PARAMETERS="$GIT_CONFIG_PARAMETERS $EXTRA_CONFIG"
   set -x
   if test "$(pr.comment.id)" -lt 0
   then
       case "$BUILD_SOURCEBRANCH" in
       refs/pull/[1-9]*/head|refs/pull/[1-9]*/merge)
           branchname=${BUILD_SOURCEBRANCH#refs/pull/}
           prnumber=${branchname%/*}
           ;;
       *) echo "Invalid source branch: $BUILD_SOURCEBRANCH">&2; exit 1;;
       esac
       node build/script/misc-helper.js handle-pr-push "$prnumber"
   else
       node build/script/misc-helper.js handle-pr-comment "$(pr.comment.id)"
   fi

   Like the second step, the third step configures the environment variable EXTRA_CONFIG specifically; It sets it to 'gitgitgadget.smtpUser=gitgitgadget@gmail.com' 'gitgitgadget.smtpHost=smtp.gmail.com' 'gitgitgadget.smtpPass=$(gitgitgadget.smtpPass)' (again, the single-quotes are required, and gitgitgadget.smtpPass refers to the secret variable that contains the SMTP password required to send mails on GitGitGadget's behalf).

   For simplicity's reasons, you will probably want to register your own GMail account to send mails from.

   I considered a long time ago to convert the Azure Pipeline to a GitHub workflow, which would make things less cumbersome to set up, but the major problem here is that we want the run to show up in PRs' Checks tab, therefore it has to be installed as a Pipeline on the repository (or it would have to be added to the repository in .github/workflows/ for GitHub workflows, which would make it next to impossible to maintain GitGitGadget independently from the target repository in question).

   The GitHub App, BTW, needs to be installed explicitly on the repository in question. For procedural reasons, GitGitGadget comes in the form of two GitHub Apps: gitgitgadget and gitgitgadget-git. The reason is that the latter requires permissions to read Checks, read Contents, read & write Issues, read & write Pull Requests and read & write Workflows, while the former additionally requires permissions to read & write Checks and to read _& write Contents (and for historical reasons, to read & write Commit statuses). The reason for this split is that GitGitGadget stores its state under the refs/notes/gitgitgadget ref in https://github.com/gitgitgadget/git, and for understandable reasons the Git project did not want to give GitGitGadget write access to https://github.com/git/git. The second step of the corresponding git/git Pipeline therefore installs two tokens: the one for gitgitgadget to fetch and push the refs/notes/gitgitgadget ref, and gitgitgadget-git to perform the rest of the operations. You probably would stick to one App, and give it write access to the repository to maintain its state in that notes ref.

   So you see, this is all already pretty complicated. Doable, but complicated. And as first fork, you might need to spend a bit of time on streamlining a lot of this.

   While doing so, you will probably run into quite a couple of instances where gitgitgadget/git is hard-coded in non-central places. This would obviously need to be refactored. Also, there are a couple of Git-specific idiosyncrasies in GitGitGadget, such as asking for a certain commit message format. These will be relatively easy to refactor out of the way.

   Please note: despite this wall of text, I do not want to discourage you from embarking on this trip. I just want to caution you that this will be a non-trivial project that will require time and effort.
2. lucasdemarchi commented on Sep 7, 2021

   

   lucasdemarchi

   on Sep 7, 2021

   Author

   More actions

   Thanks a lot for walking through the changes that would be required. It's definitely not a small set and probably outside my area of expertise to say I'm going to tackle it, but I will try as time permits. I found it pretty cool what you're doing here, congrats!

   ❤️React with ❤️1Reacted by Johannes Schindelin
3. dscho commented on Sep 2, 2025

   

   dscho

   on Sep 2, 2025

   Member

   More actions

   @lucasdemarchi I am happy to report that after a lot of work done by @webstech and myself, #2002 and its corresponding PRs are a reality. In fact, it is so real that I was able to use GitGitGadget to submit a patch to Cygwin!