Here at WebDevStudios (WDS) we use a GitFlow-like process during development. The branch and merge strategy allows for multiple developers to work on a multitude of various features concurrently. In practice, it allows us to have greater development speed and product stability with a larger team. The names here are arbitrary, but these are some conventions we use:
- Production – “Clean” (reviewed) code that will usually automatically be deployed to a server where it can be QA-d and or client-reviewed.
- Master – “Dirty” branch where developers can merge changes for testing; typically automatically deployed to a development server.
- Feature – A short-lived branch for one particular feature. They always start from the clean Production branch. There’s sometimes tens of these in progress all at once. They can get merged into Master at anytime, and merged into Production after a code review.
With larger teams, it can seem like things are moving fast, because they are. When something gets screwed up in Git, it can bring everyone to a screeching halt. This happened to me recently so I wanted to share what I learned, in case you find yourself in the same situation.
If you’re a hands-on learner, you can reproduce this entire scenario in a local Git repository by running the preformatted commands in a terminal window. Skip any highlighted git push/fetch commands, unless you’ve also set up a remote (origin) repository at GitHub or similar.
We’re going to recreate the following scenario. I’ll explain as we go along.
Create our initial repository with one file (C1):
mkdir git-merge-revert cd git-merge-revert git init wget https://themes.svn.wordpress.org/twentyten/2.2/style.css git add style.css git commit -m "Initial check-in"
Create the Production branch from Master where reviewed code will go from now on:
git checkout -b prod
Branch Feature 1 from Production and edit the theme title (C2):
git checkout -b feature1/change-title # Edit theme title in style.css vim style.css git commit -a -m "Feature 1 - changed title"
Create another branch from Production to edit the theme author (C5):
git checkout prod git checkout -b feature2/change-author # Edit theme author in style.css vim style.css git commit -a -m "Feature 2 - changed author"
Create a third branch from Production, and edit the title again (C7). In the real world, you’d never purposefully assign two developers the same task concurrently, but we’re going to do this to artificially introduce a merge conflict.
git checkout prod git checkout -b feature3/change-title # Edit theme title differently in style.css vim style.css git commit -a -m "Feature 3 - changed title"
Feature 1 is ready for testing, so merge it to master (C3):
git checkout master git merge feature1/change-title
After a code review, Feature 1 is deemed ready for Production (C4). There’s an explanation section about --no-ff below and why you always want to use it when merging into clean branches.
git checkout prod git merge --no-ff feature1/change-title
Feature 2 is merged to Master for testing (C6):
git checkout master git merge feature2/change-author
Feature 3 is a minor change and you want it out quickly. There would be a merge conflict into Master, but for whatever reason it didn’t get merged to Master for testing. There’s no code involved, just text, so it doesn’t seem like a big deal, right? It’s immediately submitted for code review. However, the pull request has conflicts (dotted red line) because the title was already changed.
git checkout prod git merge --no-ff feature3/change-title git merge --abort
In an effort to resolve conflicts, Master is (accidentally) merged into the Feature 3 branch instead of Production (C8):
git checkout feature3/change-title git merge master # Fix conflicts in style.css vim style.css git commit -a
This doesn’t seem like a big deal, because I just had to fix the title conflict (which is what the pull request to Production was complaining about). But the Feature 3 branch is now “dirty” because it contains Feature 2 (and everything else from Master), which may not be release-ready. If this gets merged into Production, that branch will be dirty as well (C9).
git checkout prod git merge --no-ff feature3/change-title
The scenario that we’re following has played out like this:
Production is now dirty.
Pause and regroup
Hopefully, the problem is noticed right away. The longer it goes, the more feature branches get created off of a dirty Production branch, and the more clean-up is needed.
The best thing to do once the problem is noticed is have all development pause so the situation can be assessed and a clean-up path identified.
But where should we start?
An aside about fast-forward
By default, when Git does a merge, it tries to do a “fast forward,” if possible, without adding a merge commit. It just moves the commit pointer ahead when it can. This article explains the concept nicely (see figures 21 & 22), as does this image:
While Git is being fast and smart about its operations (not introducing extra commits), it can make the history more difficult to grok, especially if there are multiple feature branches in progress concurrently. With fast-forwards (if you omit the --no-ff on all merges), your Git log history would look like this:
git checkout prod git log --oneline --graph * SHA-C8 Merge branch 'master' into feature3/change-title | | * SHA-C6 Merge branch 'feature2/change-author' | | | | * SHA-C5 Feature 2 - changed author | * | SHA-C2 Feature 1 - changed title | |/ * | SHA-C7 Feature 3 - changed title |/ * SHA-C1 Initial check-in
While the history is simpler (less commits), it is more difficult see where things went wrong. In this case, you probably want to move the Production branch pointer to C2. But it’s difficult to distinguish from this log, without any context, that Feature 1 is the only code out of these commits that has been approved to go to Production.
A better way forward (pun intended) would be to have your entire team enable this option globally in Git; so --no-ff never gets accidentally omitted on an important merge:
git config --global merge.ff false
This will always introduce a merge commit when performing a merge. FYI GitHub does a --no-ff merge on all pull requests. Having the extra merges doesn’t bother me, especially if it helps when you need to make fixes.
Reviewing history to pinpoint the problem
With a clearer history, Git log can more helpful in determining where things went wrong:
git checkout prod git log --oneline --graph * SHA-C9 Merge branch 'feature3/change-title' into prod | | * SHA-C8 Merge branch 'master' into feature3/change-title | | | | * SHA-C6 Merge branch 'feature2/change-author' | | | | | | * SHA-C5 Feature 2 - changed author | | * | SHA-C3 Merge branch 'feature1/change-title' | | | | | | |/ | | |/| | * | | SHA-C7 Feature 3 - changed title | |/ / * | | SHA-C4 Merge branch 'feature1/change-title' into prod | | |/ / |/| / | |/ | * SHA-C2 Feature 1 - changed title |/ * SHA-C1 Initial check-in
If you didn’t configure the merge option stated above, your log may be missing merge commits into Master – C3 “Merge branch ‘feature1/change-title'” and C6 “SHA-C6 Merge branch ‘feature2/change-author'” due to fast forwards. Nonetheless, we should have enough history to set things right.
The previous commit before C9 when Production was clean was C4.
The problem with reverting a merge
Revert sounds like what you want to do in this situation, right? It may be, but we need to understand the consequences. Reverting a single commit is almost always safe, but reverting a merge can have side effects.
At first glance, it seems like we want to revert the merge into Production (C9) to undo the dirtying.
To the Production branch, the merge from Feature 3 looked like commits C5, C6, C7, and C8 because Master was merged into Feature 3.
Reverting a merge creates a new commit (C10 – not shown) that undoes all of those commits.
git checkout prod git revert -m 1 [SHA-C9]
What it does (from the Git-revert main page):
Reverting a merge commit declares that you will never want the tree changes brought in by the merge. As a result, later merges will only bring in tree changes introduced by commits that are not ancestors of the previously reverted merge. This may or may not be what you want.
Put more simply, reverting a merge commit sort-of black lists any of those commits from ever being part of the Production branch again. In this case, the Feature 2 (C5) won’t ever make it into Production. Git will say it’s up to date:
git checkout prod git merge --no-ff feature2/change-author # Already up-to-date.
More complex examples about Git reverting merges (and reverting reverts) can be found here: https://git-scm.com/blog/2010/03/02/undoing-merges.html
Reset to get that shiny clean branch back
Instead, let’s reset to the last clean point. By using git reset --hard and git push --force-with-lease we can effectively remove all bad commits and reset back to a known good point. These are unforgiving operations, so double check your work before executing.
git checkout prod git reset --hard [SHA-C4]
When pushing the reset to the remote repository, you’ll have to force it (with lease). To Git, it looks like you’re rewinding, which it doesn’t like to do unless you really mean it:
git push --force-with-lease origin prod
At this point, the Production branch is clean again. Other developers can do a reset on their local copies of Production:
git checkout prod git fetch --all git reset --hard origin/prod
Now, new (clean) feature branches can be created from Production, and work can continue. But what about existing branches?
Recreating clean, existing branches with cherry-pick
The developer of Feature 3 can reset their branch to before the merge from Master:
git checkout feature3/change-title git reset --hard [SHA-C7]
git push --force-with-lease origin feature3/change-title
But, there may have been additional changes made past C7 that are wanted. You could cherry-pick them directly into the reset branch, if you’d like. To keep history intact, a better approach would be to not reset the branch and instead create a new branch from Production (after Production has been cleaned up). Then, cherry-pick the commits from your dirty branch that you want to keep:
git checkout prod git checkout -b feature3/change-title-clean git cherry-pick [SHA-C7]
You’ll have conflicts here: the ones that should have been resolved on your branch from Production (instead of Master) earlier.
# Fix conflicts in style.css vim style.css git add style.css git cherry-pick --continue
Continue to cherry-pick any other commits that you want to add to this branch in order from earliest to latest. Then, you can push this new branch up and use it for your pull request / code review (if you want to keep the history of the dirty branch around). If you’re absolutely sure things are fixed, you can delete your other dirty branch and rename this one before pushing:
git checkout prod # b/c you can't rename the branch you're on git branch -D feature3/change-title git branch -m feature3/change-title-clean feature3/change-title
git push --force-with-lease origin feature3/change-title
Fixing branches started after the bad merge
If some time passed before all of this was noticed, and some new feature branches were created from C9, you can locate them with:
git branch -a --contains [SHA-C9]
That branch search will look in your local Git repository and on any remotes. But if a fellow developer has an un-pushed branch on their local system that stems from C9, you still may not find it until later. The best thing to do in these situations is be honest and communicate. Having them run that command locally will help you grasp how far-reaching the problem is.
You can use the same cleanup and cherry-pick process outlined above. Create a new branch from the clean Production branch, and cherry-pick only the commits that had to do with your feature.
If this happens to you, I hope these guidelines help. Always remember to learn from your mistakes.
I never lose. I either win or learn.
– Nelson Mandela
Comments