Merge tools

Posted 30 Apr 2000 at 23:56 UTC by lkcl Share This

Ever tried to keep up, on an hourly basis, with two other developers? Each of you is working in your own branch. One is doing far-out research. One is doing the next stable release. One is doing the current stable release. Never seriously used X-windows or emacs? Where do you compromise your development-style principles? When you're faced with 300,000 lines of code, and daily merges [that took one developer only one hours' work] are cut from six hours to 30 minutes. Enter dirdiff.

So you use vi, linux-console-mode with six 80x60 screens, diff, patch, ctags and grep. Get a life and use emacs? No thanks. tried it. Was told that it didn't have different modes for text-editing and text-entry, it was better all-round. Had this wonderful thing called emerge, it could solve all my problems of how to do, as Jeremy Allison described it from over a year ago, "The MFH (tm)" - The Merge From Hell.

One Samba team member had already tried, and failed. To give you an idea of the size of the task, the diff file between the stable Samba 2.0 tree and what is now the Samba TNG tree (The Next Generation), was 7 megabytes, and that was six months ago.

Two and a half years ago, the NT Domains for Unix project started, with Paul Ashton's "Welcome to the SAMBA Domain" work. Since that time, when the Samba source was only 150,000 or so lines of code, both the stable Samba 2.0 production release tree and the TNG branch added an extra 50,000 lines of code - each. Fortunately, as it turns out, the majority of the work in each branch was done in totally different areas. The remaining areas are a nightmare to merge, and a pain in the wrists to keep up-to-date on a daily basis.

So, what tools were available? Well... emerge. Except that means, emacs. Eventually, I cornered someone, late on a Friday, for instructions on how to use emerge. and emacs. Starting with one file, it quickly became obvious that this simply wasn't going to cut it. Press a, b, b, ab OOPS urr, Paul, how do I get out of this, to which the response was, "'Bye, have a good weekend". to which my response was, "Ctrl-c, argh, ctrl-d, ARGH, ctrl-z killall emacs".

If people laugh at me for using an editor that causes me to press keystrokes such as mx:0d'x and yyP in notepad and pine, I will respond, "and what's emerge, then?"

The very same Paul (who is going to kill me for this article, he really doesn't want any more maintainer-request-email than he already gets) is a fan of tcl and wish. In a former existence, instead of abandoning a really large tcl script in favour of a better, faster language, he wrote a code-optimiser for tcl.

One of his tools that he has written in tcl is dirdiff. dirdiff, in a previous incarnation, was able to show you one window with a comparative diff of two or more directories, and you select one file from two of thoes directories and it shows you the differences. If you liked one file better than the other, you could select the "copy" menu to copy one file over the other. Great for getting rid of all those pesky little mods by another developer that you never really liked, anyway, their code always got in the way of yours.

Anyway.

dirdiff-previous-life showed up the different lines in pretty colours. To give you some idea, take a straight diff (diff file1.c file2.c). If there's a <, display that bit in red. If there's a >, display that bit in green. Insert red bits and green bits on-screen in amongst a bit of context before and after, and voila, that's a dirdiff file display. Then someone (I forget who) thought, you know, it would be really good if i could clickety-click on da red bit or da green bit, and it goes into one of the files...

*Fortunately*, Paul *also* needed something like this, as he needed to merge three sets of pppd branches, and so rather than spend time manually going back to diff and patch, he set about adding little windows all the way down the left-hand-side of dirdiff's file display, one for each line of the read bits and green bits. This means that if the diffs are large, wish gets *really* pushed to the limit: that's a lot of little windows. Tell you what, though: I don't care if it's slow, it saves me that much time. And Paul's speed-optimisations help a lot.

So, how do you use the new, improved, dirdiff? Well, at an xterm, you type:

dirdiff ~/samba-tng/source/smbd ~/samba-main/source/smbd &
and up comes a windows with all the files that are common between the two directories. select a file, and you get a dirdiff file-window, with red bits and green bits, and on each red bit and green bit, there is a series of buttons at the left. If you want to make a "change", you select a button, with left-mouse. If you want to make a "group change", use right-mouse and an entire group of buttons will select or deselect. Having selected all your "changes" you wish to make, from the menu, you then select the file to which all the "changes" are to be applied. Behind the scenes, this creates a straight-diff patch file, which is then applied to the file you selected. The result: only the "changes" you selected are made to the file. Hit "Rediff", and wow, you have less differences between the two files than when you started.

To give you some idea of the speed at which this can be done, I use dirdiff by positioning the mouse cursor (IBM thinkpad nibble-thing) over the left-hand-set-of-buttons, then rapidly press down-arrow on the keyboard (about 3 or 4 times a second), interspersed with either right-mouse or left-mouse clicks *without* changing the position of the mouse. In this way, a 3,000-line file with... 50 to 100 sets of diff-style modifications can be updated in about two minutes, especially if you've already gone over exactly the same files about once every two days, for the last three weeks, and you know pretty much instantly the bits that you want to keep and those that you recognise were added by someone else, very recently.

I wasn't kidding about taking six hours to merge one hours' work by another developer, using diff and patch. I wasn't sure exactly which files in which directories had been modified (about ten, out of about a hundred files), and so had to check the whole lot... again, having just spent 48 hours manually merging an extra 70,000 lines of smbd code into the Samba TNG branch. Now, using dirdiff, it's a pretty trivial task to keep up-to-date. I still have to do cvs diff -d -u -w -b -B > diff.output and check the output, before committing... just to make absolutely sure.

I hate X-windows. I think it's a complete waste of a good computers' time to run graphics. You can't edit two files and press alt-f1,f2,f1,f2 page-down f1 page-down, alt-f2,f1 page-down f1 page-down to compare two files, and even if you could, they would be displayed in some stupid variable-width font. [I use this console-screen-switch technique *as well as* diff, quite frequently: the human eye works by detecting changes, and they show up *really* well when you do this, *especially* when doing comparative hex-dumps of Windows NT Network traffic and corresponding Samba Network traffic to find out the exact, unknown, undocumented bit-patterns that *might* be the cause of a major lack of functionality in Samba - SMB is like that...].

So what caused me, who on receipt of a .doc or .xls file will resort to running strings on it in order to deduce its contents, or sends it to someone else for printing, even if it's confidential, to use graphical interfaces?

dirdiff.

http://samba.org/cgi-bin/cvsweb/dirdiff
public cvs instructions


Emacs, posted 1 May 2000 at 04:14 UTC by aaronl » (Master)

If you are looking for power, there is no reason to be afraid of emacs. While people can come up with odd solutions, ediff and friends are great tools. Unless you have an emacs phobia like the author of this article, stick with the best editor that ever existed!

Killing an editor war without it starting, posted 1 May 2000 at 05:17 UTC by k » (Journeyer)

I knew someone would reply this way. I haven't tried dirdiff, but it sounds rather nifty. To me, he's simply written an article about a tool that he has found nifty. I don't like emacs either, just because "I don't". Does that make me a stubborn emacs hater? No.

I plan on looking at dirdiff RSN and seeing what its like. If it can help me merge large codebase changes quicker than cvs update, I'll be happy. :-P

don't let someone else's weird friend ruin your day, posted 1 May 2000 at 06:51 UTC by graydon » (Master)

if you don't want to use emerge, don't use it. but don't pretend that your friend's unwillingness to help you learn it is a blemish on the package. it has its merits, and you completely fail to evaluate them, so why even mention it? for those who might care, emerge has the following useful features:

  • regexp-based masking of diff regions
  • fully keyboard-operable, and even runs on the console; no more X for you!
  • easily customized narrowing, summarizing, hiding, and hilighting of diff regions
  • support for 3-way merges and different diff/patch backends
  • since it runs in emacs, you can merge directly against version control repositories, patches from your mailbox, or through compressed/ftp'ed files
  • directory/multifile merges
  • easily suspend/resume sessions, so that you can hop into a file and fix it, then return to merging

like most emacs things, you will need to read the manual (easily located -- it's a self documenting editor) and be careful the first few times, while getting used to it. but it's a good package, and you really do it a disservice by your comments.

emacs is good, posted 1 May 2000 at 08:33 UTC by lkcl » (Master)

graydon, i mean to do no offense to emacs. i know how to open files, move around, edit and save them: that's it. by contrast, with vi, i know obscure commands such as :1,$s/^*test//g which will do a search on every line, deleting on each line from the beginning of a line up to and including the word test, if the word test occurs on that line. i more frequently use this - :1,$s/:*$//g (for all lines, delete from any colon onwards, to the end of the line) with, say, grep */*.c some_function > foo. in this way (and there are probably better ways, i am sure :) i get to find all the filenames with some_function in them, so that i can do further damage to them, say, with sed :)

my point is, i tried emacs, and with no knowledge, and with a task that took others who also do not know emacs (or dirdiff, it wasn't available) three full weeks to *fail to complete*, i used dirdiff with practically zero learning curve. the only tricky bit was remembering that the buttons mean "action this change", but you haven't said (yet) which file to apply it to: you do that from the menu *after* you have selected all "changes to action".

Re: emacs, posted 1 May 2000 at 08:38 UTC by lkcl » (Master)

*grin*. tried it in 1990, that's when i learned about ctrl-x ctrl-s and ctrl-x ctrl-c, and um... i forget, is it esc-v for page down and ctrl-v for page up? um... :) it's a learning-curve thing: i just never got round to it.

[thinking back] incredibly, in 1988, imperial college gave us zero-prior-unix-experience first-year students a basic tutorial in how to use vi, and that's what we all used! a few of the more savvy students decided to learn emacs, but the rest of us... :)

Do we need this kind of articles?, posted 1 May 2000 at 10:27 UTC by neo » (Master)

I don't want to discourage anyone from posting articles on the frontpage, but in this case, I think that most of this article should go to /dev/null (haven't we all seen enough editor-wars before ?) and the rest might be worse a note on freshmeat. But then freshmeat already shows six different diff-tools, which makes me think that dirdiff sort of reinvents the wheel once again.

Re: do we need this kind of articles, posted 1 May 2000 at 11:19 UTC by lkcl » (Master)

hi neo, thanks for your comments.

editor-bashing? you're absolutely right: no thanks, not interested. telling people about a new tool that, particularly for someone with really bad r.s.i, has saved an awful lot of time and typing, and telling people in a light-hearted way? yehh, any day.

the title of the article is "merge tools". it's great that you went to the trouble of doing a lookup on freshmeat.net. do you have either the urls of the six merge-assistance tools you found, or if it is simpler, could you please let us know what search criteria you used on freshmeat.net, to find them? if people are then interested in discussing them, they can. thanks very much.

[following neo's suggestion, i searched with just the word "merge" and it shows something called tkdiff, which looks like it is also open source, and can even do a 3-way diff / merge].

What kind of articles do we want?, posted 1 May 2000 at 15:04 UTC by Iain » (Master)

People complain that there's too many Advogato based articles, people complain when there's an article that doesn't really involve computers, people complain when there's an article that describes a new tool. All of which brings me to the question:

What sort of articles do we want on the front page?

I personally don't care what goes on the front page (although the number of Advogato articles were just getting a bit much :). What I'd really like to see, are articles that provide some insight into life in general, maybe computer related, maybe not meantioning computers at all. Something that could create a discussion (one that doesn't degenerate into a flame war...)

I'm def. going to check out dirdiff, it sounds like it could be useful. And saying "there's 6 things on Freshmeat to do this" doesn't help if you didn't know that tools like this existed.

viper-mode, posted 1 May 2000 at 18:25 UTC by jimd » (Master)

As to the charge that emacs has no separate modes for entry and editing: Actually it has vi-mode, vip-mode and viper-mode (all different implementations of vi emulation for emacs).

viper is the best of these. It's what I use with xemacs.

However, I will note that even with viper (which handles basic vi emulation within your buffers) you'll want a set of macros to handle the many emacs/xemacs features that have no vi equivalent.

Yes, emerge has one of those interfaces that only rms could love. It is hard to use and harder to learn.

I just figured I'd let all my fellow console/curses mode vi sufferers know that your fingers need not pine away for their familiar sequences just because you find yourself needing emacs applications like GNUS mh-e etc.

BTW: xemacs does support ncurses colors. That's the main reason I use it over GNUmacs.

Re: do we need this kind of articles, posted 1 May 2000 at 20:35 UTC by neo » (Master)

lkcl wanted to know what search criteria I used for my freshmeat search. Follow this link for the result.

merge tools, posted 2 May 2000 at 03:14 UTC by lkcl » (Master)

thanks, neo. i used the word "merge", it came up with more trash (35 entries). both searches show "tkdiff", which apparently paul evaluated before writing dirdiff, and found it didn't do what he needed.

[update], posted 2 May 2000 at 03:16 UTC by lkcl » (Master)

if anyone has any ideas about how to improve dirdiff, please post them. for example, tkdiff looks like it has a "preview" mode, and paul says that tcl / wish has a "text widget", so it looks like he's going to add that [to dirdiff] because it's trivial to do (and the text widget even has edit and search capabilities).

Some Emacs recommendations, posted 2 May 2000 at 05:43 UTC by dhd » (Master)

I must admit I was also rather turned off by emerge (and actually many other parts of Emacs) when I first tried to use it. However there are better tools that run in Emacs ... of course I'm religious about Emacs but I really do find that I am much more productive having all of these bits in one place.

Ediff is a replacement for emerge, but is much better and better documented - it does nifty things like color-highlighting chunks of diffs to show which bits have actually changed and what's just whitespace, has decent on-line help, and handles all kinds of crazy diff situations (3-way diffs, recursive diffs of directories, diffing only the selected portions of buffers, etc, etc). Plus it integrates with PCL-CVS and the built-in version control interface in Emacs.

Another thing I like to use is diff-mode.el. It lets you navigate diffs efficiently (moving hunk-to-hunk or file-to-file) and also edit them in place - you can delete hunks or files, reverse the direction of diffs, and it will even readjust the offset/length pairs for you if you edit a hunk manually.

Talking of diff display..., posted 2 May 2000 at 14:24 UTC by eivind » (Master)

... I use a tiny little script to show me my diffs with color - it is available from http://people.freebsd.org/~eivind/cdiff

I find this a great improvement when evaluating diffs for any purpose, including merges - though the diffs can't be used for merge directly after having passed through it, of course.

Request, posted 2 May 2000 at 16:29 UTC by Satan » (Master)

More editor wars and flames about people who write articles about writing articles about navel staring. Less content. Please. >:-)

editors at IC, posted 2 May 2000 at 19:21 UTC by listen » (Journeyer)

Interesting that they used to teach vi at Imperial - now they teach people nedit! And that was before it was actually Free. A lot of my fellow students have never even touched the command line, they just use KDE or something... ah well

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page