Zet: Take the union, intersection, etc of files

[yarrow]

Hi all,

Zet (crates.io, Github) is a command-line utility for doing set operations on files considered as sets of lines. For instance, zet union x y z outputs the lines that occur in any of x , y , or z , and zet intersect x y z those that occur in all of them.

Here are the subcommands of zet and what they do:

• zet union x y z outputs the lines that occur in any of x , y , or z.
• zet intersect x y z outputs the lines that occur in all of x , y , and z.
• zet diff x y z outputs the lines that occur in x but not in y or z.
• zet single x y z outputs the lines that occur in exactly one of x , y , or z.
• zet multiple x y z outputs the lines that occur in two or more of x , y , and z.

Zet handles UTF-16 files, so should work OK on Windows. You can install with cargo install zet , or the Github release page has binaries for Linux, Mac, and Windows.

Notes

• Each output line occurs only once, because we're treating the files as sets and the lines as their elements.
• We do take the file structure into account in one respect: the lines are output in the same order as they are encountered. So zet union x prints out the lines of x , in order, with duplicates removed.
• Zet translates UTF-16LE and UTF-16BE files to UTF-8, and ignores Byte Order Marks (BOMs) when comparing lines. It prepends a BOM to its output if and only if its first file argument begins with a BOM.
• Zet ignores all lines endings ( \r\n or \n ) when comparing lines, so two input lines compare the same if their only difference is that one ends in \r\n and the other in \r . Zet ends each output line with \r\n if the first line of its first file argument ends in \r\n , and \n otherwise (if the first line ends in \n or the first file has only one line and that line has no line terminator.)
• Zet reads entire files into memory. Its memory usage is roughly proportional to the file size of its largest argument plus the size of the (eventual) output.

[H2CO3]

@yarrow

Zet reads entire files into memory.

A suggestion for improving this: you could add an option for assuming sorted files. Then you wouldn't need to read everything into memory.

[yarrow]

True. The algorithms would be different, though. Currently Zet is essentially a wrapper around indexmap, which has the advantage that we only need to process one file at a time against the IndexMap (actually IndexSet for most of the subcommands).

To avoid having all of a file in memory at once, we'd need to open all the file arguments and our inner loop would look at each file's current line. For instance:

• Union — output the smallest current line, and advance every file with that line.
• Intersect — if all current lines are identical, output that line and advance all files. Otherwise, advance the file with the least line.
• Diff — advance all files but the first until their current line is equal to or greater than the first file's current line. If the first files current line is different from all the others, output it. Advance the first file.

(Here "advance" means read lines until we come to a line greater than the current one, or the end of the file, and we need to remove files from our list when we've read every line.)

So — doable but completely different code.