DeDup 0.14 by Loren Merritt, Copyright 2004
based on Dup 2.20 beta 1 by Donald Graft/Klaus Post

DeDup is an Avisynth 2.5 filter, intended to remove duplicate frames in the
interest of compression quality and speed. In some cases, it can also reduce
noise or compression artifacts in the source.

DeDup differs from Dup in that it completely drops the duplicate frames,
rather than keeping several copies of each in the output stream. Then, to
compensate for the dropped frames, DeDup outputs a Matroska-compatible
timecode file to put the remaining frames back in the right places.

This is a two pass filter. While my algorithm could work with one pass, it
wouldn't fit in the Avisynth API.



Usage example:

--- first pass ---

LoadPlugin("DeDup.dll")
AVISource("blah.avi")
DupMC(log="blah.dup.txt")

-------------------
--- second pass ---

LoadPlugin("DeDup.dll")
AVISource("blah.avi")
DeDup(threshold=0.5, maxcopies=10, maxdrops=4, log="blah.dup.txt", times="blah.times.txt")

-------------------




First Pass Options:

DupMC(clip c, bool chroma, string log)

chroma:
	default: true
	Use chroma planes when calculating differences.
	(This is the same as in Dup)

log:
	default: none. required.
	File to save dup metrics for the second pass.

DupMC does not modify the frames passed through it. It only generates the log.
There is no harm in seeking during the first pass: The log will be out of order,
but DeDup doesn't care.
Unlike Dup, which compares the last frame in a run to the first frame,
DupMC always compares adjacent frames. You don't decide on a threshold until
the second pass.



Second Pass Options:

DeDup(clip c, float threshold, float threshold2, int range2, float trigger2,
	  bool show, bool dec, int maxcopies, int maxdrops, bool blend,
      string log, string times, string timesin, string ovr, string debug)

threshold:
	range: 0 - 100
	default: 0.5
	Consecutive frames that differ by less than or equal to this much will be
	merged into a single output frame, taken from the end of the merged sequence.
	(This uses the same units as in Dup)

threshold2:
	range: 0 - 100
	default: 1.0 (but disabled due to range2)
	Use this instead of threshold if there are nearby high-motion frames in
	both directions. (Actually, linearly interpolate between threshold and
	threshold2 based on the amount of motion.)

range2:
	range: 0 - lots (sane values: 0 - maxdrops)
	default: 0 (disabled)
	Search range for threshold2.

trigger2:
	range: 0 - 100
	default: 5.0
	What counts as "high-motion" for threshold2.

show:
	default: false
	Print diagnostic information into each frame.

dec:
	default: true
	Drop frames. Without this, DeDup won't do much. But you might want to
	disable this with show=true, to see the frames that would be dropped.

maxcopies:
	range: 1 - 20
	default: 12
	Break any string of duplicates longer than this.
	(This is the same as in Dup)

maxdrops:
	range: 1 - maxcopies
	default: 3
	Break any string of drops longer than this.
	Don't set this above 4 or so if you want to display subtitles, because many
	media players can't update the subtitles until they get a new video frame.

blend: (BROKEN)
	default: false
	Instead of keeping the last frame from a string of dupes, take the average
	of all the frames in the string.
	(This would be the same as in Dup, but it's currently broken)

log:
	default: none.
	Required, unless you want to specify every frame with ovr.
	The same file you created in the first pass.

times:
	default: none.
	File to write matroska timecodes to.
	Not strictly required, but you'll need it in order to mux.

timesin:
	default: none.
	Read timestamps for a VFR input.
	Matroska timecode formats v1 and v2 accepted.

ovr:
	default: none.
	File to override settings on a per-frame basis. See below for format.

debug:
	default: none.
	File to write the same information as with show=true.



Override File Format:
Example:

  0-350 threshold=.3
350-900 threshold=.7 threshold2=1.5 trigger2=8.0
100 k
200-220 ddk


Any line not beginning with a frame number is ignored as a comment.

There is no notation for 'last frame', aside from using it's realy frame
number. However, it's OK for ranges to extend past the end of the movie.

If two lines override the same property of the same frame, the one later in
the ovr file takes precedence.

Each line is of the format:
	startframe-endframe option [more options]
or
	frame option [more options]

Option overrides accepted:

A string matching /[kd]+/:
	This forces the frame(s) to be (k) kept or (d) considered dups.
	The string of [kd]s will be repeated as necessary to fill the frame range. 
	Forcing a dup does not necessarily cause that frame to be dropped;
	it's still subject to maxdrops.
	This option modifies thresholds, so put any [kd] option after any relevant
	threshold override.

threshold=%f
threshold2=%f
trigger2=%f
	These are the same as the global options.

range2=%d
	The range2 parameter is taken from the frame being considered from dropping,
not from the frames being searched over.

maxdrops=%d
maxcopies=%d
	These values are taken from the frame at the end of the clump of dups, but
it's better to just set them the same over the whole clump.



Tips:
	If you're doing other CPU-intensive filtering, you can save time
(at the cost of space) by saving the first pass (with DupMC) to a huffyuv,
and then run the second pass (with DeDup) from that, instead of repeating
the other filters.
	Don't set threshold too high (unless maxcopies is small):
Unlike the original Dup, I don't do sanity checking on slow zooms/pans/fades.
	The second pass options 'times' and 'debug' take effect as soon as you
open the avs. You don't have to play/encode the video in order to produce
those files.
