VIDZOOM README

This README describes the software used for the paper
Thomas Deselaers, Philippe Dreuw, Hermann Ney. Pan, Zoom, Scan - Time-coherent, Trained Automatic Video Cropping. CVPR 2008


The main program is vidzoom. Its usage can be obtained by calling vidzoom -h


Usage: vidzoom  [-m|--mode=train|zoom] [--stepSize=<N>] [--salDirect] [--OFDirect] [--haar] [--haar-cascade=<file>] [--salLLM] [--ofLLM] [--imgLLM] [--intsalLLM] [
ntimgLLM] [--intsal2ndLLM] [--intof2ndLLM] [--intimg2ndLLM] [--llmWeight=<weight>] [--salWeight=<weight>] [--ofWeight=<weight>] [--winRadius=<radius>] [--testsX=<n
o>] [--steps=<step>] [--lread=<lambda file>] [<video file>] [--results-file=<results file>] [-i <img>] [-o <oldimg>] [-g <ground truth image>] [-j <joerg file>] [-
h=<width>] [-H|--trackingHeight=<heigth>] [-J|--trackingJump=<radius>] [-D|--trackingDelay=<frames>] [-P|--trackingPruning=<threshold>] [-F|--trackingZoomFull] [-Z
actor=<factor> (0.0-1.0)] [-S|--trackingScoring=sum|motion|absolute-motion|squared-motion] [--trackingJumpPenalty=<weight>] [--trackingZoomPenalty=<weight>] [--tra
ZoomPenalty=<weight>] [--trackingAspectRatioPenalty=<weight>] [--trackingCenterPenalty=<weight>] [--output-display] [--output-display-images] [--output-file=<file>
-images] [--FPDEBUG] [-v|--verbose] [-h|--help]

Here is a detailed description of the options:


SOME HINTS FIRST:
 - Memory problems with long/large videos -> -D option
 - weird bugs: do not use --output-file but --output-file-images

COMPILING: 
   There is a Makefile which works fine on linux (for us ;-) and a
   Makefile.macports which works fine on my MacBook pro (snow leopard)
   with opencv 2.1 from macports.

EXAMPLES:
  see below 



GLOBAL OPTIONS:
========================================


*****
 -m, --mode=train|zoom                              default: zoom 

train has to be used to write J-Files which can then be used to train
log-linear models. The software to train these is not contained in
this package.

*****
 --FPDEBUG                                          floating point debug
 -v, --verbose                                      verbose output
 -h, --help                                         show help and exit



OPTIONS FOR THE CUES:
========================================


*****
 --salDirect                                        use saliency directly for finding the region    
 --OFDirect                                         use optical flow directly for finding the region

use the saliency cue or the optical flow cue directly



*****
 --llmWeight=<weight>                               specify weight for llm scoring
 --salWeight=<weight>                               specify weight for direct saliency
 --ofWeight=<weight>                                specify weight for direct optical flow

change the weights of the directy cues. Can be used for tuning but
typically not setting is fine.


*****
 --lread=<lambda file>                              model file

specify the file where the trained lambdas of the log-linear model are
loaded.


*****
 --winRadius=<radius>                               window size = (2*winRadius+1)^2

size of the local region on which the log-linear model is trained

*****
 --stepSize=<N>                                     stepSize for LLM scoring      

how densely are scores computed using the log-linear models 


*****
 --salLLM                                           use saliency in llm
 --ofLLM                                            use optical flow in llm
 --imgLLM                                           use image in llm

use saliency/optical flow/appearance patches as features for log
linear models. Needs to be combined with appropriate --lread flags and
appropriate lambda files



OPTIONS FOR ZOOMING
========================================


Modify the tracking behaviour
 -W, --trackingWidth=<width>                        tracking rectangle width (otherwise autodetect)
 -H, --trackingHeight=<heigth>                      tracking rectangle height (otherwise autodetect)
 -J, --trackingJump=<radius>                        tracking jump width (otherwise autodetect)
 -D, --trackingDelay=<frames>                       autotraceback delay (default full sequence, D=1 only local decisions)
           
    -> if you run into memory problems with large/long videos, try this option e.g. with -D 100 which will trace back the decisions every 100 frames.

 -P, --trackingPruning=<threshold>                  tracking pruning threshold
 -F, --trackingZoomFull                             zoom IN, else zoom OUT
 -Z, --trackingZoomFactor=<factor> (0.0-1.0)        allow zooming IN/OUT in tracking (i.e. letter- or pillarboxing)
 -S, --trackingScoring=sum|motion|absolute-motion|squared-motion tracking scoring function [sum]
 --trackingJumpPenalty=<weight>                     penalise jumps between two rectangles
 --trackingZoomPenalty=<weight>                     penalise smaller/bigger than necessary sizes
 --trackingPredecessorZoomPenalty=<weight>          penalise size changes between two rectangles
 --trackingAspectRatioPenalty=<weight>              penalise aspect ratio different to target aspect ratio (i.e. 16/9 or 4/3)
 --trackingCenterPenalty=<weight>                   penalise leaving the center of the image



 --results-file=<results file>                      file to write results to, in eval compatible format

write results to a text file

 --output-display                                   display tracking results (can be very memory expensive) [default if 'output-file' is set]
 --output-display-images                            output all displayed images to file
 --output-file=<file>                               output video filename of cropped images
   
    -> VERY EXPERIMENTAL 
  
 --output-file-images                               output all cropped images to file


OPTIONS FOR TRAINING
========================================

 -i <img>                                           training image (only for mode training)
 -o <oldimg>                                        predecessor of training image (only for mode training)
 -g <ground truth image>                            gt image (only for mode training)
 -j <joerg file>                                    joerg file to create (only for mode training)


EXPERIMENTAL:

 --haar                                             use haar cascades for finding the object regions
 --haar-cascade=<file>                              trained haar cascade filename

allows to use a Viola/Jones face detection cascade as a cue


 --intsalLLM                                        use integral saliency in llm
 --intofLLM                                         use integral optical flow in llm
 --intimgLLM                                        use integral image in llm
 --intsal2ndLLM                                     use integral saliency in llm (second order)
 --intof2ndLLM                                      use integral optical flow in llm (second order)
 --intimg2ndLLM                                     use integral image in llm (second order)
 --testsX=<no>                                      number of tests in X direction
 --testsY=<no>                                      number of tests in Y direction
 --steps=<step>                                     step for test threshold, can/should occur several times


use sums of features (integral image-style) as features for the
log-linear model. Was a wild idea, never really worked





A purely feature driven methods (similar to block 2 in Table 2 in the paper)
can be obtained using commandlines like this

 ./vidzoom --OFDirect -Z 0.3  VIDEO.avi --results-file RESULT.txt
 ./vidzoom --salDirect -Z 0.3   VIDEO.avi --results-file RESULT.txt
 ./vidzoom --salDirect --OFDirect -Z 0.3  VIDEO.avi --results-file RESULT.txt

However, the results obtained by these are going to be pretty ugly
because the zooming in and out will look terribly.

A much smoother results is obtained by a commandline like this:

./vidzoom --salDirect --OFDirect -J 2 -Z 0.3 --trackingPredecessorZoomPenalty=0.0001  -W 540 -H 360 -D 100 --output-file-images VIDEO.avi

Here, the parameters have the following effects:
 (explained by parameters)
  --salDirect:  use saliency
  --OFDirect: use optical flow
  -- allow for moving the frame by two pixels between two frames
  -- allow a maximally zooming out by 0.3 (i.e. at most 30% of the image is black frames)
  --trackingPredecessorZoomPenalty=0.0001 penalize non-smooth zooming
  --W 540 --H 360 - default target size is 540x360, and due to -Z 0.3 it will maximally be 702x360
  --D 100 optimize over chunks of 100 images
  --output-file-images - dump the result as images
  VIDEO.avi -> input video

To get the purely feature driven method without zoom (Table 1) remove
the -Z 0.3 option.



The result which is reported in the paper in Table 2 using a learned
combination of cues ("S+OF+A (all)") was created using the cues from
this commandline:

./vidzoom --stepSize=2 --lread=lambdas/setup-S_OF_A_all -Z 0.3 --salLLM --ofLLM --imgLLM VIDEO-s2.avi --results-file RESULT-s2.avi.results

for the -s2 videos the setup-S_OF_A_all lambda file was used 
for the -s1 videos the setup-S_OF_A_all_2 lambda file was used

./vidzoom --stepSize=2 --lread=lambdas/setup-S_OF_A_all -Z 0.3 --salLLM --ofLLM --imgLLM VIDEO-s2.avi --results-file RESULT-s2.avi.results-salLLMofLLMimgLLM-2007-12-06.--lread=lambdasTR-TE-Z0.3


To get smooth results here, look at the example above with --trackingPredecessorZoomPenalty=0.0001


ACKNOWLEDGEMENTS: We would like to thank Taoran Lu from University of
Florida, USA for asking lots of smart questions which led to releasing
the code and writing this readme.



CHANGELOG:
2010-09-10: first version of the readme
2010-09-14: Makefile.macports and hints
2010-09-15: more examples and how to get pretty results with a detailed explanation on the parameters