Camera Selection, Handoff

and Control
BIR BHANU
bhanu@cris.ucr.edu

Center for Research in Intelligent Systems

University of California at Riverside
Riverside, CA 92521, USA
August 3, 2018
www.vislab.ucr.edu
Camera Selection and Hand-off
• How to track multiple persons using
multiple cameras?
• Which camera is the ‘best’ for each
person? Is the ‘best’ camera unique?
• How to get a stable solution?
• No detail calibration is required.
• Some Related Work
• Need topology information (Not
flexible for active cameras)
• Master-slave scheme (require
overlapping FOVs)
• No “best camera” is selected

www.vislab.ucr.edu
 Why Game Theory?
Gam e Theory Cam era Assignm ent
Useful for analyzing Needs
• Interaction • Cooperation
• Conflict • Competition

Game: the interactive continuous process among agents

Player: agents in the game, decision-maker

Cameras Players
Camera Assignments/Hand-off Multi-player Game
IEEE Sensors Journal 2010

www.vislab.ucr.edu
 Potential Game Approach

• Framework

Best Paper Award IEEE/ACM ICDSC 2008, Stanford

www.vislab.ucr.edu
 Bargaining in a Video Network

W hy Bargaining is needed?
• Competition among cameras, for tracking a person, will finally lead
to a Nash Equilibrium, which may NOT be unique or stable.
• Bargaining mechanism makes the cameras come to a compromise
and generate a stable solution.

W hat do w e do for bargaining?

• Assign a camera to each person according to a set of probabilities of
assigning different cameras to this person. The camera with the
highest probability wins.
• The probabilities are calculated based on the previous bargaining
steps and will finally converge.

www.vislab.ucr.edu
 Performance Evaluation
• Camera selection and handoff algorithms:
 Potential game approach (ICDSC 2008)
 Weakly acyclic game approach (ICDSC 2009)
 Co-occurrence to occurrence ratio approach
• Trackers (color-based):
 CamShift tracker
 Particle filter tracker
• Face detector:
 OpenCV face detector

www.vislab.ucr.edu
 COMPARISONS
Method Approach Comments
The Potential Models the camera assignment problem as a The conflicts that arise in the camera
Game (PG) potential game. Applies a bargaining mechanism assignment are not stated clearly. The
to negotiate among available cameras. theoretical model falls into a narrow
category of a game.

The Weakly Uses a weakly acyclic game model for camera Performs the camera assignment and
Acyclic Game assignment, hand-off and control. Learning is hand-off based on multiple criteria using
(WAG) performed at each time instant to make an the weakly acyclic game model. No
assignment. calibration or topology of the camera
network is needed.

Co-occurrence Intuitive efficient approach; Acceptable results Time consuming point correspondence.
to Occurrence when there are few occlusions and few cameras When correspondence fails or occlusion
Ratio Approach and objects. happens, there is handoff ambiguity;
(VSSN 06) computing becomes complicated with the
# of cameras/objects; FOVs have to be
overlapped.

www.vislab.ucr.edu
 Examples
• Indoor, 3 cameras, 5 persons

www.vislab.ucr.edu
 Examples

• Outdoor, 4 cameras, 6 persons

www.vislab.ucr.edu
 Camera Selection Using Game Theory

4 Cameras, 6 Persons --- 78 handoffs, 4 errors,~ 5% error rate

Different
colors for
different
persons.
Bounding
boxes for
selected
camera
and
person
only.

www.vislab.ucr.edu
 People Trajectories
• Commercially available Axis cameras
• People Trajectories in Experiments:

C1: 3 cameras C2: 3 cameras C3: 3 cameras C4: 4 cameras

5 persons 1 person 2 persons 6 persons

C5: 4 cameras C6: 4 cameras C7: 4 cameras

4 persons 4 persons 4 persons
www.vislab.ucr.edu
 Scenarios
𝑁𝑁𝑐𝑐 : No. of cameras 𝑁𝑁𝑝𝑝 : no. of persons 𝐼𝐼𝑑𝑑 : indoor 𝑂𝑂𝑑𝑑 : Outdoor
𝑂𝑂𝑓𝑓 : overlapped FOVs 𝑁𝑁𝑓𝑓 : non-overlapped FOVs 𝐷𝐷: 9am-5pm 𝑁𝑁: 5pm-7pm
𝐷𝐷𝑐𝑐 : distinct colors 𝑅𝑅𝑐𝑐 : random colors 𝐿𝐿: number of frames

Nc NP Id Od Of Nf D N Dc Rc L
C1 3 5 √ √ √ √ 173
C2 3 1 √ √ √ √ 103
C3 3 2 √ √ √ √ 128
C4 4 6 √ √ √ √ 697
C5 4 4 √ √ √ √ √ 964
C6 4 4 √ √ √ √ √ 1194
C7 4 4 √ √ √ √ √ √ 1600

www.vislab.ucr.edu
 Example Frames
3 cameras 5 persons, indoor

Camera Camera Camera

www.vislab.ucr.edu
1 2 3
 Example Frames
C6: 4 cameras 4 persons, outdoor, dark

Camera Camera Camera Camera

www.vislab.ucr.edu
1 2 3 4
 METRICS
Ground-Truth
• Ground-truth for tracking: Videos are annotated.
• Ground-truth for camera handoff: Enumerate all the camera
assignment possibilities, based on the annotated video, and
exhaustively choose the best one according to the corresponding
criteria - Size; position; view
Errors
• Tracking error: The overlap of the bounding box returned by the
tracker and that of the ground-truth is less than 30% or the former
one is over 1.5 times of the latter one.
• Face detection error: The overlap of the bounding box returned by
the face detector and that of the ground truth is less than 30% or
the former one is over 1.5 times of the latter one.
• Camera handoff error: A camera handoff error occurs when there
is a handoff in the ground-truth but no hand-off is detected in the
experimented approach or there is no hand off in the ground truth
but there is one in the experimented approach.

www.vislab.ucr.edu
 RESULTS
1. Results for tracking (T1: CamShift; T2: PF)
Tracking error rates (%) in all experimental cases.
C1 C2 C3 C4 C5 C6 C7
T1 16.7 2.0 12.2 20.8 25.3 35.2 19.6
T2 8.6 1.2 9.3 14.9 22.8 30.7 14.3

2. Result for face detection

Face detection error rates (%) on different trackers in all
experimental cases for face detection
C1 C2 C3 C4 C5 C6 C7
T1 19.5 12.3 15.5 25.5 30.1 30.0 25.7
T2 15.9 11.1 12.4 21.3 32.3 29.1 19.8

www.vislab.ucr.edu
 RESULTS
3. Camera handoff results (ICPR 2010)
For C1-C7: (x, y, z), x: the number of correct handoffs, y: number of false
alarms, z: number of false dismissal; For the overall performance: (x, y), x: the
overall number of correct handoffs, y: the overall number of error handoffs)
Approach C1 C2 C3 C4 C5 C6 C7 Overall
/ tracker
Utility- 25, 15, 7 8, 0, 0 12, 1, 2 90, 19, 65 36, 12, 60 39, 21, 50 79, 22, 34 289, 308
based / T1
Utility- 28, 10, 4 8, 0, 0 13, 1, 2 116, 12, 50, 10, 46 32, 18, 57 99, 19, 14 349, 232
based / T2 39
WAG / T1 24, 14, 8 8, 0, 0 13, 1, 1 92, 18, 63 45,16, 51 30, 18, 59 78, 18, 35 290, 302

WAG / T2 30, 12, 2 8, 0, 0 12, 0, 2 122, 17, 67, 16, 29 34, 17, 55 102, 13, 377, 205
33 11
COR / T1 18, 12, 14 5, 1, 3 10, 8, 4 69, 20, 86 32, 12, 64 29, 20, 60 50, 10, 63 213, 377

COR / T2 19, 12, 13 5, 1, 3 12, 13, 2 77, 19, 78 32, 9, 64 32, 21, 57 69, 11, 44 246, 347

Ground 32 8 14 155 96 89 113 507, 0

truth

www.vislab.ucr.edu
 Fusion of Multiple Trackers

• Approach

T1 T1
T2 𝐶𝐶1 T2 𝐶𝐶1
Confidence
⋯ Confidence

⋯
T1 T1
𝐶𝐶𝑛𝑛𝐶𝐶 (1) T2
⋯ 𝐶𝐶𝑛𝑛𝐶𝐶(𝑁𝑁𝑃𝑃) T2

best camera best camera

Camera selection for 𝑃𝑃1 Camera selection for 𝑃𝑃𝑁𝑁𝑃𝑃

Final camera selection results

www.vislab.ucr.edu
Fusion of Multiple Trackers

𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑪𝑪𝑪𝑪𝑪𝑪

𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪 𝑶𝑶𝑶𝑶𝑶𝑶 𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑪𝑪𝑪𝑪𝑪𝑪

𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑶𝑶𝑶𝑶𝑶𝑶

𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪 𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑺𝑶𝑶𝑶𝑶𝑶𝑶 Win

𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑪𝑶𝑶𝑶𝑶𝑶𝑶

www.vislab.ucr.edu
 Fusion of Multiple Trackers
• Comparison with individual trackers (PETS 2009 dataset)

Fused OB

MIL SO
B
Fused: fusion of multiple trackers; OB: the online boosting tracker
MIL: the multiple instance learning tracker; SOB: the semi-supervised online
boosting tracker (ICDSC 2011)
www.vislab.ucr.edu
 AUCTION PROTOCOL FOR CAMERA CONTROL

• Previous Approaches for Camera Selection and

Handoff - Only a few papers consider camera active control
(pan/tilt/zoom). They do not consider potentially available
other cameras.
• Build on the auction-based theories in economics for
camera networks. Develop a set of auction protocols for
camera active control. (Smith 1980, TC, Contract Net)
• Summary of the Proposed Approach
Bid price is formulated to have a vector presentation to
consider the “willingness” of a camera to track a particular
object.
The system is a hybrid of distributed/centralized systems.
Most of the computation is decentralized by computing the
bid price locally. Final decision is made by a virtual
auctioneer. ICIP 2010
www.vislab.ucr.edu
TECHNICAL APPROACH
Auction Theory For Control

www.vislab.ucr.edu
METRICS AND PRICE FUNCTION

www.vislab.ucr.edu
 EXPERIMENTS
Map of the Camera Network Used in the Experiments

www.vislab.ucr.edu
Experiment #2: 6 Cameras 4 Persons

www.vislab.ucr.edu
 Experiment #2: 6 Cameras 4 Persons

• Some experimental frames

Frame a

Frame b

Frame c

Frame d

Frame e

www.vislab.ucr.edu Frame f
 Book

www.vislab.ucr.edu
 Book

www.vislab.ucr.edu
 Acknowledgements
• Students: Hoang Nguyen, Ramiro Diaz,
Ankit Patel, Jhon Gonzalez, Yiming Li,
Giovanni Denina

www.vislab.ucr.edu
www.vislab.ucr.edu