|
|
|
1 Issues and Problems in Decision Making Under Uncertainty |
|
|
3 | (24) |
|
|
3 | (4) |
|
1.1.1 Decision Making as Constrained Optimization Problems |
|
|
3 | (1) |
|
|
4 | (1) |
|
1.1.3 The Role of Information in the Presence of Uncertainty |
|
|
5 | (2) |
|
1.2 Problem Formulations and Information Structures |
|
|
7 | (5) |
|
1.2.1 Stochastic Optimal Control (SOC) |
|
|
7 | (3) |
|
1.2.2 Stochastic Programming (SP) |
|
|
10 | (2) |
|
|
12 | (4) |
|
1.3.1 A Basic Example in Static Information |
|
|
12 | (1) |
|
1.3.2 The Communication Channel |
|
|
13 | (3) |
|
1.3.3 Witsenhausen's Celebrated Counterexample |
|
|
16 | (1) |
|
1.4 Discretization Issues |
|
|
16 | (9) |
|
1.4.1 Problems with Static Information Structure (SIS) |
|
|
17 | (1) |
|
1.4.2 Working Out an Example |
|
|
18 | (7) |
|
|
25 | (2) |
|
2 Open-Loop Control The Stochastic Gradient Method |
|
|
27 | (38) |
|
|
27 | (1) |
|
2.2 Open-Loop Optimization Problems |
|
|
28 | (3) |
|
|
28 | (2) |
|
2.2.2 Sample Approximation in Stochastic Optimization |
|
|
30 | (1) |
|
2.3 Stochastic Gradient Method Overview |
|
|
31 | (8) |
|
2.3.1 Stochastic Gradient Algorithm |
|
|
31 | (3) |
|
2.3.2 Connection with Stochastic Approximation |
|
|
34 | (5) |
|
|
39 | (6) |
|
2.4.1 Auxiliary Problem Principle |
|
|
40 | (1) |
|
2.4.2 Stochastic Auxiliary Problem Principle Algorithm |
|
|
41 | (1) |
|
2.4.3 Convergence Theorem |
|
|
42 | (3) |
|
|
45 | (1) |
|
2.5 Efficiency and Averaging |
|
|
45 | (5) |
|
2.5.1 Stochastic Newton Algorithm |
|
|
45 | (3) |
|
2.5.2 Stochastic Gradient Algorithm with Averaging |
|
|
48 | (1) |
|
2.5.3 Sample Average Approximation |
|
|
49 | (1) |
|
2.6 Practical Considerations |
|
|
50 | (6) |
|
|
51 | (1) |
|
2.6.2 Tuning the Standard Algorithm |
|
|
51 | (3) |
|
2.6.3 Robustness of the Averaged Algorithm |
|
|
54 | (2) |
|
|
56 | (1) |
|
|
57 | (8) |
|
2.8.1 Robbins-Siegmund Theorem |
|
|
57 | (1) |
|
|
58 | (1) |
|
2.8.3 Proof of Theorem 2.17 |
|
|
59 | (6) |
|
Part II Decision Under Uncertainty and the Role of Information |
|
|
|
3 Tools for Information Handling |
|
|
65 | (1) |
|
|
65 | (30) |
|
3.2 Basic Facts on Binary Relations and on Lattices |
|
|
66 | (5) |
|
|
66 | (4) |
|
|
70 | (1) |
|
3.3 Partitions and Fields Approach |
|
|
71 | (9) |
|
3.3.1 The Lattice of Partitions/Equivalence Relations |
|
|
71 | (3) |
|
3.3.2 The Lattice of π-Fields (Partition Fields) |
|
|
74 | (4) |
|
3.3.3 The Lattice of σ-Fields |
|
|
78 | (2) |
|
3.4 Mapping Measurability Approach |
|
|
80 | (7) |
|
3.4.1 Measurability of Mappings w.r.t. Partitions |
|
|
80 | (1) |
|
3.4.2 Measurability of Mappings w.r.t. π-Fields |
|
|
81 | (5) |
|
3.4.3 Measurability of Mappings w.r.t. σ-Fields |
|
|
86 | (1) |
|
3.5 Conditional Expectation and Optimization |
|
|
87 | (6) |
|
3.5.1 Conditional Expectation w.r.t. a Partition |
|
|
87 | (3) |
|
3.5.2 Interchanging Minimization and Conditional Expectation |
|
|
90 | (2) |
|
3.5.3 Conditional Expectation as an Optimal Value of a Minimization Problem |
|
|
92 | (1) |
|
|
93 | (2) |
|
4 Information and Stochastic Optimization Problems |
|
|
95 | (38) |
|
|
95 | (1) |
|
4.2 The Witsenhausen Counterexample |
|
|
96 | (5) |
|
4.2.1 A Simple Linear Quadratic Control Problem |
|
|
96 | (2) |
|
4.2.2 Problem Transformation Exploiting Sequentiality |
|
|
98 | (2) |
|
4.2.3 The Dual Effect of the Initial Decision |
|
|
100 | (1) |
|
4.3 Other Information Patterns |
|
|
101 | (3) |
|
4.3.1 Full Noise Observation |
|
|
101 | (1) |
|
4.3.2 Classical Information Pattern |
|
|
102 | (1) |
|
4.3.3 Markovian Information Pattern |
|
|
103 | (1) |
|
4.3.4 Past Control Observation |
|
|
103 | (1) |
|
4.3.5 The Witsenhausen Counterexample |
|
|
104 | (1) |
|
4.4 State Model and Dynamic Programming (DP) |
|
|
104 | (7) |
|
|
105 | (1) |
|
4.4.2 State Feedbacks, Decisions, State and Control Maps |
|
|
106 | (2) |
|
|
108 | (1) |
|
4.4.4 Stochastic Optimization Problem |
|
|
109 | (1) |
|
4.4.5 Stochastic Dynamic Programming |
|
|
109 | (2) |
|
4.5 Sequential Optimization Problems |
|
|
111 | (21) |
|
4.5.1 Sequential Optimal Stochastic Control Problem |
|
|
112 | (3) |
|
4.5.2 Optimal Stochastic Control Problem in Standard Form |
|
|
115 | (4) |
|
|
119 | (1) |
|
4.5.4 Dynamic Programming Equations |
|
|
119 | (13) |
|
|
132 | (1) |
|
5 Optimality Conditions for Stochastic Optimal Control (SOC) Problems |
|
|
133 | (22) |
|
|
133 | (1) |
|
5.2 SOC Problems, Formulation and Assumptions |
|
|
134 | (4) |
|
|
135 | (1) |
|
|
135 | (1) |
|
|
136 | (1) |
|
5.2.4 The Stochastic Programming (SP) Version |
|
|
137 | (1) |
|
5.3 Optimality Conditions for the SP Formulation |
|
|
138 | (3) |
|
5.3.1 Projection on the Feasible Set |
|
|
138 | (2) |
|
5.3.2 Stationary Conditions |
|
|
140 | (1) |
|
5.4 Optimality Conditions for the SOC Formulation |
|
|
141 | (5) |
|
5.4.1 Computation of the Cost Gradient |
|
|
141 | (3) |
|
5.4.2 Optimality Conditions with Non-adapted Co-States |
|
|
144 | (1) |
|
5.4.3 Optimality Conditions with Adapted Co-States |
|
|
145 | (1) |
|
|
146 | (5) |
|
5.5.1 Markovian Setting and Assumptions |
|
|
146 | (1) |
|
5.5.2 Optimality Conditions with Non-adapted Co-States |
|
|
147 | (2) |
|
5.5.3 Optimality Conditions with Adapted Co-States |
|
|
149 | (1) |
|
5.5.4 Optimality Conditions from a Functional Point of View |
|
|
150 | (1) |
|
|
151 | (4) |
|
Part III Discretization and Numerical Methods |
|
|
|
6 Discretization Methodology for Problems with Static Information Structure (SIS) |
|
|
155 | (26) |
|
|
156 | (4) |
|
6.1.1 Set-Theoretic Quantization |
|
|
156 | (1) |
|
6.1.2 Optimal Quantization in Normed Vector Spaces |
|
|
157 | (3) |
|
6.2 A Systematic Approach to Discretization |
|
|
160 | (14) |
|
6.2.1 The Problematics of Discretization |
|
|
160 | (1) |
|
6.2.2 The Approach Inspired by Pennanen's Work |
|
|
161 | (7) |
|
6.2.3 A Constructive Proposal |
|
|
168 | (6) |
|
6.3 A Handicap of the Scenario Tree Approach |
|
|
174 | (5) |
|
6.3.1 How to Sample Noises to Get Scenario Trees |
|
|
174 | (1) |
|
|
175 | (4) |
|
|
179 | (2) |
|
|
181 | (30) |
|
|
181 | (2) |
|
7.2 A Simple Benchmark Problem |
|
|
183 | (4) |
|
|
183 | (3) |
|
7.2.2 Numerical and Functional Data |
|
|
186 | (1) |
|
7.3 Manipulating Functions with a Computer and Implementation in Dynamic Programming (DP) |
|
|
187 | (5) |
|
|
187 | (1) |
|
7.3.2 Discrete Representation of a Function |
|
|
188 | (2) |
|
7.3.3 The Discrete DP Equation |
|
|
190 | (2) |
|
7.3.4 Application to the Benchmark Problem |
|
|
192 | (1) |
|
7.4 Resolution by the Scenario Tree Technique |
|
|
192 | (8) |
|
7.4.1 General Considerations |
|
|
193 | (1) |
|
7.4.2 Formulation of the Problem over a Scenario Tree |
|
|
194 | (2) |
|
7.4.3 Optimality Conditions and Resolution |
|
|
196 | (1) |
|
7.4.4 About Feedback Synthesis |
|
|
197 | (1) |
|
7.4.5 Results Obtained for the Benchmark Problem |
|
|
198 | (2) |
|
|
200 | (6) |
|
|
201 | (3) |
|
7.5.2 Results Obtained for the Benchmark Problem and Comments |
|
|
204 | (2) |
|
|
206 | (5) |
|
Part IV Convergence Analysis |
|
|
|
8 Convergence Issues in Stochastic Optimization |
|
|
211 | (44) |
|
|
211 | (2) |
|
|
213 | (6) |
|
8.2.1 Epi-Convergence and Mosco Convergence |
|
|
213 | (4) |
|
8.2.2 Convergence of Subfields |
|
|
217 | (2) |
|
8.3 Operations on Integrands |
|
|
219 | (8) |
|
|
219 | (1) |
|
|
220 | (2) |
|
|
222 | (1) |
|
8.3.4 Conditional Expectation of a Normal Integrand |
|
|
223 | (1) |
|
8.3.5 Interchange of Minimization and Integration |
|
|
224 | (3) |
|
8.4 Application to Open-Loop Optimization Problems |
|
|
227 | (1) |
|
8.5 Application to Closed-Loop Optimization Problems |
|
|
228 | (24) |
|
8.5.1 Main Convergence Theorem |
|
|
228 | (3) |
|
8.5.2 Revisiting a Basic Example in Static Information |
|
|
231 | (2) |
|
8.5.3 Discussion About Related Works |
|
|
233 | (2) |
|
8.5.4 Revisiting the Example of Sect. 1.4.2 |
|
|
235 | (10) |
|
8.5.5 Companion Propositions to Theorem 8.42 |
|
|
245 | (7) |
|
|
252 | (3) |
|
Part V Multi-Agent Systems |
|
|
|
9 Multi-Agent Decision Problems |
|
|
255 | (38) |
|
|
255 | (1) |
|
9.2 Witsenhausen Intrinsic Model |
|
|
256 | (6) |
|
9.2.1 The Extensive Space of Decisions and States of Nature |
|
|
256 | (3) |
|
9.2.2 Information Fields and Policies |
|
|
259 | (3) |
|
9.3 Causality and Solvability for Stochastic Control Systems |
|
|
262 | (3) |
|
9.3.1 Solvability and Solvability/Measurability |
|
|
262 | (2) |
|
|
264 | (1) |
|
9.3.3 Solvability, Causality and "State" |
|
|
264 | (1) |
|
9.4 Four Binary Relations Between Agents |
|
|
265 | (9) |
|
9.4.1 The Precedence Relation B |
|
|
265 | (2) |
|
9.4.2 The Subsystem Relation |
|
|
267 | (3) |
|
9.4.3 The Information-Memory Relation M |
|
|
270 | (2) |
|
9.4.4 The Decision-Memory Relation D |
|
|
272 | (2) |
|
9.5 A Typology of Stochastic Control Systems |
|
|
274 | (12) |
|
9.5.1 A Typology of Systems |
|
|
274 | (3) |
|
9.5.2 Examples of Systems with Two Agents |
|
|
277 | (5) |
|
9.5.3 Partially Nested and Sequential Systems |
|
|
282 | (3) |
|
|
285 | (1) |
|
9.6 Policy Independence of Conditional Expectations and Dynamic Programming |
|
|
286 | (6) |
|
9.6.1 Policy Independence of Conditional Expectations |
|
|
286 | (2) |
|
9.6.2 Application to Decomposition by Dynamic Programming |
|
|
288 | (4) |
|
|
292 | (1) |
|
10 Dual Effect for Multi-Agent Stochastic Input-Output Systems |
|
|
293 | (16) |
|
|
293 | (1) |
|
10.2 Multi-Agent Stochastic Input-Output Systems (MASIOS) |
|
|
294 | (6) |
|
10.2.1 Definition of Multi-Agent Stochastic Input-Output Systems |
|
|
294 | (1) |
|
|
295 | (2) |
|
10.2.3 Precedence and Memory-Communication Relations |
|
|
297 | (1) |
|
10.2.4 A Typology of MASIOS |
|
|
298 | (2) |
|
10.3 No Open-Loop Dual Effect and No Dual Effect Control Laws |
|
|
300 | (7) |
|
10.3.1 No Open-Loop Dual Effect (NOLDE) |
|
|
301 | (1) |
|
10.3.2 No Dual Effect Control Laws |
|
|
301 | (1) |
|
10.3.3 Characterization of No Dual Effect Control Laws |
|
|
302 | (5) |
|
|
307 | (2) |
Appendix A Basics in Analysis and Optimization |
|
309 | (18) |
Appendix B Basics in Probability |
|
327 | (22) |
References |
|
349 | (8) |
Index |
|
357 | |