About the Authors |
|
ix | |
Preface |
|
xi | |
Symbols and Abbreviations |
|
xiii | |
1 Introduction |
|
1 | (12) |
|
1.1 Elements of System Identification |
|
|
1 | (2) |
|
1.2 Traditional Identification Criteria |
|
|
3 | (1) |
|
1.3 Information Theoretic Criteria |
|
|
4 | (4) |
|
|
6 | (1) |
|
1.3.2 Minimum Information Divergence Criteria |
|
|
7 | (1) |
|
1.3.3 Mutual Information-Based Criteria |
|
|
7 | (1) |
|
1.4 Organization of This Book |
|
|
8 | (1) |
|
Appendix A: Unifying Framework of ITL |
|
|
9 | (4) |
2 Information Measures |
|
13 | (16) |
|
|
13 | (6) |
|
|
19 | (2) |
|
2.3 Information Divergence |
|
|
21 | (2) |
|
|
23 | (1) |
|
|
24 | (2) |
|
Appendix B: a-Stable Distribution |
|
|
26 | (1) |
|
Appendix C: Proof of (2.17) |
|
|
26 | (1) |
|
Appendix D: Proof of Cramer-Rao Inequality |
|
|
27 | (2) |
3 Information Theoretic Parameter Estimation |
|
29 | (32) |
|
3.1 Traditional Methods for Parameter Estimation |
|
|
29 | (5) |
|
3.1.1 Classical Estimation |
|
|
29 | (2) |
|
|
31 | (3) |
|
3.2 Information Theoretic Approaches to Classical Estimation |
|
|
34 | (6) |
|
3.2.1 Entropy Matching Method |
|
|
34 | (1) |
|
3.2.2 Maximum Entropy Method |
|
|
35 | (2) |
|
3.2.3 Minimum Divergence Estimation |
|
|
37 | (3) |
|
3.3 Information Theoretic Approaches to Bayes Estimation |
|
|
40 | (16) |
|
3.3.1 Minimum Error Entropy Estimation |
|
|
40 | (11) |
|
|
51 | (5) |
|
3.4 Information Criteria for Model Selection |
|
|
56 | (1) |
|
|
57 | (1) |
|
Appendix F: Minimum MSE Estimation |
|
|
58 | (1) |
|
Appendix G: Derivation of AIC Criterion |
|
|
58 | (3) |
4 System Identification Under Minimum Error Entropy Criteria |
|
61 | (106) |
|
4.1 Brief Sketch of System Parameter Identification |
|
|
61 | (11) |
|
|
62 | (3) |
|
|
65 | (1) |
|
4.1.3 Identification Algorithm |
|
|
65 | (7) |
|
4.2 MEE Identification Criterion |
|
|
72 | (10) |
|
4.2.1 Common Approaches to Entropy Estimation |
|
|
73 | (3) |
|
4.2.2 Empirical Error Entropies Based on KDE |
|
|
76 | (6) |
|
4.3 Identification Algorithms Under MEE Criterion |
|
|
82 | (22) |
|
4.3.1 Nonparametric Information Gradient Algorithms |
|
|
82 | (4) |
|
4.3.2 Parametric IG Algorithms |
|
|
86 | (5) |
|
4.3.3 Fixed-Point Minimum Error Entropy Algorithm |
|
|
91 | (2) |
|
4.3.4 Kernel Minimum Error Entropy Algorithm |
|
|
93 | (2) |
|
4.3.5 Simulation Examples |
|
|
95 | (9) |
|
|
104 | (18) |
|
4.4.1 Convergence Analysis Based on Approximate Linearization |
|
|
104 | (2) |
|
4.4.2 Energy Conservation Relation |
|
|
106 | (5) |
|
4.4.3 Mean Square Convergence Analysis Based on Energy Conservation Relation |
|
|
111 | (11) |
|
4.5 Optimization of 0-Entropy Criterion |
|
|
122 | (7) |
|
4.6 Survival Information Potential Criterion |
|
|
129 | (14) |
|
|
129 | (2) |
|
4.6.2 Properties of the SIP |
|
|
131 | (5) |
|
|
136 | (3) |
|
4.6.4 Application to System Identification |
|
|
139 | (4) |
|
|
143 | (18) |
|
4.7.1 Definition of 0-Entropy |
|
|
145 | (3) |
|
4.7.2 Some Properties of the 0-Entropy |
|
|
148 | (4) |
|
4.7.3 Estimation of A-Entropy |
|
|
152 | (5) |
|
4.7.4 Application to System Identification |
|
|
157 | (4) |
|
4.8 System Identification with MCC |
|
|
161 | (3) |
|
Appendix H: Vector Gradient and Matrix Gradient |
|
|
164 | (3) |
5 System Identification Under Information Divergence Criteria |
|
167 | (38) |
|
5.1 Parameter Identifiability Under KLID Criterion |
|
|
167 | (19) |
|
5.1.1 Definitions and Assumptions |
|
|
168 | (1) |
|
5.1.2 Relations with Fisher Information |
|
|
169 | (4) |
|
5.1.3 Gaussian Process Case |
|
|
173 | (3) |
|
5.1.4 Markov Process Case |
|
|
176 | (4) |
|
5.1.5 Asymptotic KLID-Identifiability |
|
|
180 | (6) |
|
5.2 Minimum Information Divergence Identification with Reference PDF |
|
|
186 | (19) |
|
|
188 | (8) |
|
5.2.2 Identification Algorithm |
|
|
196 | (2) |
|
5.2.3 Simulation Examples |
|
|
198 | (3) |
|
5.2.4 Adaptive Infinite Impulsive Response Filter with Euclidean Distance Criterion |
|
|
201 | (4) |
6 System Identification Based on Mutual Information Criteria |
|
205 | (34) |
|
6.1 System Identification Under the MinMI Criterion |
|
|
205 | (11) |
|
6.1.1 Properties of MinMI Criterion |
|
|
207 | (4) |
|
6.1.2 Relationship with Independent Component Analysis |
|
|
211 | (1) |
|
6.1.3 ICA-Based Stochastic Gradient Identification Algorithm |
|
|
212 | (2) |
|
6.1.4 Numerical Simulation Example |
|
|
214 | (2) |
|
6.2 System Identification Under the MaxMI Criterion |
|
|
216 | (23) |
|
6.2.1 Properties of the MaxMI Criterion |
|
|
217 | (5) |
|
6.2.2 Stochastic Mutual Information Gradient Identification Algorithm |
|
|
222 | (5) |
|
6.2.3 Double-Criterion Identification Method |
|
|
227 | (11) |
|
Appendix I: MinMI Rate Criterion |
|
|
238 | (1) |
References |
|
239 | |