Back to Parameter Estimation in Engineering and Science Home
Table of Contents
P ARAMETER ESTIMATION IN ENGINEERING AND S CIENCE J AMES V. B ECK Department of Mechanical Engineering and Division of Engineering Research Michigan State University and K ENNETH J. A RNOLD Department of Statistics and Probability Michigan State University f JOHN WILEY & SONS \ I f New York • C hichester • B risbane • T oronto ( 977 PREFACE Several different courses can be taught from the book. A beginning course at the undergraduate level could be taken from the first five chapters with the emphasis on Chapter 5, which gives an algebraic treatment of linear parameter estimation. Chapters 2 and 3 provide a concise treatment of the required statistical background so that the student would find the course easier if he had previously taken a probability o r statistics course. A graduate course would include Chapters I, 6, 7, a nd 8 with the matrix approach being used freely. Ideally, students would have a background in elementary probability, statistics, and matrices for such a course. Because of the first author's background in heat transfer, many of the examples are taken from this field but knowledge of this field o r any other particular field in engineering or science is not required. This book has a number of unique aspects. One o f these relates to the care in treatment of statistical assumptions. Eight standard assumptions are identified, and a numbering system is given to provide a convenient means of designating them. These assumptions are listed in Sections 5.1.3 and 6.1.5, with an abbreviated form inside the back cover. There are many people who have helped in the preparation of this text to whom we express our appreciation. The notes in various revisions were typed by Mrs. Noralee Burkhardt, Ms. Roberta Smith, Miss Gloria Mannino, Mrs. Kathy Winnie, Mrs. Barbara Coolen, Miss Jan Furtaw, Mrs. Constance Geraci, Miss Patti Truax, Mrs. Pat Holewinski, Mrs. Brenda Stott, and Ms. Mary Beth Mac Donell. The encouragement of the editor, Ms. Beatrice Shube, is particularly appreciated. Some of the results given in this book are related to research done by the first author and supported by National Science Foundation Grants GK-16526, GK-240, GK-2075, and GK-41495. J AMEs V . BECK KENNETH J. A RNOLD CONTENTS Chapter 1 IDtrocIuctioD to aDd Survey of P anmeter EstimatioD 1 Parameters, Properties, a nd States, 2 Purpose o f This Chapter, 3 Related Research, 4 . Relation to Analytical DeSign T heory,S 1.1 I NTRODUcnON, 1.2 FUNDAMENT AL PROBLEMS, 1.3 SIMPLE EXAMPLES, December / 976 E MI Loming. Michigan 1.1.1 1.1.2 1.1.3 1.1.4 1.2.1 1.2.2 1.2.3 1.2.4 1.2.5 1.2.6 1.3.1 1.3.2 1.3 .3 1.4 5 Deterministic o r Classical Problem, 5 State Estimation Problem, 6 Parameter Estimation Problem, 7 Optimum Experiment Problem, 7 Discrimination Problem, 8 Identification Problem, 8 8 Linear Algebraic Model, 8 . Linear First Order Differential Equation Model, 12 Partial Differential Equation Example, 16 SENSITIVITY COEffICIENTS, 17 1 • CONTE/IrJ'S 1.5 IDENTIFIABILITY, 1.6 SUMMARY AND CONCLUSIONS, REFERENCES, PROBLEMS, at 19 2.8.3 2.8.4 Poisson Distributions, 66 Uniform Distributions, 61 2.8.S Normal Distributions, 67 2.8.6 Multivariate Normal Distributions, 71 2.8.7 G amma Distributions, 72 2.8.8 Chi-squared Distributions, 73 2.8.9 t Distributions, 75 2.8.10 F Distributions, 76 2.8.11 Tables a nd C omputer Programs for Commonly used Statistics, 77 23 24 24 Chapter Z Probability 2.1 2.2 RANDOM HAPPENINGS, EVENTS, 2.2.1 2.2.2 2.2.3 2.3 2.3.2 2.3.3 2.6 EXPECTATIONS, 2.6.4 3.1 Chebyshev's Inequality, 62 Weak Law of Large Numbers, 63 Central Limit Theorem, 64 65 Bernoulli Distributions, 65 Binomial Distributions, 65 EXAMPLES OF D lSTRIBtmONS, 2.8.1 2.8.2 62 78 3.1.2 3.2 89 Unbiasedness, 89 Consistency, 90 Efficiency. Minimum Variance Unbiased Estimators, 91 Sufficiency, 93 MaXimum Likelihood Estimators, 94 Estimators a posteriori, 97 Bayes Squared Error Loss Estimators. M AP Estimators, 98 Bayes Intervals, 100 Minimizing Expected Cost, 101 P ROPERTIeS OF ESTIMATORS, 3.2.1 3.2.2 3.2.3 3.2.4 3.2.5 3.2.6 3.2.7 3.2.8 3.2.9 3.3 85 T wo Estimators o f the Center o f a Symmetric Distribution, 85 Estimating a Variance, 87 SOME EXAMPLES O F I!S11MATORS, 3.1.1 5I Expected Value, 5 I Variance, Covariance, and Correlation, 56 Stochastic Processes. Autocovariance, Cross-covariance,59 Stationarity, 61 77 Chapter 3 Introductloa t o Statlsdcs 48 L AW OF LAROE NUMBERS. CENTRAL LIMIT THEOREM, 2.7.1 2.7.2 2.7.3 2.8 PROBLEMS, 43 Conditional Distributions. Discrete Case, 43 Conditional Distributions. Continuous Case, 44 Bayes's Theorem, 46 FUNCTIONS OF RANDOM VARIABLES, 2.6.1 2.6.2 2.6.3 2.7 36 Univariate Probability Distributions. Distribution Functions, 36 Multivariate Distributions, 39 Sample Paths, 42 REFERENCES, CONDITIONAL PROBABIUTlES, 2.4.1 2.4.2 2.4.3 2.S 32 Events. Random Variables and Probabilities, 32 Discrete and Continuous Sample Spaces a nd Associated Probabilities, 34 Assigned Probabilities a nd Experience with Chance Events, 36 PROBABILITY DISTRIBUTIONS, 2.3.1 2.4 29 102 Confidence Intervals for the Mean of a Normal Population when the Population Standard Deviation is Known, 102 Confidence Intervals for the S tandard Deviation o f a N ormal Population, lOS Confidence Intervals for the Mean of a Normal Population when the Population Standard Deviation is Unknown, 106 CONFIDENCE INTERVALS, 3.3.1 3.3.2 3.3.3 CONTENTS 3.4 HYPOTHESIS TESTING, 3.4.1 3.4.2 CONTENTS 108 Two Simple Hypotheses, 109 Problems Reducible to Problems o f Two Simple Hypotheses, I II Generalized Likelihood Ratio Tests. Power, 112 3.4 .3 REFERENCES, 5.2.3 114 5.3 PROBLEMS, Chapter 4 Parameter Estimation Methods 4.1 INTRODUCTION, 4.2 RELATIONS BETWEEN PARAMETERS, 117 117 117 OBSERVED RANDOM VARIABLES 4.3 EXPECTED VALUES, VARIANCES, COVARIANCES, 4.4 LINEAR PROBLEMS, 4.5 LEAST SQUARES, 4.6 GAUSS-MARKOV ESTIMATION, 4.7 SOME OTHER ESTIMATORS, 4.8 COST, 4.9 MONTE CARLO METHODS, REFERENCES, AND 120 5.5 120 121 122 5.2 5.2.2 167 Sum o f Squares, 168 Parameter Estimates, 170 5.6 COEFFICIENT OF MULTIPLE DETERMINATION (R2), 5.7 125 ANALYSIS OF VARIANCE ABOUT m E SAMPLE MEAN, 5.8 125 173 175 ANALYSIS OF VARIANCE ABOUT THE REGRESSION LINE FOR MUL- XI' 178 Expected Values o f $ 2 for Incorrect Model, 180 F-Test with Repealed Data, 181 TIPLE MEASUREMENTS AT EACH 129 5.8.1 5.8.2 130 130 Motivation, 130 Models, 131 Statistical Assumptions Regarding the Measurement Errors, 134 ORDINARY LEAST SQUARES ESTIMATION (OLS), 5.2.1 159 R andom Parameter Case, 159 Subjective Prior Information, 162 Comparison of Viewpoints, 165 MULTIPLE DATA POrNTS, 5.5.1 5.5.2 MOTIVATION, MODELS, AND ASSUMPTIONS, 5.1.1 5.1.2 5.1 .3 MAXIMUM A POSTERIORI (MAP) ESTIMATION, 5.4.1 5.4.2 5.4.3 120 Chapter 5 Introduction to Linear Estimation 5.1 5.4 154 One-Parameter Cases, 155 Two-Parameter Cases, 156 Estimating 0 2 Using Maximum Likelihood, 157 Maximum Likelihood Estimation Using Information from Prior Experiments, 158 MAXIMUM LIKELIHOOD ( ML) ESTIMATION, 5.3.1 5.3.2 5.3.3 5.3.4 114 5.2.2.3 Estimators for Model 4, 1/1 =fJ~ + fJ,<XI = X), 149 5.2.2.4 . Optimal Experiments for Models 3 a nd 4, 149 Comments Regarding Definitions, 151 135 Models I a nd 2 (1/; = Po a nd 1/; = PIX;), 135 5.2 . 1.1 Mean and Variances of Estimates, 136 5.2.1.2 Expected Value of Smin' 137 Two-Parameter Models, 140 5.2.2.1 M odelS, 1/; = P,X/I + P2 X,"l' 140 5.2.2.2 Model 3, 1/;=Po+P,X;, 143 5.9 CONFIDENCE INTERVAL ABOUT THE POINTS ON m E REGRESSION LINE, 184 5.10 VIOLATION OF m E STANDARD ASSUMPTION O F ZERO MEAN ERRORS, 185 5.11 VIOLATION O F m E STANDARD ASSUMPTION OF NORMALITY, 5.12 VIOLATION OF T IlE STANDARD ASSUMPTION OF CONSTANT VARIANCE, 188 186 5.12.1 Variance of e, Given by o l=(X;/8)2a 2, 189 5.12.2 Variance o f £1 E quallo 02."l, 190 5.13 VIOLATION OF STANDARD ERRORS, 190 ASSUMPTION O F UNCORRELATED C ONttNTS 5.14 0 ') C ONttNfS 6.2.4 192 Method of Lagrange Multipliers. 192 Problem of Errors in the Independent and Dependent Variables. 195 5.14.3 Model 2 ('Ii=P~I) Example with Errors in both 'II and t. 198 ERRORS IN INDEPENDENT AND DEPENDENT VARIABLES. 5.14.1 5.14.2 REfERENCES. PROBLEMS. 6.4 20S 213 Elementary Matrix Operations. 213 6.1.1.1 Product of Matrices, 214 6.1.1.2 Transpose of Matrix, 215 6.1. L3 Inverse. Determinant. and Eigenvalues, 215 6.1.1.4 Partitioned Matrix, 218 6.1.1.5 Positive Definite Matrices, 218 6.1.1.6 Trace, 219 Matrix Calculus. 219 Quadratic Form. 221 Expected Value of Matrix and Variance-Covariance Matrix, 222 6.1.4.1 Expected Value Matrix, 222 6.1.4.2 Variance-Covariance Matrix. 222 6.1.4.3 Covariance of Linear Combination of Vector Random Variables. 223 6.1.4.4 Expected Value of a Quadratic Form. 224 Model in Matrix Terms, 225 6.1.5.1 Identifiability Condition, 228 6.1.5.2 Assumptions, 228 Maximum Likelihood Sum of Squares Functions, 230 6.1.6.1 Single Dependent Variable (Single Response) Case, 230 6.1.6.2 Several D ependent Variables (M ultiresponse) Case, 231 Gauss-Markov Theorem. 232 6.1.2 6.1.3 6.1.4 6.1.5 6.1.6 6.1.7 234 Ordinary Least Squares Estimator COLS), 234 Mean of the OLS Estimator, 238 Variance-Covariance Matrix of bu , 238 O RTHOGONAL POLYNOMIALS IN OLS ESTIMATION, 113 6.5 6.7.1 6.7.2 6.7.3 6.7.4 6.7.5 6.7.6 6.7.7 6.8 269 Introduction, 269 Assumptions, 270 Estimation Involving Random Parameters, 270 Estimation with Subjective Information, 272 Uncertainty in 273 +, 275 Introduction, 275 Direct Method, 275 Sequential Method Using Matrix Inversion Lemma, 276 6.7.3.1 Estimation with Only One Observation a t Each Time ( m-I). 278 6.7.3.2 Sequential Analysis of Example 5.2.4, 282 Sequential MAP Estimation, 284 Multiresponse Sequential Parameter Estimation, 286 Ridge Regression Estimation, 287 Comments a nd Conclusions o n the Sequential Estimation Method, 288 SEQUENTIAL ESTIMATION, LEAST SQUARES ESTIMATION. 6.2.1 6.2.2 6.2.3 259 ML Estimation, 259 Estimation of a l , 262 Expected Values of S ML a nd RML, 267 LINEAR MAXIMUM A POSTERIORI E mMATOR ( MAP), 6.6.1 6.6.2 6.6.3 6.6.4 6.6.5 6.7 252 Introduction, 252 Two-Level Factorial Design, 253 Coding the Factors, 254 Inclusion of Interaction Terms in the Model. 255 Estimation, 255 Importance of Replicates, 258 O ther Experiment Designs, 258 MAXIMUM LIKELIHOOD ESTIMATOR, 6.5.1 6.5.2 6.5.3 6.6 248 f ACTORIAL EXPERIMENTS, 6.4.1 6.4.2 6.4.3 6.4.4 6.4.5 6.4.6 6.4.7 INTRODUCTION TO MATRIX NOTATION AND OPERATIONS. 6.1.1 6.2 6.3 204 Chapter 6 Matrix Analysis lor LInear P an meter Estimation 6.1 6.2.S 6.2.6 Relations Involving the Sum of Squares o f Residuals, 240 Distributions of R u a nd bu , 241 Weighted Least Squares (WLS), 247 MATRIX fORMULATION FOR CONFIDENCE INTERVALS AND R EGlONS, 6.8.1 289 Confidence Intervals, 290 ... CONTENTS xvi 6.8.2 6.8.3 6.9 - -- ... ---.~ ............... --..- .. --,---.- --- CONTENTS xvii 7.6 MODEL 7.7.1 7.7.2 7.7 .3 7.7.4 7.7.5 7.8 SEQUENTIAL ESTIMATION FOR MULTIRESPONSE DATA, 387 7.8 .1 Assumptions, 388 7.8.2 Direct Method, 389 7.8.3 Sequential Method Using the Matrix Inversion Lemma, 391 7.8.3.1 Sequential Method for m = I, p Arbitrary, 392 7.8.4 Correlated Errors with Known Correlation Parameters, 393 7.9 MATRIX ANALYSIS WITH CORRELATED OBSERVATION ERRORS, 301 6.9 .1 Introduction, 301 6.9.2 Autoregressive Errors (AR), 303 6.9.2.1 OLS Estimation with AR Errors, 306 6.9.2.2 M L Estimation with AR Errors, 308 6.9 .3 Moving Average Errors (MA), 312 6.9.4 Summary of First-Order Correlated Cases for the Model7j=,8,313 a nd Physical 6.9 .5 Simultaneous Estimation of p, Parameters for the a l Cases, 315 MODIFICATIONS OF GAUSS METHOD, 362 7.6.1 Box-Kanemasu Interpolation Method, 362 7.6.2 Levenberg Damped Least Squares Method, 368 7.6.3 Marquardt's Method, 370 7.6.4 Comparison of Methods, 371 7.6.4.1 Box-Kanemasu Example, 372 7.6.4.2 Bard Comparisons, 375 7.6.4.3 Davies and Whitting Comparison, 376 7.7 Confidence Regions for Known t/J, 290 Confidence Regions for t/J= 0 2 D with D Known and 2 0 Unknown, 299 EXAMPLES UTILIZING SEQUENTIAL ESTIMATION, 393 7.9.1 Simple M AP Example Involving Multiresponse Data, 394 7.9.2 Cooling Billet Problem, 397 7.9.2.1 Other Possible Models, 398 7.9.3 Semi-Infinite Body Heat Conduction Example, 400 7.9.4 Analysis of Finite Heat-Conducting Body with Multiresponse Experimental Data, 402 7.9.4.1 Description of Equipment, 402 7.9.4.2 Physical Model of Heat-Conducting Body, 406 7.9.4.3 .Parameter Estimates, 407 0:, REFERENCES, 319 APPENDIX 6A AUTOREGRESSIVE MEASUREMENT ERRORS, 320 APPENDIX 6B MATRIX INVERSION LEMMA, 326 PROBLEMS, 327 Chapter 7 Minimization of Sum of Squares Functions for Models Nonlinear in Parameters 7.1 INTRODUCTION, 334 7.1.1 Trial and Error Search, 335 '7.1.2 Exhaustive Search, 336 7. 1.3 Other Methods, 337 7.2 MATRIX FORM OF TAYLOR SERIES EXPANSION, 338 7.3 SUM OF SQUARES FUNCTION, 338 7.4 GAUSS 7.4.1 7.4.2 7.4.3 7.4.4 7.5 EXAMPLES TO ILLUSTRATE GAUSS MINIMIZATION METHOD INVOLVING ORDINARY DIFFERENTIAL EQUATIONS , 350 7.5.1 Estimation of a Parameter for a Long Fin, 350 7.5 .2 Example of Estimation of Parameters in Cooling Billet Problem, 357 334 METHOD OF MINIMIZATION, 340 Derivation, 340 Components of Gauss Linearization Equation, 342 Comments on Gauss Linearization Equation, 346 Linear Dependence of Sensitivity Coefficients, 349 I BUILDING AND CONFIDENCE REGIONS, 378 Approximate Covariance Matrix of Parameters, 378 Approximate Correlation Matrix, 379 Approximate Variance of i ,380 Approximate Confidence Intervals and Regions, 380 Model Building Using the F Test, 386 7. 10 SENSITIVITY COEFFICIENTS, 410 7.10.1 Finite Difference Method, 410 7. 10.2 Sensitivity Equation Method, 411 ... REFERENCES, 414 PROBLEMS, 415 CONTENTS ItUl Chapter 8 CONTENTS 419 8.5.3 8.5.4 Design o ' Optimal Experiments 8.1 0> INTRODUCTION. 419 8.2 ONE PARAMETER EXAMPLES, 420 8.2.1 Linear Examples for One Parameter. 420 8.2.1.1 Model'll - PX1 with No Constraints. 420 8.2.1.2 Model '1 = fJX ( I) for Fixed Large n and Equally Spaced Measurements, 421 8.2.1.3 Model '1 = P X(/) for Fixed Large n and Fixed Maximum Value of I'll , 426 8.2 .2 One-Parameter Nonlinear Cases, ' 1'" tJ( p, I), 428 8.2.3 Iterative Search Method, 431 8.3 8.4 8.5 J ...... , CRITERIA FOR OPTIMAL EXPERIMENTS FOR MULTIPLE PARAMETERS,432 8.3.1 General Criteria. 432 8.3.2 Case of Same Number of Measurements as Parameters ( n=p), 434 8.3.2.1 Linear Examples for p ... 2, 435 8.3.2.2 Nonlinear Example for p -2, 438 ALGEBRAIC EXAMPLES FOR TWO PARAMETERS AND LARGE n, 440 8.4.1 Linear Model '1 = PI + fJ2 sin I. 440 8.4.2 Exponential Models with One Linear and One Nonlinear Parameter, 441 OPTIMAL PARAMETER ESTIMATION INVOLVING THE PARTIAL DIFFERENTIAL EQUATION OF HEAT CONDUCTION. 444 8.5.1 Semi-Infinite Body Examples, 445 8.5 . 1.1 Temperature Boundary Condition (Single Parameter), 445 8.5.1.2 Constant Heat Flux Boundary Condition (Two Parameters), 448 8.5.1.3 Heat Flux Boundary Condition to Cause a Step Change in Surface Temperature, 450 8.5.1.4 Summary of Optimal Designs for Semi-Infinite Bodies Subjected to Heat Flux Boundary Conditions. 451 8.5.2 Finite Body Examples. 453 8.5.2.1 Sinusoidal Initial Temperature in a Plate, 453 8.5.2.2 Constant Heat Flux at x:os 0, Insulated at x = L , 454 Additional Cases, 457 Optimal Heat Conduction Experiment, 458 8.6 NONStANDARD ASSUMPTIONS, 459 8.6.1 Nonconstant Variance, 459 8.6.2 Correlated Errors, 460 8.7 SEQUENTIAL OPTIMIZATION, 460 8.8 NOT ALL PARAMETERS GF INTEREST, 461 8.9 DESIGN CRITERIA FOR MODEL DISCRIMINATION, 464 8.9.1 Linearization Method, 465 8.9.2 Information Theory Method, 467 8.9.2.1 Termination Criteria, 470 .'. REFERENCES, 474 APPENDIX 8A OPTIMAL EXPERIMENT CRITERIA FOR ALL PARAMETERS OF INTEREST, 475 APPENDIX 8a OPTIMAL EXPERIMENT CRITERIA FOR NOT ALL PARAMETERS OF INTEREST, 477 PROBLEMS, 478 Appendix A ldentinablllty Condition 481 Appendix B EstImators and Covarlaaces'OI' Various &tlmatlon M ethods the U near M odel,. =:a XIJ '01' 488 Appendix C U st Symbols 490 Appendix D Some Estimation Programs Index 0' 493 495