bioRxiv2018
Successfully predicting the future states of systems that are complex, stochastic and potentially chaotic is a major challenge. Model forecasting error (FE) is the usual measure of success; however model predictions provide no insights into the potential for improvement. In short, the realized predictability of a specific model is uninformative about whether the system is inherently predictable or whether the chosen model is a poor match for the system and our observations thereof. Ideally, model proficiency would be judged with respect to the systems intrinsic predictability - the highest achievable predictability given the degree to which system dynamics are the result of deterministic v. stochastic processes. Intrinsic predictability may be quantified with permutation entropy (PE), a model-free, information-theoretic measure of the complexity of a time series. By means of simulations we show that a correlation exists between estimated PE and FE and show how stochasticity, process error, and chaotic dynamics affect the relationship. This relationship is verified for a dataset of 461 empirical ecological time series. We show how deviations from the expected PE-FE relationship are related to covariates of data quality and the nonlinearity of ecological dynamics.\n\nThese results demonstrate a theoretically-grounded basis for a model-free evaluation of a systems intrinsic predictability. Identifying the gap between the intrinsic and realized predictability of time series will enable researchers to understand whether forecasting proficiency is limited by the quality and quantity of their data or the ability of the chosen forecasting model to explain the data. Intrinsic predictability also provides a model-free baseline of forecasting proficiency against which modeling efforts can be evaluated.\n\nGlossaryActive information: The amount of information that is available to forecasting models (redundant information minus lost information; Fig. 1).\n\nO_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=45 SRC=\"FIGDIR/small/350017_fig1a.gif\" ALT=\"Figure 1A\">\nView larger version (9K):\norg.highwire.dtl.DTLVardef@934d9eorg.highwire.dtl.DTLVardef@ccdc10org.highwire.dtl.DTLVardef@1839ed9org.highwire.dtl.DTLVardef@31bd70_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOFigure 1A.C_FLOATNO The total information content of an observation of a system at a given state in time, St, is depicted by filled circles with past states (St-1 and St-2) represented by shades of grey, i) lack of overlap between past and present states illustrating a case where no information is transmitted from past states (i.e. a purely stochastic system), with low redundancy and high Shannon entropy rate, ii) intermediate overlap indicating a case when some information is transferred from past to present (i.e. a deterministic system strongly driven by stochastic forcing), with intermediate redundancy and Shannon entropy rate, iii) large overlap indicating a case when the current state is mostly determined by the previous state (i.e. a highly deterministic system), with high redundancy and low Shannon entropy rate. Note that both the redundancy and Shannon entropy rate of a system are intrinsic properties of the system and will only change if the system itself changes.\n\nC_FIG Forecasting error (FE): A measure of the discrepancy between a models forecasts and the observed dynamics of a system. Common measures of forecast error are root mean squared error and mean absolute error.\n\nEntropy: Measures the average amount of information in the outcome of a stochastic process.\n\nInformation: Any entity that provides answers and resolves uncertainty about a process. When information is calculated using logarithms to the base two (i.e. information in bits), it is the minimum number of yes/no questions required, on average, to determine the identity of the symbol (Jost 2006). The information in an observation consists of information inherited from the past (redundant information), and of new information.\n\nIntrinsic predictability: the maximum achievable predictability of a system (Beckage et al. 2011).\n\nLost information: The part of the redundant information lost due to measurement or sampling error, or transformations of the data (Fig. 1).\n\nNew information, Shannon entropy rate: The Shannon entropy rate quantifies the average amount of information per observation in a time series that is unrelated to the past, i.e., the new information (Fig. 1).\n\nNonlinearity: When the deterministic processes governing system dynamics depend on the state of the system.\n\nPermutation entropy (PE): permutation entropy is a measure of the complexity of a time series (Bandt & Pompe, 2002) that is negatively correlated with a systems predictability (Garland et al. 2015). Permutation entropy quantifies the combined new and lost information. PE is scaled to range between a minimum of 0 and a maximum of 1.\n\nRealized predictability: the achieved predictability of a system from a given forecasting model.\n\nRedundant information: The information inherited from the past, and thus the maximum amount of information available for use in forecasting (Fig. 1).\n\nSymbols, words, permutations: symbols are simply the smallest unit in a formal language such as the letters in the English alphabet i.e., {\"A\", \"B\",..., \"Z\"}. In information theory the alphabet is more abstract, such as elements in the set {\"up\", \"down\"} or {\"1\", \"2\", \"3\"}. Words, of length m refer to concatenations of the symbols (e.g., up-down-down) in a set. Permutations are the possible orderings of symbols in a set. In this manuscript, the words are the permutations that arise from the numerical ordering of m data points in a time series.\n\nWeighted permutation entropy (WPE): a modification of permutation entropy (Fadlallah et al., 2013) that distinguishes between small-scale, noise-driven variation and large-scale, system-driven variation by considering the magnitudes of changes in addition to the rank-order patterns of PE.