11.3 Error and accuracy measures

Geospatial data accuracy is crucial for reliable analyses and decision-making. Understanding error types, sources, and measures helps engineers identify, quantify, and mitigate issues in their work. This knowledge is essential for producing high-quality geospatial information.

Minimizing errors through calibration, redundancy, and quality control is key to improving data reliability. Proper error reporting and accuracy assessment ensure transparency and help users make informed decisions based on geospatial information's limitations and uncertainties.

Types of errors

• Errors in geospatial data and measurements can significantly impact the accuracy and reliability of geospatial analyses and decision-making
• Understanding the different types of errors is crucial for identifying, quantifying, and mitigating their effects in geospatial engineering applications

Systematic vs random errors

Top images from around the web for Systematic vs random errors

• 

  ESD - Calibrating large-ensemble European climate projections using observational data View original

  Is this image relevant?

• 

  GMD - Cutting out the middleman: calibrating and validating a dynamic vegetation model (ED2 ... View original

  Is this image relevant?

• 

  SOIL - Improved calibration of the Green–Ampt infiltration module in the EROSION-2D/3D model ... View original

  Is this image relevant?

• 

  ESD - Calibrating large-ensemble European climate projections using observational data View original

  Is this image relevant?

• 

  GMD - Cutting out the middleman: calibrating and validating a dynamic vegetation model (ED2 ... View original

  Is this image relevant?

1 of 3

Top images from around the web for Systematic vs random errors

• 

  ESD - Calibrating large-ensemble European climate projections using observational data View original

  Is this image relevant?

• 

  GMD - Cutting out the middleman: calibrating and validating a dynamic vegetation model (ED2 ... View original

  Is this image relevant?

• 

  SOIL - Improved calibration of the Green–Ampt infiltration module in the EROSION-2D/3D model ... View original

  Is this image relevant?

• 

  ESD - Calibrating large-ensemble European climate projections using observational data View original

  Is this image relevant?

• 

  GMD - Cutting out the middleman: calibrating and validating a dynamic vegetation model (ED2 ... View original

  Is this image relevant?

1 of 3

• Systematic errors exhibit a consistent pattern or bias in measurements, often due to factors such as instrumental drift, improper calibration, or methodological flaws
  • These errors can be difficult to detect and correct, as they may not be apparent from individual measurements
  • Examples include a consistently misaligned sensor or a software bug that introduces a constant offset in calculations
• Random errors are unpredictable fluctuations in measurements, caused by factors such as environmental noise, human inconsistencies, or inherent variability in the measured phenomena
  • These errors tend to follow a normal distribution and can be reduced by averaging multiple measurements or applying statistical techniques
  • Examples include variations in GPS positions due to atmospheric conditions or inconsistencies in manual digitization of features

Gross vs minor errors

• Gross errors, also known as blunders or outliers, are substantial deviations from the true value, often resulting from human mistakes, equipment malfunctions, or data corruption
  • These errors can significantly skew analyses and should be identified and removed before further processing
  • Examples include mistyping coordinates, using the wrong units, or incorrectly labeling features
• Minor errors are small deviations from the true value, typically caused by the inherent limitations of measuring devices or the precision of data storage and processing
  • These errors are often unavoidable but can be minimized through proper equipment selection, calibration, and data handling procedures
  • Examples include rounding errors in calculations or the limited resolution of digital elevation models

Absolute vs relative errors

• Absolute errors represent the difference between a measured value and the true value, expressed in the same units as the measurement
  • These errors provide a direct indication of the accuracy of individual measurements but may not be suitable for comparing errors across different scales or units
  • An example is a GPS position that deviates from the true location by 5 meters
• Relative errors express the absolute error as a percentage or fraction of the true value, allowing for a standardized comparison of errors across different scales or units
  • These errors are useful for assessing the significance of errors in relation to the magnitude of the measured values
  • An example is a 2% error in a distance measurement, which may be more or less significant depending on the total distance being measured

Accuracy measures

• Accuracy measures quantify the closeness of measurements or estimates to the true values, providing a standardized way to assess and compare the quality of geospatial data and analyses
• Different accuracy measures are used depending on the type of data, the application requirements, and the available reference information

Root mean square error (RMSE)

• RMSE is a widely used accuracy measure that quantifies the average magnitude of errors in a dataset, expressed in the same units as the measurements
  • It is calculated as the square root of the mean of the squared differences between the measured and true values: $RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (x_i - \hat{x}_i)^2}$
  • RMSE is sensitive to outliers and provides a good overall indication of the spread of errors in the dataset
• RMSE is commonly used to assess the accuracy of continuous variables, such as elevation values in a digital elevation model or predicted values in a spatial interpolation

Mean absolute error (MAE)

• MAE is another common accuracy measure that quantifies the average magnitude of errors in a dataset, expressed in the same units as the measurements
  • It is calculated as the mean of the absolute differences between the measured and true values: $MAE = \frac{1}{n} \sum_{i=1}^{n} |x_i - \hat{x}_i|$
  • MAE is less sensitive to outliers than RMSE and provides a more intuitive interpretation of the average error magnitude
• MAE is often used to assess the accuracy of continuous variables, particularly when the error distribution is not expected to be normal or when outliers are less of a concern

Horizontal accuracy

• Horizontal accuracy refers to the closeness of planimetric coordinates (e.g., latitude and longitude) to their true values, typically expressed as a distance or a percentage of the map scale
  • It is assessed by comparing the measured coordinates of well-defined points to their known reference coordinates, such as those obtained from high-accuracy surveys or geodetic control points
  • Horizontal accuracy is crucial for applications that rely on the correct positioning of features, such as navigation, land surveying, or infrastructure mapping
• Examples of horizontal accuracy measures include the circular error probable (CEP), which represents the radius of a circle that contains 50% of the measured points, and the root mean square error (RMSE) of the coordinate differences

Vertical accuracy

• Vertical accuracy refers to the closeness of elevation or depth values to their true values, typically expressed as a distance or a percentage of the elevation range
  • It is assessed by comparing the measured elevations of well-defined points to their known reference elevations, such as those obtained from high-accuracy surveys or benchmark data
  • Vertical accuracy is crucial for applications that rely on the correct representation of terrain or surface features, such as flood modeling, volume calculations, or line-of-sight analyses
• Examples of vertical accuracy measures include the linear error at 90% confidence (LE90), which represents the vertical distance that contains 90% of the measured elevations, and the root mean square error (RMSE) of the elevation differences

Positional accuracy

• Positional accuracy is a combined measure of both horizontal and vertical accuracy, representing the overall closeness of a measured point to its true position in three-dimensional space
  • It is assessed by comparing the measured coordinates and elevations of well-defined points to their known reference values, often using a three-dimensional root mean square error (RMSE) or a spherical accuracy standard
  • Positional accuracy is important for applications that require the correct positioning and representation of features in 3D, such as 3D city modeling, subsurface mapping, or augmented reality
• Examples of positional accuracy measures include the spherical accuracy standard (SAS), which represents the radius of a sphere that contains a specified percentage of the measured points, and the root mean square error (RMSE) of the 3D coordinate differences

Precision measures

• Precision measures quantify the consistency or repeatability of measurements, indicating the degree to which repeated measurements of the same quantity yield similar results
• Precision is related to the random errors in a dataset and is often used to assess the quality of measuring devices, data collection methods, or analysis techniques

Standard deviation

• Standard deviation is a widely used precision measure that quantifies the dispersion of a set of measurements around their mean value, expressed in the same units as the measurements
  • It is calculated as the square root of the variance, which is the average of the squared deviations from the mean: $\sigma = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}$
  • A smaller standard deviation indicates higher precision, as the measurements are more tightly clustered around the mean
• Standard deviation is commonly used to assess the precision of repeated measurements of the same quantity, such as the coordinates of a survey point measured multiple times or the elevation values of a specific location obtained from different sources

Coefficient of variation

• Coefficient of variation (CV) is a standardized precision measure that expresses the standard deviation as a percentage of the mean value, allowing for the comparison of precision across different scales or units
  • It is calculated as the ratio of the standard deviation to the mean, multiplied by 100: $CV = \frac{\sigma}{\bar{x}} \times 100\%$
  • A smaller coefficient of variation indicates higher precision, as the standard deviation represents a smaller proportion of the mean value
• Coefficient of variation is useful for comparing the precision of measurements with different units or magnitudes, such as the precision of distance measurements obtained from different surveying techniques or the precision of area estimates derived from different remote sensing datasets

Confidence intervals

• Confidence intervals are a precision measure that provides a range of values within which the true value of a quantity is likely to fall, given a specified level of confidence (e.g., 95%)
  • They are calculated using the sample mean, the standard deviation, and the desired confidence level, assuming a normal distribution of the measurements: $CI = \bar{x} \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}}$
  • Narrower confidence intervals indicate higher precision, as there is a smaller range of values that are likely to contain the true value
• Confidence intervals are commonly used to report the precision of estimated quantities, such as the mean elevation of a region derived from a sample of measurements or the average positional accuracy of a GPS device based on a series of tests

Propagation of errors

• Error propagation refers to the accumulation and transformation of errors as they are carried through a series of calculations or operations, potentially leading to a significant impact on the final results
• Understanding error propagation is essential for estimating the uncertainty of derived quantities, assessing the sensitivity of analyses to input errors, and designing robust geospatial workflows

Error accumulation in calculations

• Errors in input data can accumulate and propagate through a series of calculations, leading to increasingly large errors in the output results
  • The magnitude and direction of error accumulation depend on the nature of the calculations and the correlations between the input errors
  • Examples include the accumulation of positional errors when combining multiple datasets, the propagation of elevation errors in terrain analysis, or the compounding of classification errors in multi-stage image processing
• To assess error accumulation, one can use analytical error propagation techniques, such as the Taylor series method or the Monte Carlo simulation, which estimate the uncertainty of the output based on the uncertainties of the inputs and the functional relationships between them

Error estimation techniques

• Error estimation techniques are used to quantify the uncertainty of derived quantities or the sensitivity of analyses to input errors, providing a measure of the reliability and robustness of the results
  • These techniques often involve the propagation of input uncertainties through the calculations using statistical or numerical methods
  • Examples include the use of variance-covariance matrices to estimate the uncertainty of coordinate transformations, the application of sensitivity analysis to assess the impact of DEM errors on hydrological modeling, or the estimation of confidence intervals for spatial interpolation results
• Common error estimation techniques include the first-order Taylor series approximation, which linearizes the functional relationships and propagates the input variances, and the Monte Carlo simulation, which repeatedly samples from the input error distributions and analyzes the distribution of the output results

Sensitivity analysis

• Sensitivity analysis is a technique used to assess the impact of input errors or variations on the output results, identifying the most influential factors and the robustness of the analyses to uncertainties
  • It involves systematically varying the input parameters within their plausible ranges and observing the corresponding changes in the output values
  • Sensitivity analysis can help prioritize the sources of errors that need to be addressed, optimize data collection and processing strategies, and communicate the reliability of the results to stakeholders
• Examples of sensitivity analysis include assessing the impact of GPS positional errors on the accuracy of field boundary delineation, evaluating the sensitivity of flood inundation models to variations in terrain data resolution, or determining the robustness of land cover classifications to changes in training sample selection

Sources of errors

• Errors in geospatial data and analyses can originate from various sources, including the limitations of measuring devices, the imperfections of data collection and processing methods, and the inherent variability of the natural and built environment
• Identifying and understanding the sources of errors is crucial for developing strategies to minimize their impact and improve the quality of geospatial products and services

Instrumental errors

• Instrumental errors are caused by the limitations, malfunctions, or improper use of the measuring devices or sensors used to collect geospatial data
  • These errors can be systematic, such as a consistent bias due to poor calibration or sensor drift, or random, such as noise due to electronic fluctuations or environmental interference
  • Examples include the positional errors of GPS receivers due to clock drift or multipath effects, the radiometric errors of remote sensing instruments due to sensor degradation or atmospheric attenuation, or the angular errors of total stations due to collimation or leveling issues
• To minimize instrumental errors, it is important to regularly calibrate and maintain the equipment, follow proper data collection protocols, and use robust data processing techniques that can detect and correct for systematic biases or outliers

Human errors

• Human errors are caused by the mistakes, inconsistencies, or subjective judgments of the individuals involved in the data collection, processing, or analysis stages
  • These errors can be unintentional, such as misreading a measurement or mistyping a value, or intentional, such as selectively omitting or manipulating data to achieve a desired outcome
  • Examples include the mislabeling of features during field surveys, the inconsistent digitization of boundaries from aerial photographs, or the biased interpretation of analysis results based on preconceived notions or vested interests
• To minimize human errors, it is important to provide adequate training and supervision to the personnel, establish clear and standardized operating procedures, implement quality control and cross-checking mechanisms, and foster a culture of transparency and accountability

Environmental factors

• Environmental factors are the external conditions or phenomena that can introduce errors or uncertainties in geospatial data and analyses, often by affecting the performance of measuring devices or the properties of the measured objects
  • These factors can be natural, such as weather conditions, topographic variations, or vegetation cover, or anthropogenic, such as urban development, land use changes, or electromagnetic interference
  • Examples include the impact of cloud cover or atmospheric haze on the quality of satellite imagery, the effect of terrain complexity on the accuracy of digital elevation models, or the influence of building materials and structures on the propagation of GPS signals in urban environments
• To minimize the impact of environmental factors, it is important to carefully plan and execute data collection campaigns, taking into account the specific characteristics and limitations of the study area, use ancillary data or models to correct for known environmental effects, and clearly communicate the assumptions and uncertainties associated with the data and analyses

Minimizing errors

• Minimizing errors is a critical aspect of ensuring the quality, reliability, and usability of geospatial data and analyses, as it helps to reduce the uncertainties and improve the decision-making processes that rely on this information
• There are various strategies and techniques that can be employed to minimize errors at different stages of the geospatial data lifecycle, from data collection and processing to analysis and visualization

Calibration techniques

• Calibration is the process of comparing a measuring device or sensor against a known standard and adjusting its parameters to ensure that it produces accurate and consistent measurements
  • Regular calibration helps to detect and correct for systematic errors, such as biases or drifts, that can accumulate over time and affect the quality of the collected data
  • Examples include the calibration of GPS receivers using known control points to correct for clock errors and atmospheric delays, the radiometric calibration of remote sensing instruments using reference targets to correct for sensor degradation and atmospheric effects, or the geometric calibration of cameras using calibration patterns to correct for lens distortions and misalignments
• Effective calibration requires the use of reliable and traceable standards, the adherence to established calibration protocols and frequencies, and the proper documentation and communication of the calibration results and uncertainties

Redundancy and cross-checking

• Redundancy and cross-checking are techniques that involve the collection or processing of multiple independent measurements or estimates of the same quantity, allowing for the detection and correction of errors or inconsistencies
  • Redundancy can be achieved by using multiple measuring devices or sensors, collecting data at different times or locations, or employing different data processing or analysis methods
  • Cross-checking involves the comparison of the redundant measurements or estimates, either manually or automatically, to identify and resolve discrepancies or outliers
• Examples include the use of multiple GPS receivers to improve the positional accuracy and reliability of surveys, the comparison of satellite imagery from different sensors or platforms to detect and correct for atmospheric or geometric errors, or the verification of land cover classifications using field observations or high-resolution aerial photographs

Quality control procedures

• Quality control (QC) procedures are systematic processes that are implemented to assess, monitor, and maintain the quality of geospatial data and analyses throughout their lifecycle
  • QC procedures can be applied at various stages, such as data collection, processing, analysis, and delivery, and can involve both automated checks and manual inspections
  • Examples include the use of data validation scripts to check for format consistency, completeness, and plausibility of collected data, the implementation of topology rules to ensure the logical consistency and integrity of spatial data, or the visual inspection of map products to detect and correct for cartographic errors or anomalies
• Effective QC procedures require the establishment of clear quality standards and specifications, the use of appropriate tools and techniques for quality assessment and control, and the continuous monitoring and improvement of the quality management system based on feedback and performance metrics

Reporting errors and accuracy

• Reporting errors and accuracy is an essential aspect of ensuring the transparency, reproducibility, and usability of geospatial data and analyses, as it helps users to understand the limitations and uncertainties associated with the information and to make informed decisions based on it
• There are various standards, guidelines, and best practices that provide recommendations on how to report errors and accuracy in geospatial data and metadata, ensuring consistency and interoperability across different sources and applications

Significant figures

• Significant figures are the number of digits in a measurement or estimate that are considered reliable and meaningful, based on the precision and accuracy of the data collection and processing methods
  • Reporting the appropriate number of significant figures helps to avoid overstating the precision or accuracy of the data and to communicate the level of uncertainty associated with the values
  • Examples include reporting GPS coordinates with a sufficient number of decimal places to reflect the positional accuracy of the measurements, or reporting elevation values with a number of significant figures consistent with the vertical resolution and accuracy of the digital elevation model
• To