Data quality evaluation, outlier detection and missing data imputation methods for IoT in smart cities

Abstract

Low-cost IoT devices allow data collection in smart cities at a high spatio-temporal resolution. Data quality evaluation is needed to investigate the pre-processing steps required to use these data. Besides data pre-processing, outlier detection techniques are required to detect anomalies in the spatio-temporal IoT dataset. We distinguish between erroneous outliers and events based on spatio-temporal autocorrelation patterns, as well as correlations with other dynamic processes in the environment. We consider missing data imputation to fill gaps caused by sensor failures, maintenance, pre-processing and outlier detection. In this study, we use the temporal covariance structure within the data to impute missing data. We apply the methods for outlier detection and missing data imputation to an IoT testbed for air quality monitoring in the city of Eindhoven, the Netherlands. The methods can be applied in a more general sense to other continuous environmental variables which show a similarly strong spatio-temporal autocorrelation structure.