4.3 New data abstraction and operations for missing data in time

Figure 4.1 is a typical time series plot, plotting the present data and leaving gaps between to indicate what is not available. The result is that missing values receive little attention, due to a lack of visual emphasis. A need to better represent and display missing data in time is exposed. To begin this process, it is convenient to first address appropriate computer representation. In R, missing values are encoded as NA. However, the notion of an ordered NA does not exist. A new abstraction for ordered NA provides the scope for conveying the temporal locations and dependencies in missing data.

4.3.1 New encoding for indexing missing data by time

Inspired by run-length encoding (RLE), a new encoding is proposed to solely extract the NAs from time-indexed data and compress them in a simpler form, namely “RLE <NA>”. It comprises three components to locate the missings and mark their corresponding runs: (1) positions where NA starts, (2) run lengths (NA in a row), and (3) interval (for example, hourly or yearly intervals). This implies that time indices should be unique.

This new encoding purely focuses on indexed missing values, separated from its data input. It is partially lossless, because its reverse operation can recover the original positions of missing values, but not the whole data. It is most useful and compact on indexing runs of missing data, requiring less storage than its original lengthy form. However, when missings mostly involve runs of length one, it is not that advantageous. Considering the missingness types of Missing at Occasions and Missing at Runs in Figure 4.1, the former occupies 14 positions to store NAs; while the latter uses 7 positions for storing more NAs than the former as a sparser representation. The RLE <NA> is easy to interpret: a sequence of 12 NAs beginning at 1949 March, followed by 13 NAs since 1950 August, and so on, for the latter.

#> <list_of<Run Length Encoding <NA>>[2]>
#> [[1]]
#> <Run Length Encoding <NA>[13]>
#> $lengths: <int> 1 1 1 1 1 1 1 1 1 1 ...
#> $indices: <date> 1951 Apr 1952 May 1953 May 1954 Apr 1955 Feb 1955
Nov 1956 Jul 1957 Feb 1957 Aug 1958 Jan ...
#> [[2]]
#> <Run Length Encoding <NA>[7]>
#> $lengths: <int> 12 13 5 3 4 8 4
#> $indices: <date> 1949 Mar 1950 Aug 1951 Dec 1953 Jan 1955 Jun 1956
Apr 1958 Dec

The instance of RLE <NA> is a reduced form for representing NA in time, built on top of the new vctrs framework (H. Wickham, Henry, and Vaughan 2019).

4.3.2 Supporting functions operating on RLE <NA>

The RLE <NA> prioritizes indexed missing data as the raw data itself, that provides the opportunities to manipulate the missings with many useful operations.

It is computationally efficient to sum (sum()) and count (length()) the run lengths over a standalone RLE <NA>, than directly dealing with its original form for identical results. Other mathematical functions, such as mean() and median(), make it accessible to compute runs-related statistics. For example, mean() gives the average of missings per run. If not going on the route of RLE, it would be cumbersome to compute these statistics otherwise.

These math operations primarily require a singular RLE <NA> at a time. The other set is the set operators that performs set union (union()), intersection (intersect()), and asymmetric difference (setdiff()) on a pair of RLE <NA>. They are useful for exploring the association between multiple sets of missing data. For example, the intersect() operator could tell if they overlap with each other and by how much, which powers one of the plots in the next section. Since set operators are binary functions, a collection of series can be successively combined and applied to give an overall picture about all.