References to real- and virtual columns are stored in the table_proxy

[MarcusKlik]

A table proxy has a specific state. That state reflects the current table that it represents. But no actual data is read until necessary, so the state is a collection of row- and column transformations that provide a view on the underlying physical fst file.

For example: when the user does a row selection:

ft2 <- ft[Year > 2000]

a new fsttable object is created (ft2). That object contains a filter that reflects the result of Year > 2000 in binary format, no other data from the table is stored. The filter is computed in chunks so it has no significant memory requirements except that of creating the filter (number of bytes equal to the length of the vector divided by 8). The filter could be stored on disk.

When the user does:

ft3 <- ft2[, .(ColA, ColB = ColD * 100)]

again a new fsttable is created (ft3). That object holds the original filter and a column selection (ColA). It also holds a virtual table selection (ColB), No actual data is computed at this point. Because fsttable knows that * operates per-element, it is enough to hold the expression, the primitive value (100) and the column reference (ColD). With that information, the column can be computed at a later time.

: the selected columns in the fsttable
virtual columns: for constructed columns that have single element operators these are expressions that define the new column (no actual data stored).