Code associated with the work "Discerning the painter’s hand: machine learning on surface topography".
Attribution of painted works is a critical problem in art history. In painted works, surface topography adds important unintentional stylistic components such as the deposition and drying of the paint, in addition to the intentional brushwork of the artist. Current art historical scholarship includes a growing emphasis on workshop practice used in the creation of paintings by renowned artists. Many famous artists, including El Greco, Rembrandt, and Peter Paul Rubens, utilized workshops to meet market demands for their art. In this study, we hypothesize that significant unintentional stylistic information exists in the 3D surface structure embedded by the painter during the painting process, and that this can be captured through optical profilometry. In addition, by investigating patches at much smaller length scales than any recognizable feature of the painting, differences in the artist’s intrinsic and unintentional style will be revealed. We examine these hypotheses using a controlled study of bespoke paintings by several artists that mimic certain workshop practices.
In our experiment, a series of twelve paintings by four artists, and their associated topographical profiles, is subject to analysis to attribute the works and to ascertain the important properties involved in that attribution. To analyze these profiles, we use convolutional neural networks (CNNs) to provide a robust and quantitative method to distinguish among random small areas of the triplicate paintings by multiple artists. Bi-directional multivariate empirical mode decomposition (EMD) is applied to the height data collected by profilometry to separate the data into complementary intrinsic mode functions (IMFs) of varying spatial frequency. Finally, we compare ML on photographic and topographic data.
Each artist (artist 1-4) created three painting, one of which was reserved for testing the training CNN algorithm. These test paintings are presented for all four artist with first row showing high-resolution photographs (all paintings used in the study can be found in SI in our paper), and second row showing height data shaded in grayscale from low (darker) to high (lighter).
In preparation for the experiments, the height information is digitally split into small patches with a range of patch sizes from 10 pixels (0.5 mm) to 1200 pixels (6 cm). We train those patches using the pretrained VGG-16 network. The following figure shows sample patches of height data from each artist, each patch is 200x200 pixels (10x10 mm).
By processing the digital surface reconstructions using the EMD we extract intrinsic spatial scales of increasing size. The first IMF contains the most granular of textures, and subsequent IMFs contain larger and larger features until the sifting procedure is halted and a residual is all that remains. The following figure shows a sample patch of side-length 80 pixels (4mm) and the corresponding first five IMFs calculated using EMD.
Image recognition by ML is most often performed on photographic images of the subject depicted by arrays of RGB channels performed on the entire image. We sought to determine how well the same ML algorithm on patches of the high-resolution photographs would perform compared to the profilometry data. To generate qualitatively distinct training and testing sets, we divided each painting into patches of side-length 100 pixels (5 mm) and then sorted the patches into three categories: background, foreground, and border depending on the color composition of each patch. We excluded all the border patches to make the analysis more stringent. We then trained and validated on foreground patches, tested on background patches, or vice versa, using both height data and high resolution photo data.
Data are available at https://osf.io/chj62/files/
height_ensemble.ipynb: the environment where we use VGG-16 to quantitatively attribute the individual patches of the painting using height data.
ensemble_analysis.ipynb: an example script showing how we calculate ensemble accuracy, f1-score, precision, and recall.
emd_individual_imfs.ipynb: the environment where we investigate how the individual IMFs affect the attribution results.
foreground_background_height.ipynb: the environment where we train and validate on foreground patches, test on background patches, or vice versa, using height data.
mask_info_with_borders.csv: a file that divides a whole painting into background/foreground/border patches, with a 1 indicating a background patch, a 0 indicating a foreground patch, and a 0.5 indicating a border (not to be used for training or testing). The patch size if 100x100 pixels, the patches are ordered left-to-right, top-to-bottom, starting from the top left (patch 1) and ending on the bottom right (patch 720).