-
Notifications
You must be signed in to change notification settings - Fork 44
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[HOPSWORKS-2280] Save statistics configuration also for training data…
…sets (#217)
- Loading branch information
1 parent
9302c27
commit 4c48a3b
Showing
22 changed files
with
344 additions
and
127 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
# StatisticsConfig | ||
|
||
{{statistics_config}} | ||
|
||
## Properties | ||
|
||
{{statistics_config_properties}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
# Statistics | ||
|
||
HSFS provides functionality to compute statistics for [training datasets](training_dataset.md) and [feature groups](feature_group.md) and save these along with their other metadata in the [feature store](feature_store.md). | ||
These statistics are meant to be helpful for Data Scientists to perform explorative data analysis and then recognize suitable [features](feature.md) or [training datasets](training_dataset.md) for models. | ||
|
||
Statistics are configured on a training dataset or feature group level using a `StatisticsConfig` object. | ||
This object can be passed at creation time of the dataset or group or it can later on be updated through the API. | ||
|
||
{{statistics_config}} | ||
|
||
For example, to enable all statistics (descriptive, histograms and correlations) for a training dataset: | ||
|
||
=== "Python" | ||
```python | ||
from hsfs.statistics_config import StatisticsConfig | ||
|
||
td = fs.create_training_dataset("rain_dataset", | ||
version=1, | ||
label=”weekly_rain”, | ||
data_format=”tfrecords”, | ||
statistics_config=StatisticsConfig(true, true, true)) | ||
|
||
``` | ||
=== "Scala" | ||
```scala | ||
val td = (fs.createTrainingDataset() | ||
.name("rain_dataset") | ||
.version(1) | ||
.label(”weekly_rain”) | ||
.dataFormat(”tfrecords”) | ||
.statisticsConfig(new StatisticsConfig(true, true, true)) | ||
.build()) | ||
``` | ||
|
||
And similarly for feature groups. | ||
|
||
!!! note "Default StatisticsConfig" | ||
By default all training datasets and feature groups will be configured such that only descriptive statistics | ||
are computed. However, you can also enable `histograms` and `correlations` or limit the features for which | ||
statistics are computed. | ||
|
||
## Properties | ||
|
||
{{statistics_config_properties}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
53 changes: 53 additions & 0 deletions
53
java/src/main/java/com/logicalclocks/hsfs/StatisticsConfig.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
/* | ||
* Copyright (c) 2021 Logical Clocks AB | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* | ||
* See the License for the specific language governing permissions and limitations under the License. | ||
*/ | ||
|
||
package com.logicalclocks.hsfs; | ||
|
||
import lombok.AllArgsConstructor; | ||
import lombok.Builder; | ||
import lombok.Getter; | ||
import lombok.NoArgsConstructor; | ||
import lombok.Setter; | ||
|
||
import java.util.ArrayList; | ||
import java.util.List; | ||
|
||
@AllArgsConstructor | ||
@NoArgsConstructor | ||
@Builder | ||
public class StatisticsConfig { | ||
@Getter | ||
@Setter | ||
private Boolean enabled = true; | ||
|
||
@Getter | ||
@Setter | ||
private Boolean histograms = false; | ||
|
||
@Getter | ||
@Setter | ||
private Boolean correlations = false; | ||
|
||
@Getter | ||
@Setter | ||
private List<String> columns = new ArrayList<>(); | ||
|
||
public StatisticsConfig(Boolean enabled, Boolean histograms, Boolean correlations) { | ||
this.enabled = enabled; | ||
this.histograms = histograms; | ||
this.correlations = correlations; | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.