Merge branch 'master' of https://github.com/tinkoff-ai/etna into deep…

…_state_model

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -11,21 +11,10 @@ body:
   validations:
     required: true
 
-- type: textarea
-  attributes:
-    label: Motivation
-    description: |
-      Please outline the motivation for the proposal.
-      Is your feature request related to a problem? e.g., I'm always frustrated when [...];
-      Answers the question why it should be done.
-      *If this is related to another GitHub issue, please link here too*
-  validations:
-    required: true
-
 - type: textarea
   attributes:
     label: Proposal
-    description: A clear and concise description of what you want to happen. Ideally step by step.
+    description: A clear and concise description of what you want to happen. Ideally step by step with interface examples.
   validations:
     required: true
 
@@ -36,26 +25,11 @@ body:
   validations:
     required: false
 
-- type: textarea
-  attributes:
-    label: Alternatives
-    description: |
-      A clear and concise description of any alternative solutions or features you've considered.
-  validations:
-    required: false
-
 - type: textarea
   attributes:
     label: Additional context
     description: Add any other context or screenshots about the feature request here.
   validations:
     required: false
-
-- type: checkboxes
-  attributes:
-    label: Checklist
-    options:
-      - label: I discussed this issue with ETNA Team
-        required: false
 
 ...

.github/ISSUE_TEMPLATE/how-to-question.yml → .github/ISSUE_TEMPLATE/other_issue.yml

@@ -1,8 +1,7 @@
 ---
-name: "How to question"
+name: "Other issue"
 description: Asking how-to questions
 labels: help wanted, question
-assignees: iKintosh, julia-shenshina, martins0n
 
 body:
 - type: checkboxes
@@ -22,10 +21,10 @@ body:
 
 - type: textarea
   attributes:
-    label: ❓ Questions and Help
+    label: ❓ What is the issue?
     description: |
       What is your question? What have you tried?
       *Please paste a code snippet if your question requires it!*
   validations:
     required: true
-...
+...

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,5 +1,3 @@
-**IMPORTANT: Please do not create a Pull Request without creating an issue first.**
-
 ## Before submitting (must do checklist)
 
 - [ ] Did you read the [contribution guide](https://github.com/tinkoff-ai/etna/blob/master/CONTRIBUTING.md)?
@@ -8,22 +6,10 @@
 - [ ] Did you update the [CHANGELOG](https://github.com/tinkoff-ai/etna/blob/master/CHANGELOG.md)?
 <!-- For CHANGELOG separate each item in unreleased section by blank line to reduce collisions -->
 
-## Type of Change
-<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->
-
-- [ ] Examples / docs / tutorials / contributors update
-- [ ] Bug fix (non-breaking change which fixes an issue)
-- [ ] Improvement (non-breaking change which improves an existing feature)
-- [ ] New feature (non-breaking change which adds functionality)
-- [ ] Breaking change (fix or feature that would cause existing functionality to change)
-
 ## Proposed Changes
 <!-- Add a more detailed description of the changes if needed. 
 No need for typos and docs improvements  -->
 
-## Related Issue
-<!-- Link Issue here. -->
-
 ## Closing issues
 <!-- Link Issue you are closing here. 
 Put `closes #XXXX` in your comment to auto-close the issue that your PR fixes (if such). -->

CHANGELOG.md

@@ -22,22 +22,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - 
 - 
 ### Changed
+- Add columns and mode parameters in plot_correlation_matrix ([#726](https://github.com/tinkoff-ai/etna/pull/753))
 - 
 - 
 - 
 - 
+- Make LagTransform, LogTransform, AddConstTransform vectorized ([#756](https://github.com/tinkoff-ai/etna/pull/756))
 - 
 - 
 - 
-- 
-- 
+- Make native prediction intervals for DeepAR ([#761](https://github.com/tinkoff-ai/etna/pull/761))
 - 
 - 
 - 
 ### Fixed
 - 
-- 
-- 
+- Fix missing prophet in docker images ([#767](https://github.com/tinkoff-ai/etna/pull/767))
+- Add `known_future` parameter to CLI ([#758](https://github.com/tinkoff-ai/etna/pull/758))
+- FutureWarning: The frame.append method is deprecated. Use pandas.concat instead ([#764](https://github.com/tinkoff-ai/etna/pull/764))
 - 
 - 
 - 

docker/etna-cpu/Dockerfile

@@ -4,7 +4,9 @@ FROM ${BASE_IMAGE}
 RUN apt-get -y update && apt-get -y --no-install-recommends install build-essential git openssh-client && rm -rf /var/lib/apt/lists/*
 
 COPY requirements.txt /
-RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/cpu/torch_stable.html && rm -rf ~/.cache
+RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/cpu/torch_stable.html \
+    && pip install --no-cache-dir prophet \
+    && rm -rf ~/.cache
 WORKDIR /code
 
 CMD [ "bash" ]

docker/etna-cuda-10.2/Dockerfile

@@ -25,7 +25,9 @@ RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
     && rm get-pip.py
 
 COPY requirements.txt /
-RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/${CUDA_VERSION_FOR_TORCH}/torch_stable.html
+RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/${CUDA_VERSION_FOR_TORCH}/torch_stable.html \
+    && pip install --no-cache-dir prophet \
+    && rm -rf ~/.cache
 WORKDIR /code
 
 CMD [ "bash" ]

docker/etna-cuda-11.1/Dockerfile

@@ -25,7 +25,9 @@ RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py \
     && rm get-pip.py
 
 COPY requirements.txt /
-RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/${CUDA_VERSION_FOR_TORCH}/torch_stable.html
+RUN pip install --no-cache-dir -r requirements.txt -f https://download.pytorch.org/whl/${CUDA_VERSION_FOR_TORCH}/torch_stable.html \
+    && pip install --no-cache-dir prophet \
+    && rm -rf ~/.cache
 WORKDIR /code
 
 CMD [ "bash" ]

etna/analysis/eda_utils.py

@@ -743,7 +743,7 @@ def _prepare_seasonal_plot_df(
         elif SeasonalPlotAlignment(alignment) == SeasonalPlotAlignment.last:
             to_add_index = pd.date_range(end=timestamp.min(), periods=num_to_add + 1, closed="left", freq=freq)
 
-        df = df.append(pd.DataFrame(None, index=to_add_index)).sort_index()
+        df = pd.concat((df, pd.DataFrame(None, index=to_add_index))).sort_index()
 
     return df
 

etna/analysis/plotters.py

@@ -343,9 +343,9 @@ def plot_backtest(
 
         # plot history
         if history_len == "all":
-            plot_df = segment_history_df.append(segment_backtest_df)
+            plot_df = pd.concat((segment_history_df, segment_backtest_df))
         elif history_len > 0:
-            plot_df = segment_history_df.tail(history_len).append(segment_backtest_df)
+            plot_df = pd.concat((segment_history_df.tail(history_len), segment_backtest_df))
         else:
             plot_df = segment_backtest_df
         ax[i].plot(plot_df.index, plot_df.target, color=lines_colors["history"])
@@ -633,14 +633,19 @@ def plot_anomalies(
 
 
 def get_correlation_matrix(
-    ts: "TSDataset", segments: Optional[List[str]] = None, method: str = "pearson"
+    ts: "TSDataset",
+    columns: Optional[List[str]] = None,
+    segments: Optional[List[str]] = None,
+    method: str = "pearson",
 ) -> np.ndarray:
     """Compute pairwise correlation of timeseries for selected segments.
 
     Parameters
     ----------
     ts:
         TSDataset with timeseries data
+    columns:
+        Columns to use, if None use all columns
     segments:
         Segments to use
     method:
@@ -659,16 +664,23 @@ def get_correlation_matrix(
     """
     if method not in ["pearson", "kendall", "spearman"]:
         raise ValueError(f"'{method}' is not a valid method of correlation.")
+
     if segments is None:
         segments = sorted(ts.segments)
-    correlation_matrix = ts[:, segments, :].corr(method=method).values
+    if columns is None:
+        columns = list(set(ts.df.columns.get_level_values("feature")))
+
+    correlation_matrix = ts[:, segments, columns].corr(method=method).values
     return correlation_matrix
 
 
 def plot_correlation_matrix(
     ts: "TSDataset",
+    columns: Optional[List[str]] = None,
     segments: Optional[List[str]] = None,
     method: str = "pearson",
+    mode: str = "macro",
+    columns_num: int = 2,
     figsize: Tuple[int, int] = (10, 10),
     **heatmap_kwargs,
 ):
@@ -678,6 +690,8 @@ def plot_correlation_matrix(
     ----------
     ts:
         TSDataset with timeseries data
+    columns:
+        Columns to use, if None use all columns
     segments:
         Segments to use
     method:
@@ -689,23 +703,48 @@ def plot_correlation_matrix(
 
         * spearman: Spearman rank correlation
 
+    mode: 'macro' or 'per-segment'
+        Aggregation mode
+    columns_num:
+        Number of subplots columns
     figsize:
         size of the figure in inches
     """
     if segments is None:
         segments = sorted(ts.segments)
+    if columns is None:
+        columns = list(set(ts.df.columns.get_level_values("feature")))
     if "vmin" not in heatmap_kwargs:
         heatmap_kwargs["vmin"] = -1
     if "vmax" not in heatmap_kwargs:
         heatmap_kwargs["vmax"] = 1
 
-    correlation_matrix = get_correlation_matrix(ts, segments, method)
-    fig, ax = plt.subplots(figsize=figsize)
-    ax = sns.heatmap(correlation_matrix, annot=True, fmt=".1g", square=True, ax=ax, **heatmap_kwargs)
-    labels = list(ts[:, segments, :].columns.values)
-    ax.set_xticklabels(labels, rotation=45, horizontalalignment="right")
-    ax.set_yticklabels(labels, rotation=0, horizontalalignment="right")
-    ax.set_title("Correlation Heatmap")
+    if mode not in ["macro", "per-segment"]:
+        raise ValueError(f"'{mode}' is not a valid method of mode.")
+
+    if mode == "macro":
+        fig, ax = plt.subplots(figsize=figsize)
+        correlation_matrix = get_correlation_matrix(ts, columns, segments, method)
+        labels = list(ts[:, segments, columns].columns.values)
+        ax = sns.heatmap(correlation_matrix, annot=True, fmt=".1g", square=True, ax=ax, **heatmap_kwargs)
+        ax.set_xticks(np.arange(len(labels)) + 0.5, labels=labels)
+        ax.set_yticks(np.arange(len(labels)) + 0.5, labels=labels)
+        plt.setp(ax.get_xticklabels(), rotation=45, ha="right", rotation_mode="anchor")
+        plt.setp(ax.get_yticklabels(), rotation=0, ha="right", rotation_mode="anchor")
+        ax.set_title("Correlation Heatmap")
+
+    if mode == "per-segment":
+        fig, ax = prepare_axes(len(segments), columns_num=columns_num, figsize=figsize)
+
+        for i, segment in enumerate(segments):
+            correlation_matrix = get_correlation_matrix(ts, columns, [segment], method)
+            labels = list(ts[:, segment, columns].columns.values)
+            ax[i] = sns.heatmap(correlation_matrix, annot=True, fmt=".1g", square=True, ax=ax[i], **heatmap_kwargs)
+            ax[i].set_xticks(np.arange(len(labels)) + 0.5, labels=labels)
+            ax[i].set_yticks(np.arange(len(labels)) + 0.5, labels=labels)
+            plt.setp(ax[i].get_xticklabels(), rotation=45, ha="right", rotation_mode="anchor")
+            plt.setp(ax[i].get_yticklabels(), rotation=0, ha="right", rotation_mode="anchor")
+            ax[i].set_title("Correlation Heatmap" + " " + segment)
 
 
 def plot_anomalies_interactive(

etna/commands/backtest_command.py

@@ -1,7 +1,11 @@
 from pathlib import Path
 from typing import Any
 from typing import Dict
+from typing import List
+from typing import Literal
 from typing import Optional
+from typing import Sequence
+from typing import Union
 
 import hydra_slayer
 import pandas as pd
@@ -19,6 +23,12 @@ def backtest(
     freq: str = typer.Argument(..., help="frequency of timestamp in files in pandas format"),
     output_path: Path = typer.Argument(..., help="where to save forecast"),
     exog_path: Optional[Path] = typer.Argument(default=None, help="path to csv with exog data"),
+    known_future: Optional[List[str]] = typer.Argument(
+        None,
+        help="list of all known_future columns (regressor "
+        "columns). If not specified then all exog_columns "
+        "considered known_future.",
+    ),
 ):
     """Command to run backtest with etna without coding.
 
@@ -60,11 +70,13 @@ def backtest(
     df_timeseries = TSDataset.to_dataset(df_timeseries)
 
     df_exog = None
+    k_f: Union[Literal["all"], Sequence[Any]] = ()
     if exog_path:
         df_exog = pd.read_csv(exog_path, parse_dates=["timestamp"])
         df_exog = TSDataset.to_dataset(df_exog)
+        k_f = "all" if not known_future else known_future
 
-    tsdataset = TSDataset(df=df_timeseries, freq=freq, df_exog=df_exog)
+    tsdataset = TSDataset(df=df_timeseries, freq=freq, df_exog=df_exog, known_future=k_f)
 
     pipeline: Pipeline = hydra_slayer.get_from_params(**pipeline_configs)
     backtest_configs_hydra_slayer: Dict[str, Any] = hydra_slayer.get_from_params(**backtest_configs)