Support PyTorch Dataset for oscar (mars-project#2246)

README.rst

@@ -141,7 +141,7 @@ Mars learn provides a familiar interface like scikit-learn.
 Mars learn also integrates with many libraries:
 
 - `TensorFlow <https://docs.pymars.org/en/latest/user_guide/learn/tensorflow.html>`_
-- PyTorch
+- `PyTorch <https://docs.pymars.org/en/latest/user_guide/learn/pytorch.html>`_
 - `XGBoost <https://docs.pymars.org/en/latest/user_guide/learn/xgboost.html>`_
 - `LightGBM <https://docs.pymars.org/en/latest/user_guide/learn/lightgbm.html>`_
 - `Joblib <https://docs.pymars.org/en/latest/user_guide/learn/joblib.html>`_

docs/source/getting_started/learn.rst

@@ -33,9 +33,9 @@ For implemented learn API, refer to :ref:`learn API reference <learn_api>`.
 
 Mars learn can integrate with XGBoost, LightGBM, TensorFlow and PyTorch.
 
-- For XGBoost, refer to :ref:`integrate_xgboost`.
-- For LightGBM, refer to :ref:`integrate_lightgbm`.
-- For TensorFlow, refer to :ref:`integrate_tensorflow`.
-- For PyTorch, doc is coming soon.
-- For Joblib, refer to :ref:`integrate_joblib`.
-- For Statsmodels, refer to :ref:`integrate_statsmodels`.
+- :ref:`XGBoost <integrate_xgboost>`.
+- :ref:`LightGBM <integrate_lightgbm>`.
+- :ref:`TensorFlow <integrate_tensorflow>`.
+- :ref:`PyTorch <integrate_pytorch>`.
+- :ref:`Joblib <integrate_joblib>`.
+- :ref:`Statesmodels <integrate_statsmodels>`.

docs/source/index.rst

@@ -114,8 +114,14 @@ Mars learn provides a familiar interface like scikit-learn.
 |                                             |                                                    |
 +---------------------------------------------+----------------------------------------------------+
 
-Mars learn has also integrated many libraries, including :ref:`tensorflow`,
-:ref:`xgboost`, :ref:`lightgbm`, :ref:`joblib` and :ref:`statsmodels`.
+Mars learn also integrates with many libraries:
+
+- :ref:`TensorFlow <integrate_tensorflow>`
+- :ref:`PyTorch <integrate_pytorch>`
+- :ref:`XGBoost <integrate_xgboost>`
+- :ref:`LightGBM <integrate_lightgbm>`
+- :ref:`Joblib <integrate_joblib>`
+- :ref:`Statsmodels <integrate_statsmodels>`
 
 Mars remote
 -----------

docs/source/locale/zh_CN/LC_MESSAGES/getting_started/learn.po

@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: mars 0.5.0a2\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2021-08-03 18:42+0800\n"
+"POT-Creation-Date: 2021-08-20 11:06+0800\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -53,18 +53,25 @@ msgid "Mars learn can integrate with XGBoost, LightGBM, TensorFlow and PyTorch."
 msgstr "Mars learn 能和 XGBoost、LightGBM、TensorFlow 以及 PyTorch 集成。"
 
 #: ../../source/getting_started/learn.rst:36
-msgid "For XGBoost, refer to :ref:`xgboost`."
-msgstr "XGBoost 参考 :ref:`xgboost`。"
+msgid ":ref:`XGBoost <integrate_xgboost>`."
+msgstr ""
 
 #: ../../source/getting_started/learn.rst:37
-msgid "For LightGBM, refer to :ref:`lightgbm`."
-msgstr "LightGBM 参考 :ref:`lightgbm`。"
+msgid ":ref:`LightGBM <integrate_lightgbm>`."
+msgstr ""
 
 #: ../../source/getting_started/learn.rst:38
-msgid "For TensorFlow, refer to :ref:`tensorflow`."
-msgstr "TensorFlow 参考 :ref:`tensorflow`。"
+msgid ":ref:`TensorFlow <integrate_tensorflow>`."
+msgstr ""
 
 #: ../../source/getting_started/learn.rst:39
-msgid "For PyTorch, doc is coming soon."
-msgstr "PyTorch 集成的文档尚在编写中。"
+msgid ":ref:`PyTorch <integrate_pytorch>`."
+msgstr ""
 
+#: ../../source/getting_started/learn.rst:40
+msgid ":ref:`Joblib <integrate_joblib>`."
+msgstr ""
+
+#: ../../source/getting_started/learn.rst:41
+msgid ":ref:`Statesmodels <integrate_statsmodels>`."
+msgstr ""

docs/source/locale/zh_CN/LC_MESSAGES/index.po

@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: mars \n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2021-08-03 18:42+0800\n"
+"POT-Creation-Date: 2021-08-20 11:28+0800\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -25,8 +25,9 @@ msgstr "Mars 文档"
 msgid ""
 "Mars is a tensor-based unified framework for large-scale data computation"
 " which scales numpy, pandas, scikit-learn and many other libraries."
-msgstr "Mars 是基于张量的，用于进行大规模数据计算的统一计算框架，"
-"它可以用来并行和分布式 numpy、pandas、scikit-learn 以及众多其他 Python 库。"
+msgstr ""
+"Mars 是基于张量的，用于进行大规模数据计算的统一计算框架，它可以用来并行和"
+"分布式 numpy、pandas、scikit-learn 以及众多其他 Python 库。"
 
 #: ../../source/index.rst:11
 msgid "Architecture Overview"
@@ -111,32 +112,50 @@ msgid "**Mars learn**"
 msgstr "**Mars learn**"
 
 #: ../../source/index.rst:117
-msgid ""
-"Mars learn has also integrated many libraries, including "
-":ref:`tensorflow`, :ref:`xgboost`, :ref:`lightgbm`, :ref:`joblib` and "
-":ref:`statsmodels`."
+msgid "Mars learn also integrates with many libraries:"
+msgstr "Mars learn 也集成了许多库："
+
+#: ../../source/index.rst:119
+msgid ":ref:`TensorFlow <integrate_tensorflow>`"
+msgstr ""
+
+#: ../../source/index.rst:120
+msgid ":ref:`PyTorch <integrate_pytorch>`"
 msgstr ""
-"Mars learn 还集成了很多库，包括 "
-":ref:`tensorflow`, :ref:`xgboost`, :ref:`lightgbm`, :ref:`joblib` and "
-":ref:`statsmodels`."
 
 #: ../../source/index.rst:121
-msgid "Mars remote"
+msgid ":ref:`XGBoost <integrate_xgboost>`"
+msgstr ""
+
+#: ../../source/index.rst:122
+msgid ":ref:`LightGBM <integrate_lightgbm>`"
 msgstr ""
 
 #: ../../source/index.rst:123
+msgid ":ref:`Joblib <integrate_joblib>`"
+msgstr ""
+
+#: ../../source/index.rst:124
+msgid ":ref:`Statsmodels <integrate_statsmodels>`"
+msgstr ""
+
+#: ../../source/index.rst:127
+msgid "Mars remote"
+msgstr ""
+
+#: ../../source/index.rst:129
 msgid ":doc:`documentation <getting_started/remote>`"
 msgstr ":doc:`文档 <getting_started/remote>`"
 
-#: ../../source/index.rst:125
+#: ../../source/index.rst:131
 msgid "Mars remote allows users to execute functions in parallel."
 msgstr "Mars remote 允许用户并行执行函数。"
 
-#: ../../source/index.rst:161
+#: ../../source/index.rst:167
 msgid "Easy to scale in and scale out"
 msgstr "适应各种数据规模"
 
-#: ../../source/index.rst:163
+#: ../../source/index.rst:169
 msgid ""
 "Mars can scale in to a single machine, and scale out to a cluster with "
 "hundreds of machines. Both the local and distributed version share the "
@@ -147,23 +166,23 @@ msgstr ""
 "两种环境下可使用相同的代码。因此，Mars 可以方便地从单台机器迁移到集群，以"
 "处理更多数据或者获得更好的性能。"
 
-#: ../../source/index.rst:168
+#: ../../source/index.rst:174
 msgid "Mars can run in a few ways:"
 msgstr "Mars 能以若干种方式运行："
 
-#: ../../source/index.rst:170
+#: ../../source/index.rst:176
 msgid ":ref:`Local scheduling <local>`"
 msgstr ":ref:`本地执行 <local>`"
 
-#: ../../source/index.rst:171
+#: ../../source/index.rst:177
 msgid ":ref:`Run on cluster <deploy>`"
 msgstr ":ref:`在集群中运行 <deploy>`"
 
-#: ../../source/index.rst:172
+#: ../../source/index.rst:178
 msgid ":ref:`Run on Kubernetes <k8s>`"
 msgstr ":ref:`在 Kubernetes 中部署 <k8s>`"
 
-#: ../../source/index.rst:173
+#: ../../source/index.rst:179
 msgid ":ref:`Run on Yarn <mars_yarn>`"
 msgstr ":ref:`在 Yarn 中部署 <k8s>`"
 

docs/source/locale/zh_CN/LC_MESSAGES/reference/dataframe/frame.po

@@ -1192,6 +1192,14 @@ msgstr ""
 msgid "Create a scatter plot with varying marker point size and color."
 msgstr ""
 
+#: ../../source/reference/dataframe/frame.rst:244:<autosummary>:1
+msgid ":obj:`DataFrame.boxplot <DataFrame.boxplot>`\\"
+msgstr ""
+
+#: ../../source/reference/dataframe/frame.rst:244:<autosummary>:1
+msgid ":obj:`DataFrame.hist <DataFrame.hist>`\\"
+msgstr ""
+
 #: ../../source/reference/dataframe/frame.rst:248
 msgid "Serialization / IO / conversion"
 msgstr "序列化、IO 和转换"

docs/source/locale/zh_CN/LC_MESSAGES/reference/dataframe/indexing.po

@@ -35,6 +35,14 @@ msgstr ""
 msgid "Properties"
 msgstr "属性"
 
+#: ../../source/reference/dataframe/indexing.rst:28:<autosummary>:1
+msgid ":obj:`Index.dtype <Index.dtype>`\\"
+msgstr ""
+
+#: ../../source/reference/dataframe/indexing.rst:28:<autosummary>:1
+msgid ":obj:`Index.inferred_type <Index.inferred_type>`\\"
+msgstr ""
+
 #: ../../source/reference/dataframe/indexing.rst:28:<autosummary>:1
 msgid ":obj:`Index.is_monotonic <mars.dataframe.Index.is_monotonic>`\\"
 msgstr ""

docs/source/locale/zh_CN/LC_MESSAGES/reference/dataframe/series.po

@@ -8,14 +8,14 @@ msgid ""
 msgstr ""
 "Project-Id-Version: mars 0.5.0a2\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2021-08-04 16:55+0800\n"
+"POT-Creation-Date: 2021-08-03 18:42+0800\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=utf-8\n"
 "Content-Transfer-Encoding: 8bit\n"
-"Generated-By: Babel 2.9.1\n"
+"Generated-By: Babel 2.7.0\n"
 
 #: ../../source/reference/dataframe/series.rst:3
 msgid "Series"
@@ -1462,6 +1462,10 @@ msgstr ""
 "``Series.str`` 可被用于以字符串方式访问 Series，并应用若干方法。这些功能"
 "可通过 ``Series.str.<function/property>`` 来调用。"
 
+#: ../../source/reference/dataframe/series.rst:371:<autosummary>:1
+msgid ":obj:`Series.str.cgeneratedtalize <Series.str.cgeneratedtalize>`\\"
+msgstr ""
+
 #: ../../source/reference/dataframe/series.rst:371:<autosummary>:1
 msgid ":obj:`Series.str.casefold <mars.dataframe.Series.str.casefold>`\\ \\(\\)"
 msgstr ""
@@ -1893,6 +1897,10 @@ msgstr ""
 msgid ":obj:`Series.str.isspace <mars.dataframe.Series.str.isspace>`\\ \\(\\)"
 msgstr ""
 
+#: ../../source/reference/dataframe/series.rst:371:<autosummary>:1
+msgid "Check whether all characters in each string are whitespace."
+msgstr ""
+
 #: ../../source/reference/dataframe/series.rst:371:<autosummary>:1
 msgid ":obj:`Series.str.islower <mars.dataframe.Series.str.islower>`\\ \\(\\)"
 msgstr ""
@@ -2039,6 +2047,10 @@ msgstr ""
 msgid "Generate a pie plot."
 msgstr ""
 
+#: ../../source/reference/dataframe/series.rst:413:<autosummary>:1
+msgid ":obj:`Series.hist <Series.hist>`\\"
+msgstr ""
+
 #: ../../source/reference/dataframe/series.rst:415
 msgid "Serialization / IO / conversion"
 msgstr "序列化、IO 和转换"

docs/source/locale/zh_CN/LC_MESSAGES/reference/learn/generated/mars.learn.contrib.pytorch.MarsDataset.po

@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: mars 0.5.0a2\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2020-06-16 16:57+0800\n"
+"POT-Creation-Date: 2021-08-20 11:06+0800\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -21,3 +21,34 @@ msgstr ""
 msgid "mars.learn.contrib.pytorch.MarsDataset"
 msgstr ""
 
+#: mars.learn.contrib.pytorch.MarsDataset:1 of
+msgid ""
+"MarsDataset that inherit from torch.utils.data.Dataset. It converts from "
+"Mars basic datatype such as Tensor, DataFrame, Series. Additionally, it's"
+" constructor can receive np.ndarray, pd.DataFrame, pd.Series type."
+msgstr ""
+
+#: mars.learn.contrib.pytorch.MarsDataset.__init__:1 of
+msgid "Initialize self.  See help(type(self)) for accurate signature."
+msgstr ""
+
+#: ../../source/reference/learn/generated/mars.learn.contrib.pytorch.MarsDataset.rst:13
+msgid "Methods"
+msgstr ""
+
+#: ../../source/reference/learn/generated/mars.learn.contrib.pytorch.MarsDataset.rst:17:<autosummary>:1
+msgid ""
+":obj:`__init__ <mars.learn.contrib.pytorch.MarsDataset.__init__>`\\ "
+"\\(\\*tileables\\[\\, fetch\\_kwargs\\]\\)"
+msgstr ""
+
+#: ../../source/reference/learn/generated/mars.learn.contrib.pytorch.MarsDataset.rst:17:<autosummary>:1
+msgid "Initialize self."
+msgstr ""
+
+#: ../../source/reference/learn/generated/mars.learn.contrib.pytorch.MarsDataset.rst:17:<autosummary>:1
+msgid ""
+":obj:`get_data <mars.learn.contrib.pytorch.MarsDataset.get_data>`\\ "
+"\\(t\\, index\\)"
+msgstr ""
+

docs/source/locale/zh_CN/LC_MESSAGES/reference/tensor/set.po

@@ -8,7 +8,7 @@ msgid ""
 msgstr ""
 "Project-Id-Version: mars 0.5.0a2\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2020-06-16 16:57+0800\n"
+"POT-Creation-Date: 2021-08-20 11:06+0800\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
 "Language-Team: LANGUAGE <LL@li.org>\n"
@@ -25,19 +25,29 @@ msgstr "集合函数"
 msgid "Boolean operations"
 msgstr "布尔运算"
 
-#: ../../source/reference/tensor/set.rst:12:<autosummary>:1
+#: ../../source/reference/tensor/set.rst:13:<autosummary>:1
+msgid ":obj:`mars.tensor.in1d <mars.tensor.in1d>`"
+msgstr ""
+
+#: ../../source/reference/tensor/set.rst:13:<autosummary>:1
+msgid ""
+"Test whether each element of a 1-D tensor is also present in a second "
+"tensor."
+msgstr ""
+
+#: ../../source/reference/tensor/set.rst:13:<autosummary>:1
 msgid ":obj:`mars.tensor.isin <mars.tensor.isin>`"
 msgstr ""
 
-#: ../../source/reference/tensor/set.rst:12:<autosummary>:1
+#: ../../source/reference/tensor/set.rst:13:<autosummary>:1
 msgid "Calculates `element in test_elements`, broadcasting over `element` only."
 msgstr ""
 
-#: ../../source/reference/tensor/set.rst:12:<autosummary>:1
+#: ../../source/reference/tensor/set.rst:13:<autosummary>:1
 msgid ":obj:`mars.tensor.union1d <mars.tensor.union1d>`"
 msgstr ""
 
-#: ../../source/reference/tensor/set.rst:12:<autosummary>:1
+#: ../../source/reference/tensor/set.rst:13:<autosummary>:1
 msgid "Find the union of two tensors."
 msgstr ""