Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Simple engine #30

Merged
merged 27 commits into from
Aug 31, 2015
Merged

Simple engine #30

merged 27 commits into from
Aug 31, 2015

Conversation

hotpxl
Copy link
Contributor

@hotpxl hotpxl commented Aug 24, 2015

not so simple

Missing parts:
1. Workers (this should be easy since all conflicts have been resolved)
2. Runtime context (I have totally no idea)
antinucleon added a commit that referenced this pull request Aug 31, 2015
@antinucleon antinucleon merged commit 61aa571 into master Aug 31, 2015
@antinucleon antinucleon deleted the simple-engine branch August 31, 2015 22:53
@antinucleon antinucleon restored the simple-engine branch August 31, 2015 22:53
@tqchen tqchen deleted the simple-engine branch September 12, 2015 17:27
yzhliu pushed a commit to yzhliu/mxnet that referenced this pull request Feb 19, 2016
@amithr1 amithr1 mentioned this pull request Nov 17, 2017
iblislin pushed a commit to iblislin/incubator-mxnet that referenced this pull request Mar 18, 2018
piiswrong pushed a commit that referenced this pull request Jul 2, 2018
* Test input a graph.

* Update foreach to execute the subgraph.

* print inputs/outputs in foreach.

* Remove print.

* add test code for foreach.

* exec foreach outside the engine.

* Implements forward of foreach.

* Add support for variable numbers of inputs and outputs.

* Add a python wrapper for foreach.

* Fix the order of inputs.

* add test with lstm.

* hide C version of foreach.

* fix a bug temporarily.

* Test free variables.

* change for the new interface of InputGraph attribute.

* Add attribute to the subgraph.

* Handle free variables.

* Get all input symbols of a subgraph.

* Fix shape, dtype and storage inference.

* reorganize the output of foreach.

* Add a gluon RNN unroll with symbol foreach.

* print unnecessary print.

* have imperative and symbolic foreach.

* Fix an error after moving foreach.

* Fix imperative foreach

* Fix a minor problem.

* Use CachedOp to execute subgraph.

* update TODO.

* make foreach op use FStatefulComputeEx.

TODO we need to change stateful executor to handle subgraph.

* Add backward.

* Fix bugs.

* enable backward test in lstm.

* Fix a bug in foreach backward for free variables.

* change for the new CachedOp.

* Detect the backward computation.

* Fix bugs in foreach.

* fix tests.

* update tests.

* check state shape.

* enable nested foreach.

* remove print.

* fix a bug in test.

* handle infer storage type for backward.

* address comments.

* address comments.

* move some common functions out.

* address comments.

* fix lint.

* Fix lint.

* add doc.

* undo modification in imperative.h

* add doc and remove example code.

* fix lint.

* fix lint.

* Fix lint.

* make nd.foreach and sym.foreach consistent.

* fix compile error.

* address comments.

* update.

* check for loop only works for dense arrays.

* move control flow op out of nn/

* fix include.

* add a test in gluon.

* work for GPU.

* small fix.

* remove subgraph_name

* create loop state for reuse in the future.

* move code.

* Revert "remove subgraph_name"

This reverts commit 977f562.

* cut graph.

* rename new var nodes.

* Fix tests.

* Fix bugs caused by ctypes (#29)

* Add save/load json in testcases for foreach (#30)

* support subgraph in stateful executor.

* Fix compilation.

* fix a bug when a subgraph has variable nodes.

* Fix a bug of getting symbols.

* copy var nodes.

* Fix getting op states.

* fix lint error.

* address comments.

* fix lint error.

* simplify the execution of subgraph in the main thread.

* fix lint error.

* avoid waiting for computation in each iteration.

* reuse cached op for inference.

* share memory across mini-batches.

* reuse memory.

reuse memory between iterations in inference.
reuse memory between mini-batches in training.

* add tests for multiple batches.

* remove entry.

* add benchmark for foreach.

* benchmark large batch size.

* Fix the benchmark for GPU.

* address comments.

* update shape/dtype/storage inference.

* update contrib API docs.

* support nested foreach.

* use a single CachedOp for all iterations.

* use large dim.

* update benchmark.

* update benchmark.

* update benchmark.

* update benchmark.

* return symbol arrays correctly in MXSymbolCutSubgraph.

* return symbol arrays in MXSymbolGetInputSymbols.

* fix lint error.

* use cachedop to infer storage in backward.

* fix scala API.

* update comments.

* fix scala.

* fix test.

* fix attribute name.

* move benchmark.

* fix the mapping of operator inputs/outputs and subgraph inputs/outputs.

* add tests for dtype/shape inference.

* reorganize tests.

* fix a bug of cutting NodeEntry.

When two node entries refer to the same output of a node, we should
create only one var node for these two node entries.

* fix lint error.

* handle the case that outputs are inputs.

* handle the case that inputs aren't used.

* handle the case without output data.

* fix a bug in foreach backward.

* fix a bug when there isn't output data.

* Fix lint error.

* test diff Gluon RNN cells.

* test all symbol RNN cells.

* adjust the test precision.

* Fix a bug in getting a list of variable names.

We can't get a list of variable names from a hashtable. The order can't
be guaranteed. Python2 and Python3 output different orders.

* fix lint error.

* Test 1D array.

* fix a bug when subgraph inputs and outputs share NDArray.

* fix.

* fix

* add comments.
junrushao pushed a commit to junrushao/mxnet that referenced this pull request Jul 3, 2018
XinYao1994 pushed a commit to XinYao1994/incubator-mxnet that referenced this pull request Aug 29, 2018
* Test input a graph.

* Update foreach to execute the subgraph.

* print inputs/outputs in foreach.

* Remove print.

* add test code for foreach.

* exec foreach outside the engine.

* Implements forward of foreach.

* Add support for variable numbers of inputs and outputs.

* Add a python wrapper for foreach.

* Fix the order of inputs.

* add test with lstm.

* hide C version of foreach.

* fix a bug temporarily.

* Test free variables.

* change for the new interface of InputGraph attribute.

* Add attribute to the subgraph.

* Handle free variables.

* Get all input symbols of a subgraph.

* Fix shape, dtype and storage inference.

* reorganize the output of foreach.

* Add a gluon RNN unroll with symbol foreach.

* print unnecessary print.

* have imperative and symbolic foreach.

* Fix an error after moving foreach.

* Fix imperative foreach

* Fix a minor problem.

* Use CachedOp to execute subgraph.

* update TODO.

* make foreach op use FStatefulComputeEx.

TODO we need to change stateful executor to handle subgraph.

* Add backward.

* Fix bugs.

* enable backward test in lstm.

* Fix a bug in foreach backward for free variables.

* change for the new CachedOp.

* Detect the backward computation.

* Fix bugs in foreach.

* fix tests.

* update tests.

* check state shape.

* enable nested foreach.

* remove print.

* fix a bug in test.

* handle infer storage type for backward.

* address comments.

* address comments.

* move some common functions out.

* address comments.

* fix lint.

* Fix lint.

* add doc.

* undo modification in imperative.h

* add doc and remove example code.

* fix lint.

* fix lint.

* Fix lint.

* make nd.foreach and sym.foreach consistent.

* fix compile error.

* address comments.

* update.

* check for loop only works for dense arrays.

* move control flow op out of nn/

* fix include.

* add a test in gluon.

* work for GPU.

* small fix.

* remove subgraph_name

* create loop state for reuse in the future.

* move code.

* Revert "remove subgraph_name"

This reverts commit 977f562.

* cut graph.

* rename new var nodes.

* Fix tests.

* Fix bugs caused by ctypes (apache#29)

* Add save/load json in testcases for foreach (apache#30)

* support subgraph in stateful executor.

* Fix compilation.

* fix a bug when a subgraph has variable nodes.

* Fix a bug of getting symbols.

* copy var nodes.

* Fix getting op states.

* fix lint error.

* address comments.

* fix lint error.

* simplify the execution of subgraph in the main thread.

* fix lint error.

* avoid waiting for computation in each iteration.

* reuse cached op for inference.

* share memory across mini-batches.

* reuse memory.

reuse memory between iterations in inference.
reuse memory between mini-batches in training.

* add tests for multiple batches.

* remove entry.

* add benchmark for foreach.

* benchmark large batch size.

* Fix the benchmark for GPU.

* address comments.

* update shape/dtype/storage inference.

* update contrib API docs.

* support nested foreach.

* use a single CachedOp for all iterations.

* use large dim.

* update benchmark.

* update benchmark.

* update benchmark.

* update benchmark.

* return symbol arrays correctly in MXSymbolCutSubgraph.

* return symbol arrays in MXSymbolGetInputSymbols.

* fix lint error.

* use cachedop to infer storage in backward.

* fix scala API.

* update comments.

* fix scala.

* fix test.

* fix attribute name.

* move benchmark.

* fix the mapping of operator inputs/outputs and subgraph inputs/outputs.

* add tests for dtype/shape inference.

* reorganize tests.

* fix a bug of cutting NodeEntry.

When two node entries refer to the same output of a node, we should
create only one var node for these two node entries.

* fix lint error.

* handle the case that outputs are inputs.

* handle the case that inputs aren't used.

* handle the case without output data.

* fix a bug in foreach backward.

* fix a bug when there isn't output data.

* Fix lint error.

* test diff Gluon RNN cells.

* test all symbol RNN cells.

* adjust the test precision.

* Fix a bug in getting a list of variable names.

We can't get a list of variable names from a hashtable. The order can't
be guaranteed. Python2 and Python3 output different orders.

* fix lint error.

* Test 1D array.

* fix a bug when subgraph inputs and outputs share NDArray.

* fix.

* fix

* add comments.
StOnEGiggity pushed a commit to StOnEGiggity/incubator-mxnet that referenced this pull request Dec 11, 2018
channel960608 pushed a commit to channel960608/incubator-mxnet that referenced this pull request Aug 17, 2021
parent 1ea18ed
author cspchen <cspchen@amazon.com> 1629186478 +0800
committer cspchen <cspchen@amazon.com> 1629186485 +0800

# This is a combination of 21 commits.
# This is the 1st commit message:

FIX: loading on Linux platform

# This is the commit message #2:

UPDATE: ci for java-package

# This is the commit message apache#3:

UPDATE: ci for java-package

# This is the commit message apache#4:

UPDATE: ci for java-package

# This is the commit message apache#5:

UPDATE: ci for java-package

# This is the commit message apache#6:

UPDATE: ci for java-package

# This is the commit message apache#7:

UPDATE: ci for java-package

# This is the commit message apache#8:

UPDATE: ci for java-package

# This is the commit message apache#9:

UPDATE: ci for java-package

# This is the commit message apache#10:

UPDATE: ci for java-package

# This is the commit message apache#11:

UPDATE: ci for java-package

# This is the commit message apache#12:

UPDATE: ci for java-package

# This is the commit message apache#13:

UPDATE: ci for java-package

# This is the commit message apache#14:

UPDATE: ci for java-package

# This is the commit message apache#15:

UPDATE: ci for java-package

# This is the commit message apache#16:

UPDATE: ci for java-package

# This is the commit message apache#17:

UPDATE: ci for java-package

# This is the commit message apache#18:

UPDATE: ci for java-package

# This is the commit message apache#19:

UPDATE: ci for java-package

# This is the commit message apache#20:

UPDATE: ci for java-package

# This is the commit message apache#21:

UPDATE: ci for java-package

# This is the commit message apache#22:

UPDATE: ci for java-package

# This is the commit message apache#23:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#24:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#25:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#26:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#27:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#28:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#30:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#31:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#32:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#33:

UPDATE: jenkins ci scripts for java-package

# This is the commit message apache#34:

FIX: issues in Resource close methods

# This is the commit message apache#35:

FIX: issues in Resource close methods
lanking520 pushed a commit that referenced this pull request Oct 5, 2021
* ADD: first commit

* ADD: load local libraries

* UPDATE: use header files of MXNet 2.0

* ADD: load binaries from environment variable, java properties or jar files.

* ADD: add symbol loading and closing
add module integration

* ADD: [WIP] Component MxNDArray

* ADD: [WIP] Component MxNDArray

* ADD: Component MxNDArray. Pass static compilation check

* ADD: Component CachedOp

* REMOVE: module api which is no use

* FIX: dependency missing

* ADD: [WIP] add test cases for NdArray and CachedOp

* ADD: [WIP] add test cases for NdArray and CachedOp

* ADD: implement of the forward function for MxSymbolblock

* ADD: implement of the forward function for MxSymbolblock

* ADD: Sample model downloader for MLP

* ADD: doc

* ADD: Front-end module for inference, class MxModel, Predictor and so on.

* FIX: Mxnet crash when process exits.

* FIX: remove and initialize 3rdparty directory

* FIX: revert version of submodules: dlpack, dmlc-core, googletest, ps-lite

* Revert "FIX: remove and initialize 3rdparty directory"

This reverts commit d097675

* FIX: redownload files in 3rdparty

* FIX: reset --hard the version of a few submodules

* FIX: reset --hard the version of a few submodules

* FIX: reset --hard the version of a few submodules

* PERF: [WIP] optimize code structure and memory management and

* ADD: add copyright; remove Mx prefix for some classes

* ADD: add copyright

* FIX: group name, path to find header files

* UPDATE: README.md

* ADD: copyright

* ADD: copyright

* ADD: package-info

ADD: ci

ADD: ci

ADD: make modification to trigger ci

ADD: ci

ADD: ci

ADD: ci

ADD: ci

ADD: gradlew

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

ADD: java_package_ci

FIX: build failure

* FIX: ci config file

* UPDATE: remove ParameterStore and some scripts

UPDATE: remove Initializer.java

* UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

FIX: issues in Resource close methods

FIX: issues in Resource close methods

FIX: issues in Resource close methods

UPDATE: remove scripts for dev

* FIX: loading on Linux platform

* # This is a combination of 18 commits.
# This is the 1st commit message:

FIX: loading on Linux platform

# This is the commit message #2:

UPDATE: ci for java-package

# This is the commit message #3:

UPDATE: ci for java-package

# This is the commit message #4:

UPDATE: ci for java-package

# This is the commit message #5:

UPDATE: ci for java-package

# This is the commit message #6:

UPDATE: ci for java-package

# This is the commit message #7:

UPDATE: ci for java-package

# This is the commit message #8:

UPDATE: ci for java-package

# This is the commit message #9:

UPDATE: ci for java-package

# This is the commit message #10:

UPDATE: ci for java-package

# This is the commit message #11:

UPDATE: ci for java-package

# This is the commit message #12:

UPDATE: ci for java-package

# This is the commit message #13:

UPDATE: ci for java-package

# This is the commit message #14:

UPDATE: ci for java-package

# This is the commit message #15:

UPDATE: ci for java-package

# This is the commit message #16:

UPDATE: ci for java-package

# This is the commit message #17:

UPDATE: ci for java-package

# This is the commit message #18:

UPDATE: ci for java-package

* # This is a combination of 27 commits.
parent 1ea18ed
author cspchen <cspchen@amazon.com> 1629186478 +0800
committer cspchen <cspchen@amazon.com> 1629186485 +0800

# This is a combination of 21 commits.
# This is the 1st commit message:

FIX: loading on Linux platform

# This is the commit message #2:

UPDATE: ci for java-package

# This is the commit message #3:

UPDATE: ci for java-package

# This is the commit message #4:

UPDATE: ci for java-package

# This is the commit message #5:

UPDATE: ci for java-package

# This is the commit message #6:

UPDATE: ci for java-package

# This is the commit message #7:

UPDATE: ci for java-package

# This is the commit message #8:

UPDATE: ci for java-package

# This is the commit message #9:

UPDATE: ci for java-package

# This is the commit message #10:

UPDATE: ci for java-package

# This is the commit message #11:

UPDATE: ci for java-package

# This is the commit message #12:

UPDATE: ci for java-package

# This is the commit message #13:

UPDATE: ci for java-package

# This is the commit message #14:

UPDATE: ci for java-package

# This is the commit message #15:

UPDATE: ci for java-package

# This is the commit message #16:

UPDATE: ci for java-package

# This is the commit message #17:

UPDATE: ci for java-package

# This is the commit message #18:

UPDATE: ci for java-package

# This is the commit message #19:

UPDATE: ci for java-package

# This is the commit message #20:

UPDATE: ci for java-package

# This is the commit message #21:

UPDATE: ci for java-package

# This is the commit message #22:

UPDATE: ci for java-package

# This is the commit message #23:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #24:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #25:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #26:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #27:

UPDATE: jenkins ci scripts for java-package

* MERGE: resolve conflicts

* MERGE: resolve conflicts

* # This is a combination of 35 commits.
parent 1ea18ed
author cspchen <cspchen@amazon.com> 1629186478 +0800
committer cspchen <cspchen@amazon.com> 1629186485 +0800

# This is a combination of 21 commits.
# This is the 1st commit message:

FIX: loading on Linux platform

# This is the commit message #2:

UPDATE: ci for java-package

# This is the commit message #3:

UPDATE: ci for java-package

# This is the commit message #4:

UPDATE: ci for java-package

# This is the commit message #5:

UPDATE: ci for java-package

# This is the commit message #6:

UPDATE: ci for java-package

# This is the commit message #7:

UPDATE: ci for java-package

# This is the commit message #8:

UPDATE: ci for java-package

# This is the commit message #9:

UPDATE: ci for java-package

# This is the commit message #10:

UPDATE: ci for java-package

# This is the commit message #11:

UPDATE: ci for java-package

# This is the commit message #12:

UPDATE: ci for java-package

# This is the commit message #13:

UPDATE: ci for java-package

# This is the commit message #14:

UPDATE: ci for java-package

# This is the commit message #15:

UPDATE: ci for java-package

# This is the commit message #16:

UPDATE: ci for java-package

# This is the commit message #17:

UPDATE: ci for java-package

# This is the commit message #18:

UPDATE: ci for java-package

# This is the commit message #19:

UPDATE: ci for java-package

# This is the commit message #20:

UPDATE: ci for java-package

# This is the commit message #21:

UPDATE: ci for java-package

# This is the commit message #22:

UPDATE: ci for java-package

# This is the commit message #23:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #24:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #25:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #26:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #27:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #28:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #30:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #31:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #32:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #33:

UPDATE: jenkins ci scripts for java-package

# This is the commit message #34:

FIX: issues in Resource close methods

# This is the commit message #35:

FIX: issues in Resource close methods

* parent 1ea18ed
author cspchen <cspchen@amazon.com> 1629186478 +0800
committer cspchen <cspchen@amazon.com> 1629186485 +0800

FIX: loading on Linux platform

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: ci for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

UPDATE: jenkins ci scripts for java-package

FIX: issues in Resource close methods

FIX: issues in Resource close methods

FIX: issues in Resource close methods

DOC: add doc

STYLE: change code style for pmd check

FIX: avoid the register for a signal handler twice

STYLE: pass pmd check

UPDATE: remove unused scripts

* FIX: solve problems before merge

* UPDATE: remove useless files

* FIX: licence to apache

* FIX: sanity check

* FIX: sanity check

* FIX: sanity check

* FIX: remove unused files

* FIX: remove unused files

* DOC: add document

* FIX: doesn't work on osx

* FIX: clang static check

* FIX: sanity

* FIX: skip signal handler registration when building java package

* FIX: remove DataType String

* FIX: add license

Co-authored-by: cspchen <cspchen@amazon.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants