Backends prototyping #35

nietras · 2017-10-24T09:46:54Z

Creating a PR for the backends stuff to more easily see the changes.

…project has to use the old project format because of how the CNTK nuget package is made. When the package is fixed, the project can be migrated to the new format and .net standard 2.0.

nietras

Minor comments. Backends is perhaps primarily for doing learning/training, how can we separate/handle evaluation only scenarios with minimal dependencies?

nietras · 2017-10-24T09:47:52Z

src/SharpLearning.Backend.Cntk/SharpLearning.Backend.Cntk.csproj

+    </Reference>
+    <Reference Include="System" />
+    <Reference Include="System.Core" />
+    <Reference Include="System.Xml.Linq" />


clean up includes. just the core ones should be needed.

nietras · 2017-10-24T09:49:07Z

src/SharpLearning.Backend.Cntk/packages.config

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<packages>
+  <package id="CNTK.GPU" version="2.2.0" targetFramework="net461" />


can we support both GPU and CPU by using both packages? would we need two projects for that? Does CNTK not have a separate package independent of device?

The CNTK.GPU package supports both CPU and GPU, while the CNTK.CPUOnly, only supports CPU. As far as I know, the CPUOnly package was only made in order to minimize the size of the package, avoiding all the GPU dependencies, for CPU only uses. So we should be able to support both devices only using the GPU package.

This might be different with tensorflow, which to my knowledge has a separate build for CPU and GPU.

Ok, we would have the same issue then, if we wanted to minimize size and want to run CPU only, it is a bit unfortunate that "runtime" hasn't been separated from "type definitions/API", so only the project that executes the code would depend on the "runtime". Anyway, for now focus is on training so we have to live with it, but we should keep it in mind when defining types.

nietras · 2018-01-15T10:06:42Z

src/SharpLearning.Backend/TrainingApi.cs

+            //    datasource prefetcher
+
+            //    for iterations
+            //          Tuple<ITensorSymbol> batch = source.NextBatch(batchSize: 64); // load next minibatch


We have to be very careful here... symbols cannot be direct representations of loaded data. This is important. Symbols are not actual "instances". TF couples instances with symbols by coupling placeholder symbols to actual instances if I remember correctly.

The same applies to all the below. The suggestion is based on symbols, but it is unclear whether the suggestion is actually intending to use concrete instances or define some kind of operation via symbols.

No doubt this will get more concrete, when we start actually working on it. The notion of forward/backward is interesting, although I have not seen this divide explicitly in TF for example.

Also, don't like "Tuple<>" use ;)

Yes, this is rather superficial, but does serve as a starting point to chose a direction from. When we start to implement types, it will become more concrete and then we can decide which parts should be handled via symbols and which part should be concrete instances.

I agree with the use of Tuple<>, it is just a placeholder until we decide on the real types.

…issue...

…lowSharp is not net standard.

nietras · 2018-01-15T10:47:34Z

@mdabros did a bit of updating and cleanup for projects. Tensorflow backend project needs to target net461 since TensorFlowSharp is not net standard compatible. Updated to 1.4.0 too, also updated cntk.

Cntk project should be changed to sdk project type, but for now I am just continuing on trying to get minimal backends in place.

nietras · 2018-01-15T10:55:42Z

@mdabros I should note that the way these wrapper libraries for TensorFlow and Cntk have been made makes it annoying to use due to the fixed targetting of x64 and net461. The problem is that the packages are poorly factored, there is no split between api surface (pinvoke) and runtime. Like we have done for some of our native stuff.

nietras · 2018-01-15T11:10:12Z

We should consider adding a Backends solution only, for only these projects, otherwise the solution is too heavy to work with, especially with regards to tests. Right now I simply unload all the other projects.

nietras · 2018-01-15T12:09:20Z

src/SharpLearning.Backend.Cntk.Test/CntkBackendTest.cs

+            using (var b = new CntkBackend(deviceType))
+            using (var g = b.CreateGraph())
+            {
+                g.Placeholder(DataType.Single, new int[] { 1, 2, 3 }, "p");


This fails with:

Message: Test method SharpLearning.Backend.Cntk.Test.CntkBackendTest.CntkBackendTest_CreateGraph_Placeholder threw exception: System.TypeInitializationException: The type initializer for 'CNTK.CNTKLibPINVOKE' threw an exception. ---> System.TypeInitializationException: The type initializer for 'SWIGExceptionHelper' threw an exception. ---> System.DllNotFoundException: Unable to load DLL 'Cntk.Core.CSBinding-2.3.1.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

@mdabros have you seen this before? The nuget package has been added to the project...

The dll can be found at:

SharpLearning\packages\CNTK.GPU.2.3.1\support\x64\Release

but it is not copied to output.

The CNTK package definition is really weird... seems to be ill defined.

No, I have not experienced that error before. But I do know that there are some issues with the current CNTK package: microsoft/CNTK#2527 and microsoft/CNTK#2807

However, I am not sure if that relates to the error above.

As discussed, I tried changing the SharpLearning.Backend.Cntk.Test test project to .net framework to see if this was causing the issue. Sadly, this still result in the same error.

It seems that the only way to get this running is to add a x64 solution platform, and run the test using this specific platform. Using anycpu together with x64 test settings is not enough.

I guess this ties back to the packaging you mentioned earlier. This might also be another indication that the backend projects should have their own solution.

@mdabros please do not add x64 to solution platform, that is not necessary.

@mdabros no doubt this is because the Cntk targets file does some logic on the Platform attribute of the project. So basically, you just have to change the Cntk.Test project from AnyCPU to x64, you can try and make a new branch from backends and create a PR back to it again. The devil lies in trying not to mess all project files up, which can quickly become a mess ;)

You don't need to change the Solution platform in order to do this. We might still consider whether we should in fact have just a x64 solution format since a lot is tied into this. But hopefully Cntk will become more cross-plat over time. And TF is.

@nietras Good warning, I will change only the Cntk.Test project to x64. On a side note, it seems we still have to change the Cntk.Test project format to .net framework. If not, it is not possible to use CNTK directly from the project. I think this related to the two issues I linked to above.

mdabros · 2018-01-16T13:19:50Z

src/SharpLearning.Backend.TensorFlow.Test/TensorFlowRawMnistTest.cs

+    }
+
+    // Helper class used to load and work with the Mnist data set
+    public class Mnist


@nietras This might not be the ideal way to represent and load the mnist dataset. But I guess it will suffice for initial testing.

@mdabros nah it is pretty dirty and quick way of loading as I said.

…-test-project

Add new SharpLearning.Backend.Cntk.Test project

nietras · 2018-01-23T11:55:17Z

@mdabros could you merge master into this, so my tensorflow PR doesn't have this changes shown?

Add CntkRawMnistTest. Showing a simple logistic regression example using the cntk api.

mdabros · 2018-01-25T06:43:10Z

@nietras The master has been merge to the backends branch, with the additional changes since you merged, so you can now merge the backends branch to the tensorflow PR to remove the changes shown.

TensorFlow mnist simple example and python inspiration. Python and C# mnist simple are close to identical.

* cleanup * more cleanup * Start on mnist deep in tensorflow C#. * Make C# and python mnist simple TF example identical. * Add more python inspiration and docs. Trying to export graph from C# but cannot load this in tensorboard. More work on mnist deep. * More tf deep. * minor * change to 64 batch size * Make MnistDeep python deterministic with seeds and no data shuffling. * Change python mnist deep to use gradient descent, so we can test C# version. * Most code for mnist deep complete, but dropout fails... * TF Mnist Deep runs, but does not give the same as python. * Fix mnist simple target, after changing batch size.

* Add .txt to avoid adding cntk downloaded training files. * Add simple mnist example from microsoft cntk examples. Including mnist download utils * Updated readme with installation guide * Update gitignore to include pyc files and pychache * Only download and write train/test if they do not already exist * Formatting * Add .dnn to gitignore * Add cntk conv net example. Almost direct copy from original cntk example: https://github.com/Microsoft/CNTK/blob/master/Examples/Image/Classification/ConvNet/Python/ConvNet_MNIST.py * Modiefied SimpleMNIST CNTK example to be more similar to tensorflow example [WIP] * Remove use of training_session to make the example more similar to the tensorflow equivilant * Add MnistCnnExampleTest * Update python simple mnist example * Updated CNTK CSharp simple mnist example to be similar to python equivilant * Added expected error from python implementation. Currently the CSharp version has lower error rate. But this is most likely caused by the two scritps running on different data, in training and in test, due to the loader differences * Move cntk layers to separate file * Remove unused methods * Refactor MnistCnnExampleTest to be more similar to python equivilant * Add comment about multiplication * Add comments * Add dropout * Add Convolution to layers [WIP] * Add parameter checks * Set randomization to false to have same behaviour as C# * Move mnist loader to separate file. Split Simple and CNN example to separate files. * Update MnistSimpleTest_MinibatchSource to make randomization: false more clear * Add load/Save to simple mnist C# examples. Currently fails. Loaded model returns 0 error while the trained model has around 20% error. * Update saved model names * Add comments * Fix error in TestModel_Mnist_Loader and refactor simple C# cntk examples to test on the same data using the same metod. * Remove validation set from mnist loader to get same order and number of samples in C# cntk minibatch source and mnist loader examples. For the same reason, also disabled normalization in the loader. If normalization is required, this can be added as an operation on the input of the network * Update python simple mnist example to test using 1 sample at a time to make it more similar to the C# example. Also to go trough all samples. This was not done with the previous batch size of 1024 * Updated the somple C# mnist examples to use the correct python simple mnist error rate as assertion.

Python and C# cntk cnn examples now returns the same error. This is done using the CPU device. GPU is currently not deterministic, however a special deterministic mode might be available. Examples are also made to be as similar as possible between cntk in C# and cntk in python

mdabros · 2018-08-11T09:13:21Z

Since work on this has currently stopped, I am closing the pull request. We can always re-open it again if it becomes relevant.

mada@ihfood.dk added 5 commits October 23, 2017 20:39

Add SharpLearning.Backend

5535c75

Add SharpLearning.Backend.Test

2a783f0

Add SharpLearning.Backend.TensorFlow/.Test

38db8ea

Tensorflow backend: Add reference to backend project

6ad7c01

Add SharpLearning.Backend.Cntk/.Test. The SharpLearning.Backend.Cntk …

632ed30

…project has to use the old project format because of how the CNTK nuget package is made. When the package is fixed, the project can be migrated to the new format and .net standard 2.0.

nietras commented Oct 24, 2017

View reviewed changes

mdabros mentioned this pull request Nov 14, 2017

CNTK as backend for SharpLearning.Neural #9

Open

nietras and others added 4 commits January 10, 2018 13:55

brainstorm backends

03fc14a

merge master

28d57d3

Add training api brainstorm

16be7b4

Fix a few errors in suggestion1

68e8377

nietras commented Jan 15, 2018

View reviewed changes

nietras added 5 commits January 15, 2018 11:15

cleanup sln

e41fd65

minor refactoring

e4a2dd1

update nuget packages, except MathNet, to try to fix TensorFlowSharp …

6511bed

…issue...

fix Backend.TensorFlow project, needs to target net461, since TensorF…

b14c222

…lowSharp is not net standard.

tensorflow dispose

31d890e

fix cntk and tensorflow test projects by targetting net461

988bb93

nietras added 2 commits January 15, 2018 12:20

Add backend and graph types, and simple tests.

5d82251

Add placeholder abstractions, cntk fails.

7aa9459

nietras commented Jan 15, 2018

View reviewed changes

nietras added 5 commits January 15, 2018 13:22

add raw empty test

4451d66

Add simple raw tensorf flow test

342be2f

raw

114eae3

Add mnist data stuff from TFSharp.

5c2dbd9

More mnist trialling and custom gradient descent.

aa65abb

mdabros reviewed Jan 16, 2018

View reviewed changes

mada and others added 14 commits January 17, 2018 10:03

Remove cntk test project

d2621f4

Add .net framework SharpLearning.Backend.Cntk.Test project

1ffd4b7

Add CntkBackendTest

17e3ff0

Change platform for SharpLearning.Backend.Cntk.Test

65d1c21

mnist example running but not optimizing

6b2bc77

still no updating

c549561

cost changes but not a lot

a7b06df

Merge remote-tracking branch 'origin/backends' into backends-new-cntk…

9029496

…-test-project

Merge pull request #46 from mdabros/backends-new-cntk-test-project

029400c

Add new SharpLearning.Backend.Cntk.Test project

Add CntkRawMnistTest. Showing a simple cnn using the cntk api

ff89e68

Add Reshape and GlobalAveragePooling operators to helper class

16f4107

add accumulated loss

3af3c37

Merge remote-tracking branch 'upstream/master' into backends-tf

df28b6a

Add python inspiration both mnist simple and deep plus install guide.

aa0178f

Mads Dabros and others added 3 commits January 24, 2018 09:26

Simplify cntk example to be a simply logistic regression.

b692bec

Merge pull request #47 from mdabros/backends-cntk-mnist-example

522523a

Add CntkRawMnistTest. Showing a simple logistic regression example using the cntk api.

Merge master

d561f54

nietras and others added 6 commits January 25, 2018 11:50

Merge branch 'backends' into backends-tf

30d64ae

Update mnist simple so python and C# should match completely now.

518100b

Cleanup and refactor MnistSimple in C#, now not using runner...

f02255f

Merge pull request #48 from nietras/backends-tf

db280c2

TensorFlow mnist simple example and python inspiration. Python and C# mnist simple are close to identical.

mdabros mentioned this pull request Feb 16, 2018

[Question] 2d - 3d output in neural networks #3

Closed

mdabros mentioned this pull request Feb 26, 2018

OutOfMemory Exception in F64Matrix constructor - maximum array bounds exceeded #59

Open

mdabros closed this Aug 11, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backends prototyping #35

Backends prototyping #35

nietras commented Oct 24, 2017

nietras left a comment

nietras Oct 24, 2017

nietras Oct 24, 2017

mdabros Oct 24, 2017

nietras Oct 24, 2017 •

edited

Loading

nietras Jan 15, 2018

mdabros Jan 15, 2018

nietras commented Jan 15, 2018

nietras commented Jan 15, 2018

nietras commented Jan 15, 2018

nietras Jan 15, 2018

nietras Jan 15, 2018

nietras Jan 15, 2018

mdabros Jan 15, 2018

mdabros Jan 17, 2018

mdabros Jan 17, 2018

nietras Jan 17, 2018

nietras Jan 17, 2018

nietras Jan 17, 2018

mdabros Jan 17, 2018

mdabros Jan 16, 2018

nietras Jan 17, 2018

nietras commented Jan 23, 2018

mdabros commented Jan 25, 2018

mdabros commented Aug 11, 2018

Backends prototyping #35

Backends prototyping #35

Conversation

nietras commented Oct 24, 2017

nietras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nietras Oct 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nietras commented Jan 15, 2018

nietras commented Jan 15, 2018

nietras commented Jan 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nietras commented Jan 23, 2018

mdabros commented Jan 25, 2018

mdabros commented Aug 11, 2018

nietras Oct 24, 2017 •

edited

Loading