WIP: continuous monitoring support #67

breznak · 2016-01-06T21:38:56Z

cleanups
work for polling file changes
- monitoring for large files fails (speed up chunk?)
- add file proxy server from npm
broke remote file streaming, in a cleanup
add UI control for window size Add UI control for "zoom"/window size in continuous monitoring mode #73

Fixes: #17

+ change max file size to 5MB (rounded)

breznak · 2016-01-06T21:41:10Z

client/src/app/appCtrl.js

+  };
+
+  // main "load" function that supports both URL/local file
+  $scope.loadFile = function(event) {


this can be removed, but I thought if you could change the buttons to work with only this method? So there'd be only 1 button "Load". Do you think it'd work?

breznak · 2016-01-06T21:43:04Z

@jefffohl for your preliminary review, if you have time. thx

breznak · 2016-01-07T14:36:59Z

@jefffohl can you please review this?
The monitoring/online updates are working as a draft.
I have a problem with the large (2M+) file, re-reading it in PapaParse take some time, but I know the updatade file size, if possible, I'd like to seek/skip to the bytes read and only read the new bytes. I'm somewhat doing it with conditional skipping in the chunk call, but it's still slow

breznak · 2016-01-07T14:39:39Z

client/src/app/appCtrl.js

+    }
+    $scope.canDownload(); // will set the local file= true/false
+    loadFileHelper();
+    setMonitoringTimer(appConfig.POLLING_INTERVAL); //FIXME create an entry element for numeric value in UI for this, each change should call setMonitoringTimer()


@jefffohl could you do the UI for this? Some way to enter numeric values, 0 means disabled, >0 means delay in polling interval for monitoring in ms. On a change the element should call the setMonitoringTimer()

Conflicts: build/app.js client/src/app/appConfig.js client/src/app/appCtrl.js

breznak · 2016-01-13T00:08:09Z

resolved merge conflicts, hope I haven't introduced any bugs during the merge.

Conflicts: build/app.js client/src/app/appConfig.js client/src/app/appCtrl.js

breznak · 2016-01-19T10:29:28Z

@jefffohl how much of this PR are you reusing in your work on #78 ? I was thinking about rebasing a PR off this PR to have only the working functionality, introduction of the Angular's $interval and possibly limited support for monitoring with diff files only. I hope the work on proxy should then be separate and further extend the monitor. capabilities.

jefffohl · 2016-01-19T20:34:22Z

I am not using any of this code, as it seemed that the approach wouldn't work. I am sorry if I wasn't clear about that. I don't think we should be working on the same feature at the same time.

I am currently working on the server implementation, which is required for online monitoring.

breznak · 2016-01-19T21:04:50Z

Thanks Jeff.
So it would be ok if I go ahead with

introduction of the Angular's $interval and possibly limited support for monitoring with diff files only

from this PR? It would not intervene with any of your work on the server?

jefffohl · 2016-01-19T21:18:18Z

To be honest, I don't think it is a good idea to keep working on this PR. The code is too divergent between our two branches, and it is going to be tough to merge them. It might be best to wait until I am finished with my server work, and then, if you want to implement a method for monitoring files using a diff method and the fileReader API, you can do that after I have finished the server. At that point, I am hoping you will be able to rebase the server work on to this branch. I don't think it is wise for us to be working concurrently on these features, as there is so much code overlap.

breznak · 2016-01-19T21:31:44Z

Ok, I'll wait for your server part. Btw, do you have a rough ETA?

jefffohl · 2016-01-19T21:34:13Z

I am hoping to have it ready before Friday, but there are some things I haven't figured out how to do yet. If I get stuck, I will let you know, and ask for help.

jefffohl · 2016-01-23T01:21:34Z

@breznak - Sorry, I am still working on this. Making progress though. I hope that this is not holding you back from things that you need to do.

jefffohl · 2016-01-28T22:48:44Z

@breznak - sorry still working on this. Some things came up this week that I had to prioritize.

jefffohl · 2016-02-04T21:51:54Z

@breznak - I am working on this issue on this branch: https://github.com/jefffohl/nupic.visualizations/tree/issue78

There are a number of fundamental changes happening here. Specifically:

Parsing the CSV files on the server using the node module, csv-parse
Using socket.io to communicate with the server. This allows the client and server to communicate with each other using events. This will be useful in handling files that are continuously updating.
Removed the "Browse..." button, as using this does not allow us to create a file handle that can be updated, so it is no longer needed.

Remotely hosted files can be parsed, as well as local files. For local files, you must give a full path from the system root to the file. Note that because of this feature, the code as it currently stands should NEVER be hosted on a public server, as that will give the public access to all the files on the web server. We will need to figure out a good way to make sure that this feature is disabled if the app is ever hosted publicly. Right now, the intent is that the app should only be run locally.

Local files can be continuously updated. I added a little bash script: /examples/tests/update.sh that will continuously update the no_timestamp.csv file. You can use this to test the continuous update feature.

If you want to test this branch as it is, first make sure that node and npm are installed. Then run npm install followed by npm start

jefffohl · 2016-02-13T19:34:39Z

@breznak I have a working prototype on this branch: https://github.com/jefffohl/nupic.visualizations/tree/issue78

This version allows you to "play" a local file. You can enter the path to a local file, and it will load a portion of the file. Clicking the "play" button will start to stream the file from the file system into the web app.

Next steps are:

Allow the user to adjust the size of the window - that is, the range of data to graph
Allow the user to adjust the speed of play.
Allow the user to "scrub" across the file backwards and forwards to find the spot to graph.

breznak · 2016-02-25T13:23:27Z

@jefffohl sorry for the delay, I was off the grid

That sounds great, will you make a PR?

jefffohl · 2016-02-25T17:12:43Z

I could. I was thinking that I would get some of the other steps completed first, but I can make a PR if you like. I am on vacation right now, but I can do it when I return on Monday.

jefffohl · 2016-03-15T16:12:04Z

@brezank - I am thinking of making a new PR for what I have now in https://github.com/jefffohl/nupic.visualizations/tree/issue78

Note that this is almost a complete rewrite of the architecture, due to parsing the data now on the server. Some additional things I also changed along the way:

Modified the GUI layout
Changed the way notifications are presented
Modified the threshold highlighting function. I found that having the threshold highlights having padding was creating problems. If there were many highlights close together, they would all blend together, giving the user a misleading sense of what is happening. I changed the highlighting function to very precisely mark where the data is exceeding the threshold. I realize that I should have made this a separate PR. If this bothers you, let's discuss.

Let me know your thoughts.

breznak · 2016-03-22T23:33:46Z

@jefffohl thanks Jeff, sorry for the silence, I was somehow missing all notifications from this project :/
Will test over the weekend.. IMHO sounds good.

breznak added 4 commits January 6, 2016 20:51

add appconfig.POLLING_INTERVAL

84b1abb

+ change max file size to 5MB (rounded)

Merge branch 'master' into online_polling

2dfdf36

unify the download/stream*File functions

af8ee14

cleanup + loadFile() unifies getRemote/LocalFile

8b93e82

breznak reviewed Jan 6, 2016
View reviewed changes

breznak added 13 commits January 6, 2016 22:57

document the remote file loading feature

3367438

function canDownload() extended to judge RangeHeader

81636ef

new getFileSize function used

d478da6

using common getFile function

64ba327

add FIXME

2f3843f

fix getFileSize & canDownload, WIP getRemoteFile

facb3e5

initial online monitoring/polling/streaming support

3dbf998

add example/readme for the online monitoring mode

9def3e2

monitoring: speedup consecutive pollings

428959f

record file.size

c0f0e63

improve skipping for large file, till size reached

21f6277

avoid parallel timers

9eea799

fix file size updated as file grows during monitoring

5f38c9c

breznak reviewed Jan 7, 2016
View reviewed changes

breznak added 2 commits January 9, 2016 00:40

increase local CHUNK_SIZE for Papa

23f0597

Merge branch 'master' into online_polling

f86d7ab

Conflicts: build/app.js client/src/app/appConfig.js client/src/app/appCtrl.js

breznak added 4 commits January 13, 2016 01:29

use Angular's $interval

201b580

merge functionality to loadFile (local, remote)

6f10a95

WIP chaining promises

b61b3e8

Merge branch 'master' into online_polling

f34d2f4

Conflicts: build/app.js client/src/app/appConfig.js client/src/app/appCtrl.js

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: continuous monitoring support #67

WIP: continuous monitoring support #67

breznak commented Jan 6, 2016

breznak Jan 6, 2016

breznak commented Jan 6, 2016

breznak commented Jan 7, 2016

breznak Jan 7, 2016

breznak commented Jan 13, 2016

breznak commented Jan 19, 2016

jefffohl commented Jan 19, 2016

breznak commented Jan 19, 2016

jefffohl commented Jan 19, 2016

breznak commented Jan 19, 2016 via email

jefffohl commented Jan 19, 2016

jefffohl commented Jan 23, 2016

jefffohl commented Jan 28, 2016

jefffohl commented Feb 4, 2016

jefffohl commented Feb 13, 2016

breznak commented Feb 25, 2016

jefffohl commented Feb 25, 2016

jefffohl commented Mar 15, 2016

breznak commented Mar 22, 2016

WIP: continuous monitoring support #67

Are you sure you want to change the base?

WIP: continuous monitoring support #67

Conversation

breznak commented Jan 6, 2016

breznak Jan 6, 2016

Choose a reason for hiding this comment

breznak commented Jan 6, 2016

breznak commented Jan 7, 2016

breznak Jan 7, 2016

Choose a reason for hiding this comment

breznak commented Jan 13, 2016

breznak commented Jan 19, 2016

jefffohl commented Jan 19, 2016

breznak commented Jan 19, 2016

jefffohl commented Jan 19, 2016

breznak commented Jan 19, 2016 via email

jefffohl commented Jan 19, 2016

jefffohl commented Jan 23, 2016

jefffohl commented Jan 28, 2016

jefffohl commented Feb 4, 2016

jefffohl commented Feb 13, 2016

breznak commented Feb 25, 2016

jefffohl commented Feb 25, 2016

jefffohl commented Mar 15, 2016

breznak commented Mar 22, 2016