Correlation simulator

[hughbg]

For the purposes of kurtosis implementation and testing (at the moment) I am implementing a correlator simulator which has the following form:

Artificial RFI can be inserted into the pipeline to see how that effects the kurtosis. Cross-correlation of data streams can be added later.

[ledatelescope]

Hi Hugh, Does the dot file generated by bifrost include colors and options for annotation, or is that your value-added feature post-bifrost? Both would be nice features By "one channel," do you mean, select one channel? What is involved in "evaluate"? Because of Nita's peculiar definition of estimators (e.g., we do not compute a real kurtosis), additional detail, different labeling, a/o separation of steps into different flow chart boxes would be helpful. Best, L.

…

On 12/18/16 20:22, hughbg wrote: For the purposes of kurtosis implementation and testing (at the moment) I am implementing a correlator simulator which has the following form: corr <https://cloud.githubusercontent.com/assets/4837022/21298405/bb439a6e-c55e-11e6-8fc0-59c19b96992a.png> Artificial RFI can be inserted into the pipeline to see how that effects the kurtosis. Cross-correlation of data streams can be added later. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#53>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJfeMr3rIND9L6ieAJvxbKkguU94NgLmks5rJdxXgaJpZM4LQS81>.

-- Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX: 1 617-495-7345 greenhill@cfa.harvard.edu Skype: ljgreenhill www.cfa.harvard.edu/~lincoln

[hughbg]

An updated graph:

I did this by hand. It gets painful, so something to generate the graph is needed, but I don't know if Bifrost can do it and in any case it won't do colors, sub text etc.

This is the Nita equation:

SK is the Kurtosis estimator. S₁ is the sum of the integrated power values and S₂ is the sum of the squares of the integrated power values (produced by the Integrate block). There are various normalizations that get made to the FFT and power values such as dividing the FFT values by the length of the FFT window.

[hughbg]

I've added the file correlator_simulator.py to the branch correlation_simulator. The file is still being developed but it runs. It is written using low-level operations and not the high-level blocks like MultiTransformBlock, SourceBlock etc., which will be implemented in the next version.

[ledatelescope]

That is great. When sim. cross-correlation (sometime after the current system is proven out) we could feed in a mix of 100% correlated broadband noise in two inputs (e.g., random nos. with the same seed) and at a much higher amplitude uncorrelated broadband noise as random nos. w/ different seeds. Anything wrong w/ that reasoning? Best, L.

…

On 12/23/16 19:30, hughbg wrote: I've added the file correlator_simulator.py to the branch correlation_simulator. The file is still being developed but it runs. It is written using low-level operations and not the high-level blocks like MultiTransformBlock, SourceBlock etc., which will be implemented in the next version. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJfeMhKbZCmNStxToXviR2__Z2sdMTDrks5rLGe2gaJpZM4LQS81>.

-- Lincoln J. Greenhill Harvard-Smithsonian CfA Office: 1 617-495-7194 60 Garden St, Mail Stop 42 Cell: 1 650 722-7798 Cambridge, MA 02138 FAX: 1 617-495-7345 greenhill@cfa.harvard.edu Skype: ljgreenhill www.cfa.harvard.edu/~lincoln

[hughbg]

When sim. cross-correlation (sometime after the current system is proven out) we
could feed in a mix of 100% correlated broadband noise in two inputs (e.g.,
random nos. with the same seed) and at a much higher amplitude uncorrelated
broadband noise as random nos. w/ different seeds. Anything wrong w/ that
reasoning?

We can try it but I'm unsure about how cross-correlation changes things in NITA theory. I need to look at more NITA papers. At the moment I am using the power in a single channel out of an FFT and calling that auto-correlation, in this simulator.

I've added a new version of the pipeline which is substantially modified. A lot of the blocks in the pipeline do a reduction operation because they accumulate data until they have a certain amount buffered, and then process it and spit out a result - e.g. the integration block reduces to a sum. This pattern can be turned into a single block which is parameterized by the operation to be performed.

I also want to have the buffer sizes independently determined. The input chunk (span), the chunk needed to perform an operation, and the output chunk (span), can all be different sizes, for tuning purposes. That means blocks do internal buffering and the accumulator block can handle that as well.