Online2 NNet3 TCP server program

[danijel3]

Several people asked for this and I feel like it would be a nice addition to the project.

The protocol is much simpler than the audio-server program I did a while ago - audio in -> text out.

The way it's made now is nice for a live demo (I added some commands to the doxygen docs), but may still lack some features for real-life use.

The main issue I have is that the new decoder is slightly different than before. The old decoder had a way to check which part of the output is final and which is "partial". This time, I can only check the current best path every N seconds (eg once per second of input audio). I use endpointing to determine when to finalize decoding.

Now, what would be really nice it to have online speech detection and speaker diarization included with this, but I know it's probably not happening too soon. What can be done (and I may do it myself if I find time) is a multithreaded version of the program with shared acoustic model and FST. Also, I bet it could be possible to combine the grammar version with the server to allow runtime vocabulary modification.

I also have a web interface that works with this server, but I'm not sure if it would fit the main Kaldi repo, so I'll probably make a separate repo for that (if anyone wants it).

I'm open to comments and suggestions.

[danpovey]

@hhadian since you have done things of this nature, perhaps you could try to find time to review it? Not just from code details, but whether the whole approach is a direction we want to go in. No hurry.

[hhadian]

Sure, will do.

[hhadian]

Thanks. Some initial comments..
I think this could be useful for some people who want to decode single files. And it won't need a lot of maintenance because it's simple TCP and it's single-threaded. However, the issues I explained in the comments should be resolved (and I guess I'll need to test it afterwards).

[dogancan]

Yes it seems our code had a bug there!

[dogancan]

Hmm, actually what you are suggesting is not right either because the offset should be from the beginning of the last segment not the beginning of last chunk.

[dogancan]

I mean what you are saying in words is correct, what the code change does is not.

[dogancan]

I updated the pykaldi example. The offset is again from the beginning of the file but now it is an accumulator. Let me know if it does not make sense to you.

[nshmyrev]

@dogancan this is a great catch, I didn't notice that before. Actually I think it is also a design problem, in your pykaldi code it is simple to retrieve the remaining samples but in many cases processing and finalization belongs to different methods or executed in different places of the code. In that case access to audio buffer is already lost. It would be nice to have something like feature_pipeline.GetRemainingSamples then.

[dogancan]

I agree. There used to be a function for this purpose but it was not part of the feature pipeline interface, so it wouldn't be useful when access to audio buffer is already lost. It was removed some time ago as part of #2672.

[nshmyrev]

Thanks. Or probably one should not finalize the feature pipeline every time in online decoding, maybe its better to simply keep it running.

[danpovey]

I didn't really expect that people will finalize it unless they know they don't want to process that stream any more. The idea is you call that when you are truly done with it.

…

On Mon, Jan 21, 2019 at 3:26 PM Nickolay V. Shmyrev < ***@***.***> wrote: Thanks. Or probably one should not finalize the feature pipeline every time in online decoding, maybe its better to simply keep it running. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2938 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu4_iJ0pIRMkiUPggAN8ukkXfY_WNks5vFiKHgaJpZM4ZhLPG> .

[dogancan]

Unfortunately, feature pipeline needs to be reinitialized whenever decoder is reinitialized since the decoder starts counting the frames from 0 after the InitDecoding call.

[danpovey]

It might be necessary to store parts of the waveform outside the feature-pipeline object. Or if you can think of a way to extend it to support getting the remainder, I could have a look. I don't think it should be important in most applications to ensure that the frame boundaries are exactly right between decoder invocations.

…

On Mon, Jan 21, 2019 at 3:33 PM Dogan Can ***@***.***> wrote: Unfortunately, feature pipeline needs to be reinitialized whenever decoder is reinitialized since the decoder starts counting the frames from 0 after the init_decoding call. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2938 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu6lh7V4QNl_DfdFoQA9wtUjwJpyxks5vFiQEgaJpZM4ZhLPG> .

[danijel3]

Just to expand on the netcat issue, mentioned a while back, somewhere above. It is true that netcat doesn't send shutdown automatically on EOF because certain servers (when they can interpret the contents of the data they receive) the server is the one that shuts down the connection. In our case, we accept raw streaming audio and can't determine if it's done or not. Only way to determine that would be to introduce some sort of "protocol", which is something I really wanted to avoid this time. This means, that the client is responsible for terminating the connection and not the server. Thankfully, netcat has the '-N' switch to do just that. Furthermore, the timeout option will remain available if someone really wants to use it, although personally, I'd avoid it if possible. It has to be set pretty high to avoid any false alarms and that somewhat defeats the purpose of having the timeout in the first place.

[dogancan]

Seems like we missed one :)

online2-net-nnet3-latgen-faster -> online2-tcp-nnet3-decode-faster

Also let's rename the subsection to online_decoding_nnet3_tcp.

[danijel3]

fixed

[dogancan]

Let's make word_syms a constant reference instead of a constant pointer. Same for the other version of this function.

[danijel3]

Ok, I did this, but then I noticed that "word_sysm" is created as a pointer (line 193) using the ReadText function. Shall we leave it like before or derefence it every time we call LatticeToString?

[dogancan]

I think it is best to create word_syms as a pointer and dereference it when calling LatticeToString.

[dogancan]

@danijel3 I think the PR looks pretty good. I added a couple of new comments but nothing major.

@danpovey What is your view on having a command line option to carry the adaptation_state between recordings?

[danpovey]

Thanks. I hope to merge in a few days, once Hossein has a quick look and confirms that he doesn't still have any concerns.

[danpovey]

sorry pressed wrong button-- reopened again.

[hhadian]

Some very minor comments and a question: I know this program is single-threaded but I was wondering what would happen when a second client tries to connect while a client is already connected? It seems to me that the 2nd client will be rejected, is that right?

[danijel3]

Some very minor comments and a question: I know this program is single-threaded but I was wondering what would happen when a second client tries to connect while a client is already connected? It seems to me that the 2nd client will be rejected, is that right?

Yes. The server is single threaded and will reject any clients that try to connect after the first one. It's by design as this program isn't meant to distribute the decoding resources. It's just a way to offload the ASR processing to a different process or machine. The TCP/IP is just a method for IPC here.

[hhadian]

This looks good to me.

[danpovey]

Thanks a lot, guys!