Web app setup creates duplicate channels

[bennettpeter]

Reported by glenb:

on the m3u import, i am using tvheadend's exported playlist based on a tag in tvheadend to generate a playlist. if i add a single channel and then reimport the list. it makes all new channels for the whole list resulting in a large amount of duplicates. i assume it is something to do with the format and tags in the playlist. it looks like this:

its static the channels come out the same everytime. is there some way i could fix it so that the import would ignore existing channels already in myth with the same stream url ?

i can add a channel manually in the sql which is quicker by making a channel in channel editor and then adding a channel in iptvchannel table and that works.

the m3u import i am talking about is the input connections when a playlist url is specified as a capture card on freebox/iptv recorder. it is the same i assume as what was in mythtv-setup application (deprecated). when i had a manually made list it would not bring in repeated channels. when i have it automated to tvheadend, it does not. i thought maybe you knew. but i choose scan, and then choose m3u import, as type and it discovers the channels and adds them. i'm sure its coded on the 'extinf' parameters and the code does not see the entry as a duplicate probably on first parameter. anyway i'll narrow it down more. i have a way using sql commands to save time overall so i'll continue to do that.

[kmdewaal]

It would be useful to attach the m3u file to this ticket, so that we can reproduce the problem easiily.

[bennettpeter]

I have asked Glen if it I can attach his URL or his file to the ticket, since it is a private URL.

[bennettpeter]

URL of m3u file:
http://glenb.net:9981/playlist/tag/free

[kmdewaal]

I downloaded the m3u file and then used the local m3u file for testing.

I can reproduce the issue with the classic mythtv-setup; each time a scan is done the same 26 channels are appended to the channel list. So doing it twice gives you 52 channels, channel 27 the same as channel 1.

I do not really know if this a bug or intended behavior a.k.a. feature.

It would make sense however to filter out duplicate channels as this is also what is done with conventional DVB and ATSC channel scanning.

An obvious workaround may be to delete all channels before importing/scanning them again but there may be drawbacks to that such as the loss of manually entered additional channel data.

[bennettpeter]

In that case, the web app setup is consistent with mythtv-setup.

[kmdewaal]

A check on uniqueness of channel name combined with URL on insert should be sufficient I think.
If I fix this in mythtv-setup then most likely the web interface would also be correct if it still uses the same underlying code.

[kmdewaal]

I am rather deep into IPTV at the moment because of #936 so I think I can fix the scanning as well while I am at it. It would be great if @bennettpeter can then fix what is needed to make it work in the web app.

[bennettpeter]

The web app calls ChannelScanner::ImportVBox in libs/libmythtv/channelscan/channelscanner.cpp to process the file. That is the same code used by mythtv-setup, so any fix made there would fix it for both.

[kmdewaal]

The new channel numbers are allocated in iptvchannelfetcher.cpp only for the "github.com" version of the M3U parser.

This code has been added in commit a474d52 by @paul-h
"IPTV: add support for playlists from https://github.com/iptv-org/iptv/"

When the M3U file does have channel numbers then they are used; only when a channel does not have a channel number there is one automatically generated and that is then the first unused, i.e. existing highest channel number in the video source plus one.

I think the way forward is to do a lookup in the database for the channel name plus the video source; if that matches then the channel number of that channel in the database is used.

It is possible to also match on video source plus channel name plus URL to find the channel in the database. However, as I understand it, login credentials and subscription information can be encoded in the URL and that means that when you for instance extend your subscription a rescan will still result in new channels.
So possibly it is best to match on video souce plus channel name only.

[kmdewaal]

The bug fix in the above commit does not solve the problem described in this issue but it does enable a workaround; when all channels are deleted before doing the scan then the first channel scanned gets number 1, etc.

[angelaschmid]

One remark on:

best to match on video souce plus channel name only

A user (myself included) can have multiple Capture Cards (perhaps a better term would be "Capture Sources"), such as DVB and various IPTV sources, but typically has just one Video Source (perhaps a better term would be "Listing Source"), having a single guide grabber.

Channel scanning is performed at the Input Connections stage, where Capture Cards and Video Sources are linked. During this process, only the sourceid is stored in the channels table. However, what’s missing is an "input connection ID" to reference the specific Capture Card (URL playlist), which would uniquely identify the channels for updates.

Currently, to work around this limitation, users would need to configure a separate Video Source for each Capture Card and perform multiple mythfilldatabase runs. When not setup like this, the user might loose channels from other Capture Cards (playlists or even DVB) during a re-scan.

("input connection ID": this approach may not fully align with a DVB setup, as DVB can contain multiple Input Connections, combining multiple Capture Cards with a single Video Source, requiring only one scan on one of the cards. It would still fit, when the scan is allways processed on the same capture card, or otherwise taken care of.)

[kmdewaal]

Currently, to work around this limitation, users would need to configure a separate Video Source for each Capture Card and perform multiple mythfilldatabase runs. When not setup like this, the user might loose channels from other Capture Cards (playlists or even DVB) during a re-scan.

This is not a workaround but this is exactly how it all should work.
For DVB it is absolutely essential that you have different video sources, as there are different physical cables and usually also physically different capture cards.
For IPTV there is of course no physical difference in capture cards but then it is also not a good idea to combine channels from different channel lists, as there can be conflicts in channel names and in channel numbers.

I think it is a design flaw that the URL is part of capture card; it should be a part of the video source.
And if the idea is to support more than one URL per video source the video source should then support a list of URLs.

However, you mention a very valid point and that is that you need to do multiple mythfilldatabase runs, one for each video source. My understanding of mythfilldatabase is limited, but the idea is that it adds guide data to a channel based on the XMLTVID of that channel. Why should it not do the channels of all video sources, or of a collection of video sources, in one pass?

[kmdewaal]

@bennettpeter and glenb, this issue should be fixed now. Can you please test?