Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow sending 50m+ files to telegram channels using MTProto #37

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 6 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,13 @@ Pulls multiple podcast feeds (RSS) and republishes as a common feed, properly so
| feed | FM_FEED | | single feed, overrides config |
| update-interval | UPDATE_INTERVAL | `1m` | update interval, overrides config |
| telegram_chan | TELEGRAM_CHAN | | single telegram channel, overrides config |
| telegram_server | TELEGRAM_SERVER | `https://api.telegram.org` | telegram bot api server |
| telegram_server | TELEGRAM_SERVER | `149.154.167.50:443` | telegram API server from https://my.telegram.org/apps |
| telegram_token | TELEGRAM_TOKEN | | telegram token |
| telegram_timeout | TELEGRAM_TIMEOUT | `1m` | telegram timeout |
| telegram_public_keys | TELEGRAM_PUBLIC_KEYS | `/srv/etc/tg_public_keys.pem` | telegram public keys file location |
| telegram_session_file | TELEGRAM_SESSION_FILE | `/srv/var/session.json` | telegram sessions file location |
| telegram_app_id | TELEGRAM_APP_ID | | telegram app ID from https://my.telegram.org/apps |
| telegram_app_hash | TELEGRAM_APP_HASH | | telegram hash from https://my.telegram.org/apps |
| telegram_only_message| TELEGRAM_ONLY_MESSAGE | | don't send audio files to telegram, only text |
| consumer-key | TWI_CONSUMER_KEY | | twitter consumer key |
| consumer-secret | TWI_CONSUMER_SECRET | | twitter consumer secret |
| access-token | TWI_ACCESS_TOKEN | | twitter access token |
Expand All @@ -35,11 +39,3 @@ Pulls multiple podcast feeds (RSS) and republishes as a common feed, properly so
## Web UI

Web UI shows a list of items from generated RSS. It is available on `/feed/{name}`

## Telegram notifications

By default, (with only `TELEGRAM_TOKEN` provided) Telegram notifications will be sent using standard Bot API which has a limit of [50Mb](https://core.telegram.org/bots/api#sending-files) for audio file upload.

You can provide `TELEGRAM_API_ID` and `TELEGRAM_API_HASH` (from [here](https://my.telegram.org/apps)) to `telegram-bot-api` service in docker-compose.yml and uncomment `TELEGRAM_SERVER` for `feed-master`, then it would use the local bot api server to raise audio file upload limit from 50Mb [to 2000Mb](https://core.telegram.org/bots/api#using-a-local-bot-api-server).

To use local telegram bot api server, use `docker-compose up -d` command instead of `docker-compose up -d feed-master`.
48 changes: 48 additions & 0 deletions _example/etc/tg_public_keys.pem
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEAwVACPi9w23mF3tBkdZz+zwrzKOaaQdr01vAbU4E1pvkfj4sqDsm6
lyDONS789sVoD/xCS9Y0hkkC3gtL1tSfTlgCMOOul9lcixlEKzwKENj1Yz/s7daS
an9tqw3bfUV/nqgbhGX81v/+7RFAEd+RwFnK7a+XYl9sluzHRyVVaTTveB2GazTw
Efzk2DWgkBluml8OREmvfraX3bkHZJTKX4EQSjBbbdJ2ZXIsRrYOXfaA+xayEGB+
8hdlLmAjbCVfaigxX0CDqWeR1yFL9kwd9P0NsZRPsmoqVwMbMu7mStFai6aIhc3n
Slv8kg9qv1m6XHVQY3PnEw+QQtqSIXklHwIDAQAB
-----END RSA PUBLIC KEY-----

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAruw2yP/BCcsJliRoW5eB
VBVle9dtjJw+OYED160Wybum9SXtBBLXriwt4rROd9csv0t0OHCaTmRqBcQ0J8fx
hN6/cpR1GWgOZRUAiQxoMnlt0R93LCX/j1dnVa/gVbCjdSxpbrfY2g2L4frzjJvd
l84Kd9ORYjDEAyFnEA7dD556OptgLQQ2e2iVNq8NZLYTzLp5YpOdO1doK+ttrltg
gTCy5SrKeLoCPPbOgGsdxJxyz5KKcZnSLj16yE5HvJQn0CNpRdENvRUXe6tBP78O
39oJ8BTHp9oIjd6XWXAsp2CvK45Ol8wFXGF710w9lwCGNbmNxNYhtIkdqfsEcwR5
JwIDAQAB
-----END PUBLIC KEY-----

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvfLHfYH2r9R70w8prHbl
Wt/nDkh+XkgpflqQVcnAfSuTtO05lNPspQmL8Y2XjVT4t8cT6xAkdgfmmvnvRPOO
KPi0OfJXoRVylFzAQG/j83u5K3kRLbae7fLccVhKZhY46lvsueI1hQdLgNV9n1cQ
3TDS2pQOCtovG4eDl9wacrXOJTG2990VjgnIKNA0UMoP+KF03qzryqIt3oTvZq03
DyWdGK+AZjgBLaDKSnC6qD2cFY81UryRWOab8zKkWAnhw2kFpcqhI0jdV5QaSCEx
vnsjVaX0Y1N0870931/5Jb9ICe4nweZ9kSDF/gip3kWLG0o8XQpChDfyvsqB9OLV
/wIDAQAB
-----END PUBLIC KEY-----

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ditzm+mPND6xkhzwFI
z6J/968CtkcSE/7Z2qAJiXbmZ3UDJPGrzqTDHkO30R8VeRM/Kz2f4nR05GIFiITl
4bEjvpy7xqRDspJcCFIOcyXm8abVDhF+th6knSU0yLtNKuQVP6voMrnt9MV1X92L
GZQLgdHZbPQz0Z5qIpaKhdyA8DEvWWvSUwwc+yi1/gGaybwlzZwqXYoPOhwMebzK
Uk0xW14htcJrRrq+PXXQbRzTMynseCoPIoke0dtCodbA3qQxQovE16q9zz4Otv2k
4j63cz53J+mhkVWAeWxVGI0lltJmWtEYK6er8VqqWot3nqmWMXogrgRLggv/Nbbo
oQIDAQAB
-----END PUBLIC KEY-----

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvmpxVY7ld/8DAjz6F6q0
5shjg8/4p6047bn6/m8yPy1RBsvIyvuDuGnP/RzPEhzXQ9UJ5Ynmh2XJZgHoE9xb
nfxL5BXHplJhMtADXKM9bWB11PU1Eioc3+AXBB8QiNFBn2XI5UkO5hPhbb9mJpjA
9Uhw8EdfqJP8QetVsI/xrCEbwEXe0xvifRLJbY08/Gp66KpQvy7g8w7VB8wlgePe
xW3pT13Ap6vuC+mQuJPyiHvSxjEKHgqePji9NP3tJUFQjcECqcm0yV7/2d0t/pbC
m+ZH1sadZspQCEPPrtbkQBlvHb4OLiIWPGHKSMeRFvp3IWcmdJqXahxLCUS1Eh6M
AQIDAQAB
-----END PUBLIC KEY-----
45 changes: 32 additions & 13 deletions app/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"io/ioutil"
"os"
"strings"
"sync"
"text/template"
"time"

Expand All @@ -27,14 +28,18 @@ type options struct {
TelegramChannel string `long:"telegram_chan" env:"TELEGRAM_CHAN" description:"single telegram channel, overrides config"`
UpdateInterval time.Duration `long:"update-interval" env:"UPDATE_INTERVAL" default:"1m" description:"update interval, overrides config"`

TelegramServer string `long:"telegram_server" env:"TELEGRAM_SERVER" default:"https://api.telegram.org" description:"telegram bot api server"`
TelegramToken string `long:"telegram_token" env:"TELEGRAM_TOKEN" description:"telegram token"`
TelegramTimeout time.Duration `long:"telegram_timeout" env:"TELEGRAM_TIMEOUT" default:"1m" description:"telegram timeout"`
TwitterConsumerKey string `long:"consumer-key" env:"TWI_CONSUMER_KEY" description:"twitter consumer key"`
TwitterConsumerSecret string `long:"consumer-secret" env:"TWI_CONSUMER_SECRET" description:"twitter consumer secret"`
TwitterAccessToken string `long:"access-token" env:"TWI_ACCESS_TOKEN" description:"twitter access token"`
TwitterAccessSecret string `long:"access-secret" env:"TWI_ACCESS_SECRET" description:"twitter access secret"`
TwitterTemplate string `long:"template" env:"TEMPLATE" default:"{{.Title}} - {{.Link}}" description:"twitter message template"`
TelegramServer string `long:"telegram_server" env:"TELEGRAM_SERVER" default:"149.154.167.50:443" description:"telegram API server from https://my.telegram.org/apps"`
TelegramToken string `long:"telegram_token" env:"TELEGRAM_TOKEN" description:"telegram token"`
TelegramPublicKeys string `long:"telegram_public_keys" env:"TELEGRAM_PUBLIC_KEYS" default:"/srv/etc/tg_public_keys.pem" description:"telegram public keys file location"`
TelegramSessionFile string `long:"telegram_session_file" env:"TELEGRAM_SESSION_FILE" default:"/srv/var/session.json" description:"telegram sessions file location"`
TelegramAppID int `long:"telegram_app_id" env:"TELEGRAM_APP_ID" description:"telegram app ID from https://my.telegram.org/apps"`
TelegramAppHash string `long:"telegram_app_hash" env:"TELEGRAM_APP_HASH" description:"telegram hash from https://my.telegram.org/apps"`
TelegramOnlyMessage bool `long:"telegram_only_message" env:"TELEGRAM_ONLY_MESSAGE" description:"don't send audio files to telegram, only text"`
TwitterConsumerKey string `long:"consumer-key" env:"TWI_CONSUMER_KEY" description:"twitter consumer key"`
TwitterConsumerSecret string `long:"consumer-secret" env:"TWI_CONSUMER_SECRET" description:"twitter consumer secret"`
TwitterAccessToken string `long:"access-token" env:"TWI_ACCESS_TOKEN" description:"twitter access token"`
TwitterAccessSecret string `long:"access-secret" env:"TWI_ACCESS_SECRET" description:"twitter access secret"`
TwitterTemplate string `long:"template" env:"TEMPLATE" default:"{{.Title}} - {{.Link}}" description:"twitter message template"`

Dbg bool `long:"dbg" env:"DEBUG" description:"debug mode"`
}
Expand Down Expand Up @@ -67,12 +72,12 @@ func main() {
log.Fatalf("[ERROR] can't open db %s, %v", opts.DB, err)
}

telegramNotif, err := proc.NewTelegramClient(opts.TelegramToken, opts.TelegramServer, opts.TelegramTimeout)
if err != nil {
log.Fatalf("[ERROR] failed to initialize telegram client %s, %v", opts.TelegramToken, err)
p := &proc.Processor{
Conf: conf,
Store: db,
TwitterNotif: makeTwitter(opts),
TelegramNotif: makeTelegram(opts),
}

p := &proc.Processor{Conf: conf, Store: db, TelegramNotif: telegramNotif, TwitterNotif: makeTwitter(opts)}
go p.Do()

server := api.Server{
Expand All @@ -99,6 +104,20 @@ func singleFeedConf(feedURL, channel string, updateInterval time.Duration) *proc
return &conf
}

func makeTelegram(opts options) proc.TelegramNotif {
return &proc.TelegramClient{
Token: opts.TelegramToken,
Server: opts.TelegramServer,
PublicKeysFile: opts.TelegramPublicKeys,
SessionFile: opts.TelegramSessionFile,
AppID: opts.TelegramAppID,
AppHash: opts.TelegramAppHash,
OnlyMessage: opts.TelegramOnlyMessage,
Lock: &sync.Mutex{},
Version: revision,
}
}

func makeTwitter(opts options) *proc.TwitterClient {
twitterFmtFn := func(item feed.Item) string {
b1 := bytes.Buffer{}
Expand Down
26 changes: 26 additions & 0 deletions app/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import (
"time"

"github.com/stretchr/testify/assert"

"github.com/umputun/feed-master/app/proc"
)

func TestLoadConfig(t *testing.T) {
Expand Down Expand Up @@ -109,3 +111,27 @@ func TestMakeTwitter(t *testing.T) {
assert.Equal(t, client.AccessToken, "c")
assert.Equal(t, client.AccessSecret, "d")
}

func TestMakeTelegram(t *testing.T) {
opts := options{
TelegramToken: "a",
TelegramServer: "b",
TelegramPublicKeys: "c",
TelegramSessionFile: "d",
TelegramAppID: 42,
TelegramAppHash: "e",
TelegramOnlyMessage: true,
}

client := makeTelegram(opts)
telegramClient := client.(*proc.TelegramClient)
assert.Equal(t, telegramClient.Token, "a")
assert.Equal(t, telegramClient.Server, "b")
assert.Equal(t, telegramClient.PublicKeysFile, "c")
assert.Equal(t, telegramClient.SessionFile, "d")
assert.Equal(t, telegramClient.AppID, 42)
assert.Equal(t, telegramClient.AppHash, "e")
assert.True(t, telegramClient.OnlyMessage)
assert.NotNil(t, telegramClient.Lock)
assert.NotEmpty(t, telegramClient.Version)
}
Loading