Empty reply from server after setup

[funbotan]

General Description

Tried following the installation setup in README (both client and server) and didn't succeed in getting it working: the server returns an empty response. This is most likely not a bug, but rather me missing something due to the not-so-comprehensive documentation and no help from Stackoverflow and such.

How To Reproduce?

1. Follow the setup instructions in README

Operating System

Ubuntu 20.04
nginx/1.18.0
Nextcloud 23.0.0

Systemd service

[Unit]
Description=talked
Requires=network.target
After=network.target

[Service]
WorkingDirectory=/opt/talked
User=talked
Group=talked
RuntimeDirectory=talked
Environment=TALKED_CONFIG_PATH=/opt/talked/config.toml
ProtectSystem=full
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
PrivateTmp=yes
NoNewPrivileges=yes
SyslogIdentifier=talked
StandardOutput=syslog
StandardError=syslog
ExecStart=/opt/talked/talked/bin/uwsgi --socket 127.0.0.1:3031 --logto /opt/talked/uwsgi.log --master --manage-script-name --mount /=talked.main:app
Restart=always

[Install]
WantedBy=multi-user.target

Nginx config

location /talked {
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:3031;
        auth_basic "Nothing to see here, move along";
        auth_basic_user_file /opt/talked/.htpasswd;
    }

Logs

The content of uwsgi log file after a couple of failed requests:

*** Starting uWSGI 2.0.20 (64bit) on [Sun Feb 20 16:17:51 2022] ***
compiled with version: 9.3.0 on 19 February 2022 14:31:49
os: Linux-5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022
nodename: nextcloud
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /opt/talked
detected binary path: /opt/talked/talked/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 15188
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:3031 fd 3
Python version: 3.8.10 (default, Nov 26 2021, 20:14:08)  [GCC 9.3.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x557262ef09a0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145840 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
mounting talked.main:app on /
WSGI app 0 (mountpoint='/') ready in 0 seconds on interpreter 0x557262ef09a0 pid: 13259 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 13259)
spawned uWSGI worker 1 (pid: 13278, cores: 1)
[pid: 13278|app: 0|req: 1/1] 23.88.50.247 () {38 vars in 430 bytes} [Sun Feb 20 16:18:14 2022] GET /talked => generated 232 bytes in 12 msecs (HTTP/1.1 404) 2 headers in 87 bytes (1 switches on core 0)
[pid: 13278|app: 0|req: 2/2] 23.88.50.247 () {38 vars in 430 bytes} [Sun Feb 20 16:19:07 2022] GET /talked => generated 232 bytes in 1 msecs (HTTP/1.1 404) 2 headers in 87 bytes (1 switches on core 0)
invalid request block size: 21573 (max 4096)...skip

Response

>> curl 127.0.0.1:3031
curl: (52) Empty reply from server

>> sudo -u www-data php occ talked:record 'xxxxxxxx' 'status'
An unhandled exception has been thrown:
TypeError: Return value of OCA\Talked\Command\Record::sendPostRequest() must be of the type string, null returned in /var/www/nextcloud/apps/talked/lib/Command/Record.php:207
Stack trace:
#0 /var/www/nextcloud/apps/talked/lib/Command/Record.php(109): OCA\Talked\Command\Record->sendPostRequest()
#1 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Talked\Command\Record->execute()
#2 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1009): Symfony\Component\Console\Command\Command->run()
#3 /var/www/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#4 /var/www/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#5 /var/www/nextcloud/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run()
#6 /var/www/nextcloud/console.php(99): OC\Console\Application->run()
#7 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')

Talked Version

Server: 0.3.1, installed through PIP according to the instructions in README
Client: 0.3.0, installed through Nextcloud apps

[mwalbeck]

Hi, thank you for the very detailed bug issue.

When you did the test through the Nextcloud instance did you connect to the Talked instance through Nginx or directly?

If directly then I think that is where the problem lies. When starting uwsgi with the --socket argument then uwsgi doesn't understand HTTP, so we therefore need to go through Nginx which handles translating the HTTP request to a uwsgi request.

If you would like to connect directly to Talked then you can change the ExecStart line to the following:

ExecStart=/opt/talked/talked/bin/uwsgi --http-socket 127.0.0.1:3031 --logto /opt/talked/uwsgi.log --master --manage-script-name --mount /=talked.main:app

When using the --http-socket argument uswgi is expecting HTTP requests instead of the default uwsgi requests and one can therefore communicate directly with Talked using curl.

[funbotan]

Thank you for the prompt reply.
It seems that when I change --socket to --http-socket in the current configuration, the HTTP error code changes from 404 to 405. In general, I think I've already tried all combinations of parameters (UNIX socket, HTTP socket, HTTP through nginx) and none of them work, albeit for different reasons.

[mwalbeck]

Ohh, now that I think about it, you might have to use --http instead of --http-socket for you to be able to use cURL directly without having to pass some specific arguments to curl.

Otherwise, if you can provide any more details about the particular errors you're running into, I'd be happy to look further into it.

[funbotan]

Ok, I've switched to --http and here is what I get now.

>> curl 127.0.0.1:3031
{"message":"version: 0.3.1"}

>> curl 127.0.0.1:3031/status
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

>> sudo -u www-data php occ talked:record 'xxxxxxxx' 'status'
An error occured while running the command. Please try again or contact an administrator.

And there are no new lines in uwsgi.log.

[mwalbeck]

This looks for the most part correct. The reason why the /status endpoint is throwing a HTTP 405 is because only POST requests are supported.

If you look in the Nextcloud log there should be a more detailed error log for why running the occ talked:record command isn't working.

[funbotan]

Here is what the log shows:
Error | talked | cURL Error (28): Operation timed out after 30001 milliseconds with 0 bytes received

[mwalbeck]

What have you input for the URL in Nextcloud, if you're willing to share that?

And are there any messages from Talked in /var/log/syslog?

[funbotan]

https://nc.mydomain.org/talked (domain redacted)
/var/log/syslog only has lines about starting and stopping the service, nothing regarding its functioning.

[mwalbeck]

Try and switch the log level for Talked to info and see if that changes anything. It kind of seems like there might be some sort of connection issue between Nextcloud and Talked.

[funbotan]

Hello again. I've set the log level in the systemd service to debug and still haven't noticed anything suspicious that is directly related to Talked.

[mwalbeck]

To get more detailed logging from Talked itself you have to change the log level in the config.toml file, setting it to info should be good enough.

[funbotan]

Oh, I see, sorry for the misunderstanding. Still, the result after the change is the same. This, I think, confirms the idea that the problem is with the connection and not the service itself: a query sent via nginx just doesn't reach uwsgi (there isn't a corresponding line in the uwsgi.log, unlike when I query the socket directly).

[mwalbeck]

One change I just realised you need to make in your setup if you continue to use the --http-socket parameter, is the uwsgi_pass option specified in the nginx block for talked, that should be changed to proxy_pass and you should also include the proxy_params file instead of the uwsgi_params, or you can just use the following parameters, they should work:

proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

[funbotan]

Yep, that solved the current problem. Now /recording status works in chat, but /recording start returns "Failed to load the Talk room".

[mwalbeck]

Great! Is the talk room configured to allow guests?

[funbotan]

Oops, no 😅
But now that it is, the error changed to "Failed to initiate call" (when a call is ongoing).

[mwalbeck]

Mhm, interesting, what is your talk setup? Barebones, with a TURN server or with the Talk HPB backend?

[funbotan]

With TURN

[mwalbeck]

Mhm, so the error indicates that Talked is having trouble connecting to the call, and more specifically the other participants in the call since it's a TURN setup. To just rule out the low-hanging fruit, how long does it usually take for you to connect to a call? There is currently a 10 second timeout on it.

You can also try and look at the turn logs to see if a connection is being made from server talked is running on.

[funbotan]

Well, I'd say it doesn't take longer than 5 seconds most of the time. The coTURN log doesn't have any lines regarding connections, though.

[mwalbeck]

Thanks for the info, when I have some stuff I need to do some more testing in order to better help. I'm not entire sure what issue exactly there might be with Talked in a TURN only setup, as I've only tested and used Talked with the HPB.

It's reasonable to expect it to work, so if there is something I can change in the code to make it work better, I would be nice to have that implemented.

[mwalbeck]

The issue you experienced may have been caused by something completely different from what I thought. I have created a new release, v0.3.2, that hopefully resolves your issue, so give it a try and let me know.

[funbotan]

No, unfortunately, the result is the same.