Terrible refresh rate when a picture is viewed

[philayres]

I'm running Xrdp on a Centos 7 server (an EC2 instance with 30GB memory, 4 vCPUs and a good network). I have Xorgxrdp installed too.

When viewing plain backgrounds, such as the desktop background which is mostly a block colour, things draw over it reasonably quickly. A webpage with a white background like Github will load and scroll reasonably well (choppy, but bearable). But as soon as a picture (say a person's face) is to be drawn on the page, it takes up to 10 seconds to draw the whole webpage.

Similarly, if a desktop backgound has an image rather than block colour, opening even a small terminal window over it is horribly slow (many seconds).

I can understand that refresh rates aren't 60fps, but 6 frames a minute is appalling. It also seems strange that drawing a block of colour (easily compressed) over a textured image is slow. Could this be a caching issue?

Running htop in a full screen terminal for example, and I visibly see the page refresh from the bottom of the screen to the top over about 2-3 seconds.

There is nothing of any interest in xrpd.log or Xorg.0.log

The versions installed from the Centos EPEL library are:

xrdp.x86_64 1:0.9.13-2.el7
xorgxrdp.x86_64 0.2.13-1.el7

My xrdp.ini file:

[Globals]
; xrdp.ini file version number
ini_version=1

; fork a new process for each incoming connection
fork=true

; ports to listen on, number alone means listen on all interfaces
; 0.0.0.0 or :: if ipv6 is configured
; space between multiple occurrences
;
; Examples:
;   port=3389
;   port=unix://./tmp/xrdp.socket
;   port=tcp://.:3389                           127.0.0.1:3389
;   port=tcp://:3389                            *:3389
;   port=tcp://<any ipv4 format addr>:3389      192.168.1.1:3389
;   port=tcp6://.:3389                          ::1:3389
;   port=tcp6://:3389                           *:3389
;   port=tcp6://{<any ipv6 format addr>}:3389   {FC00:0:0:0:0:0:0:1}:3389
;   port=vsock://<cid>:<port>
port=3389

; 'port' above should be connected to with vsock instead of tcp
; use this only with number alone in port above
; prefer use vsock://<cid>:<port> above
use_vsock=false

; regulate if the listening socket use socket option tcp_nodelay
; no buffering will be performed in the TCP stack
tcp_nodelay=true

; regulate if the listening socket use socket option keepalive
; if the network connection disappear without close messages the connection will be closed
tcp_keepalive=true

; set tcp send/recv buffer (for experts)
#tcp_send_buffer_bytes=32768
#tcp_recv_buffer_bytes=32768

; security layer can be 'tls', 'rdp' or 'negotiate'
; for client compatible layer
security_layer=tls

; minimum security level allowed for client for classic RDP encryption
; use tls_ciphers to configure TLS encryption
; can be 'none', 'low', 'medium', 'high', 'fips'
crypt_level=high

; X.509 certificate and private key
; openssl req -x509 -newkey rsa:2048 -nodes -keyout key.pem -out cert.pem -days 365
certificate=/etc/pki/tls/certs/server.crt
key_file=/etc/pki/tls/certs/server.key

; set SSL protocols
; can be comma separated list of 'SSLv3', 'TLSv1', 'TLSv1.1', 'TLSv1.2', 'TLSv1.3'
ssl_protocols=TLSv1.2
#, TLSv1.3
; set TLS cipher suites
#tls_ciphers=HIGH

; Section name to use for automatic login if the client sends username
; and password. If empty, the domain name sent by the client is used.
; If empty and no domain name is given, the first suitable section in
; this file will be used.
autorun=

allow_channels=true
allow_multimon=false
bitmap_cache=true
bitmap_compression=true
bulk_compression=true
#hidelogwindow=true
max_bpp=24
new_cursors=true
; fastpath - can be 'input', 'output', 'both', 'none'
use_fastpath=both
; when true, userid/password *must* be passed on cmd line
#require_credentials=true
; You can set the PAM error text in a gateway setup (MAX 256 chars)
#pamerrortxt=change your password according to policy at http://url

;
; colors used by windows in RGB format
;
blue=009cb5
grey=dedede
#black=000000
#dark_grey=808080
#blue=08246b
#dark_blue=08246b
#white=ffffff
#red=ff0000
#green=00ff00
#background=626c72

;
; configure login screen
;

; Login Screen Window Title
#ls_title=My Login Title

; top level window background color in RGB format
ls_top_window_bg_color=009cb5

; width and height of login screen
ls_width=350
ls_height=430

; login screen background color in RGB format
ls_bg_color=dedede

; optional background image filename (bmp format).
#ls_background_image=

; logo
; full path to bmp-file or file in shared folder
ls_logo_filename=
ls_logo_x_pos=55
ls_logo_y_pos=50

; for positioning labels such as username, password etc
ls_label_x_pos=30
ls_label_width=65

; for positioning text and combo boxes next to above labels
ls_input_x_pos=110
ls_input_width=210

; y pos for first label and combo box
ls_input_y_pos=220

; OK button
ls_btn_ok_x_pos=142
ls_btn_ok_y_pos=370
ls_btn_ok_width=85
ls_btn_ok_height=30

; Cancel button
ls_btn_cancel_x_pos=237
ls_btn_cancel_y_pos=370
ls_btn_cancel_width=85
ls_btn_cancel_height=30

[Logging]
LogFile=xrdp.log
LogLevel=WARNING
EnableSyslog=true
SyslogLevel=WARNING
; LogLevel and SysLogLevel could by any of: core, error, warning, info or debug

[Channels]
; Channel names not listed here will be blocked by XRDP.
; You can block any channel by setting its value to false.
; IMPORTANT! All channels are not supported in all use
; cases even if you set all values to true.
; You can override these settings on each session type
; These settings are only used if allow_channels=true
rdpdr=true
rdpsnd=false
drdynvc=true
cliprdr=true
rail=true
xrdpvr=true
tcutils=true

; for debugging xrdp, in section xrdp1, change port=-1 to this:
#port=/tmp/.xrdp/xrdp_display_10

; for debugging xrdp, add following line to section xrdp1
#chansrvport=/tmp/.xrdp/xrdp_chansrv_socket_7210


;
; Session types
;

; Some session types such as Xorg, X11rdp and Xvnc start a display server.
; Startup command-line parameters for the display server are configured
; in sesman.ini. See and configure also sesman.ini.
[Xorg]
name=Xorg
lib=libxup.so
username=ask
password=ask
ip=127.0.0.1
port=-1
code=20

#[Xvnc]
#name=Xvnc
#lib=libvnc.so
#username=ask
#password=ask
#ip=127.0.0.1
#port=-1

#xserverbpp=24
#delay_ms=2000

#[vnc-any]
#name=vnc-any
#lib=libvnc.so
#ip=ask
#port=ask5900
#username=na
#password=ask
#pamusername=asksame
#pampassword=asksame
#pamsessionmng=127.0.0.1
#delay_ms=2000

#[neutrinordp-any]
#name=neutrinordp-any
#lib=libxrdpneutrinordp.so
#ip=ask
#port=ask3389
#username=ask
#password=ask

; You can override the common channel settings for each session type
#channel.rdpdr=true
#channel.rdpsnd=true
#channel.drdynvc=true
#channel.cliprdr=true
#channel.rail=true
#channel.xrdpvr=true

[philayres]

Also, the Xorg/xrdp/xrdp.conf is:

Section "ServerLayout"
    Identifier "X11 Server"
    Screen "Screen (xrdpdev)"
    InputDevice "xrdpMouse" "CorePointer"
    InputDevice "xrdpKeyboard" "CoreKeyboard"
EndSection

Section "ServerFlags"
    Option "DontVTSwitch" "on"
    Option "AutoAddDevices" "off"
EndSection

Section "Module"
    Load "dbe"
    Load "ddc"
    Load "extmod"
    Load "glx"
    Load "int10"
    Load "record"
    Load "vbe"
    Load "xorgxrdp"
    Load "fb"
EndSection

Section "InputDevice"
    Identifier "xrdpKeyboard"
    Driver "xrdpkeyb"
EndSection

Section "InputDevice"
    Identifier "xrdpMouse"
    Driver "xrdpmouse"
EndSection

Section "Monitor"
    Identifier "Monitor"
    Option "DPMS"
    HorizSync 30-80
    VertRefresh 60-75
    ModeLine "1920x1080" 138.500 1920 1968 2000 2080 1080 1083 1088 1111 +hsync -vsync
    ModeLine "1280x720" 74.25 1280 1720 1760 1980 720 725 730 750 +HSync +VSync
    Modeline "1368x768" 72.25 1368 1416 1448 1528 768 771 781 790 +hsync -vsync
    Modeline "1600x900" 119.00 1600 1696 1864 2128 900 901 904 932 -hsync +vsync
EndSection

Section "Device"
    Identifier "Video Card (xrdpdev)"
    Driver "xrdpdev"
    Option "DRMDevice" "/dev/dri/renderD128"
    Option "DRI3" "1"
EndSection

Section "Screen"
    Identifier "Screen (xrdpdev)"
    Device "Video Card (xrdpdev)"
    Monitor "Monitor"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "640x480" "800x600" "1024x768" "1280x720" "1280x1024" "1600x900" "1920x1080"
    EndSubSection
EndSection

[philayres]

To add to this, I attempted removing xorgxrdp, and using tigervnc-server instead. The result is the same (I couldn't perceive any difference in performance), demonstrating that the issue is on the xrdp side of things.

This short screen grab will show how slow response is scrolling a regular website:

https://drive.google.com/file/d/1ZLaWAALs0nY29gAlYdI3EPivd7i8sIbB/view?usp=sharing

You'll notice that once the page gets down to the real pictures the refresh rate really slows down. Presumably the compression fails to handle areas that are not just blocks of a single colour.

What is also concerning is that everything appears to then queue up. If there are several screen refreshes lined up on the web browser, viewing the dock at the side of the screen doesn't show until the browser refreshes have completed, even though it should be instantaneous.

Is this expected?

[matt335672]

Hi @philayres

What you describe could be exacerbated by a high latency link and inadequate TCP buffer sizes.

What are the ping times between your client and server?

The xrdp.ini file has a tunable tcp_send_buffer_bytes (commented out above) which can be used to configure the TCP send buffer size from the server. On my test machine with the vanilla configuration and an established connection:-

ss -m '( sport = :3389 )'
Netid  State   Recv-Q   Send-Q      Local Address:Port     Peer Address:Port    
tcp    ESTAB   0        0           172.19.73.154:3389     172.19.73.10:60268   
	 skmem:(r0,rb131072,t0,tb65536,f0,w0,o0,bl0,d19)

I believe the 'tb' value is approximately twice the actual window size - see tcp(7)

If I set tcp_send_buffer_bytes=262144:-

Netid  State   Recv-Q   Send-Q      Local Address:Port     Peer Address:Port    
tcp    ESTAB   0        0           172.19.73.154:3389     172.19.73.10:60286   
	 skmem:(r0,rb131072,t0,tb425984,f0,w0,o0,bl0,d8)

I don't have the facilities here to try this on a high latency link but it might possibly help the situation.

[philayres]

I really appreciate your help!

I'm sure the ping times are high (I can't measure them due to a VPN and firewall in between), but it is transatlantic. I have tested RDP without the VPN and got the same result, so I have ruled that piece out.

I tried your recommendation for tcp_send_buffer_bytes. I couldn't perceive any difference to be honest. Alongside this I also attempted to boost the TCP window sizes, etc in the kernel, following this: https://www.cyberciti.biz/faq/linux-tcp-tuning/

Sadly no luck. I ran iftop and it looks like the best data rate I get is about 1.5Mbps with a lot of screen refreshing happening to try and stress it.

Any other thoughts on how to get this tuned?

Is there any option to reduce the quality / increase compression (without dropping to 8 bits per pixel)? Are you aware what the compression actually is (I'm assuming a PNG related thing to be lossless, but it would be good to know).

[philayres]

Does xrdp support UDP transmission? I was looking at https://www.rdpsoft.com/blog/remote-desktop-protocol/rdp-udp-transport-azure/ and UDP does sound like it could make a huge difference on high latency links.

Since more of us are working on home broadband connections, rather than nice fast business LAN/WAN links, this could become more important.

[matt335672]

Hi Phil,

ping times would be useful to know at least so there's a chance to model what's going on with different TCP window sizes (see here for an example. Your IT guys should be able to give you a figure if you ask them nicely. You might also be going via a satellite link of course.

Looking at your xrdp.ini, you've already got the compression variables turned on.so I don't think you can improve on that. Check the Experience tab on MSTSC, and check that the client is configured to cache bitmaps.

Looking at the code and the specs, XRDP supports the Microsoft Point-to-Point Compression (MPPC) Protocol described here. It's lossless compression.

XRDP does not yet support UDP, although it has been requested and is documented by Microsoft. This is a huge amount of work however, so don't expect it soon. It looks like it needs a VPN or a corporate network to be useable however as it doesn't support any kind of encryption, or at least I can't see any in the spec.

Off the top of my head there's a couple of things you might try, but these are a bit left-field:-

• Try tunnelling the connection over ssh with compression turned on. This might give you some small compression improvement, but will be susceptible to TCP-over-TCP problems. On the other hand it's relatively quick to try.
• Try using Apache Guacamole to front the RDP connection. XRDP hasn't been developed with WAN links in mind, although I take your point that this is going to become more important. Guacamole very much has, so may well give you an improvement. It's obviously a bit more work to set up though.

Sorry I can't be more helpful than that, but hopefully that will give you something to work with.

[philayres]

Thanks for all the information you sent. That helped me understand things a little better. I'm surprised that bitmaps are RLE "compressed", admittedly within another compression wrapper. That explains why the performance is so bad as soon as an picture appears.

I went through and opened up all the firewalls to get a ping through.

We have a distributed workforce, so the pings will vary. The server is Viriginia, US (AWS east coast).

I am London UK. I got an average ping of 90ms. This had to skip around the VPN, so add a very small amount for that.
Many users are North East US. I VPN'd to a server in the corporate on premise data center, and ping averaged 15ms. But most users currently are on home broadband. So maybe guess at 20-30ms ping.
Other users are spread across the US, on various types of network, so somewhere between the two.

I did realise that I had been testing with a Linux RDP client, so the gradual sliding up of the page refresh was perhaps not normal. But on the regular PC RDC client although it doesn't slide up in the build, the performance is similar. I tried various network settings in the Experience tab, and they all achieved about the same result.

The persistent bitmap cache was set.

Hopefully US users will get a better experience, though some testers so far have complained about lag. They are coming from a VNC based system, so they like the crisp appearance of RDP and are used to lag but feel it might be worse.

I certainly don't expect UDP to be magically developed, though it should definitely be a roadmap item I feel! I hadn't thought about the security of it, so if it happens in the future I'll have to ensure that I can tunnel UDP over the Wireguard VPN.

I'm not permitted to run SSH on the servers, so unfortunately I can't test your SSH compression idea. I took a look at Guac yesterday. I'll have to see if I can test the docker version for easy deployment.

Thanks for your help. I'm open to ideas for TCP Windows if you have any!

[matt335672]

Thanks for the ping values - it's useful data.

Your 1.5Mbps sounds a bit low to me, so I had a quick re-read. I notice you've got tcp_nodelay=true set in the xrdp.ini. This will disable the TCP Nagle algorithm which will almost certainly affect your throughput.

Having it enabled is certainly consistent with your observation that changing the buffer size doesn't affect things.

Can you try turning it off to see if that improves things? Sorry I missed it earlier.

[lcarcamo1526]

I have the same problem as @philayres, when I try to connect using my public ip the refresh rate is quite laggy, it improves a little changing max_bpp=24 to 128 and using xvnc instead Xorg, maybe it will help.

[matt335672]

Hi @lcarcamo1526

Thanks for your input.

For your information, the max_bpp value is used to limit the number of bits per pixel which the client requests as part of its connection negotiation. This is done with a TS_UD_CS_CORE data block. The maximum value here is 32, so you won't see anything significant happening for max_bpp over 32.

The Xvnc backend can work better than the Xorg backend under certain circumstances. It's certainly worth a try if it fits your use cases.

Have you tried any of the things suggested in the conversation above? In particular, I'm quite keen on finding out if setting tcp_nodelay=false in combination with larger buffers works better for some users.

[philayres]

I tried tcp_nodelay=false yesterday. To be honest I don't know that it improved anything (and maybe even made things feel more jumpy in general use such as typing into a document).

If "no delay" means push the data to the networking stack immediately, that would leave the kernel in charge of things, right? So I can see how that would play into the Nagle algorithm. For small screen updates (a character being typed), Nagle would improve things, right? For large screen refreshes, it would presumably make no difference.

I don't see how the combo of settings for these and TCP window sizes interact:

tcp_nodelay=false
tcp_send_buffer_bytes=<large size>
tcp_recv_buffer_bytes=<large size>

Nothing I changed seemed to make much difference to be honest. I'm obviously missing something. Perhaps my sessions weren't shutting down correctly with just systemctl restart xrdp.service. I have seen sesman processes hanging around after this. I'll try again when I have a chance.

I did quickly try the VNC / bpp=128 thing (which I'd seen elsewhere and made no sense to me), and it made no difference. Since that is all happening within the server, I'm not surprised. That just reinforces that the issue I'm seeing is related to slow transmission of the RDP, rather than pulling the virtual screen information.

Any more input would be appreciated. Thanks!

[matt335672]

Thanks for the feedback.

Nagle will clump up lots of small writes into a larger one, so can increase throughput at the expense of delay. If enabling it isn't helping your situation that's useful to know.

This might sound strange, but you could try wiggling the mouse about while scrolling. This (I think) will increase the number of client to server messages and give the client more chances to ACK data. I'm not suggesting this as a solution by any means - it might help us understand what's going on if there's a difference.

[philayres]

I tried the mouse wiggle. No change.

Something that seems strange... rendering a patterned picture and I visibly see the refresh from bottom to top of the page (per my original screen grab). Have a mostly blank terminal window then position over that picture and there is a long delay until the refresh, then it happens really quickly. I can see that writing a pattern might be slow (RLE and all that) but I'm surprised that writing over the top of a pattern is slow, unless it is doing some kind of difference calculation from what is visible to what should be in place.

This screen grab shows it: https://drive.google.com/file/d/1SMk4p1znEP-lPIheQbxe3HdpAOyU7yv4/view?usp=sharing
Basically I click to bring the browser over the terminal (running iftop), and as expected it refreshes up the page. I scroll once and it refreshes up the page. Then I click to minize the browser as soon as my mouse goes back the bottom of the screen, but you see the delay before any refresh, then it is almost instant.

Weird!

[matt335672]

I agree - 'weird' is a good description.

The next obvious thing (I think) is to separate out XRDP and the WAN link to see what happens. That at least lets us see whether we're dealing with a software issue or something really odd regarding the network.

• Did you get a chance to look at the Docker Guacamole image?
• Have you got a port spare through the firewalls we could use to connect a VNC client directory to Xvnc? It's a bit fiddly to set up as XRDP sets a strange password on the VNC session but it's possible. Let me know if you want a hand with that.

[philayres]

I didn't get a chance to play with Guac. It sounds like a big chunk of infrastructure, so I need to review it a little more.

I can probably set up a VNC client. Does this mean switching over to Xvnc in the xrdp.ini? I still have the vncserver installed so hopefully it shouldn't be too much of a challenge. Let me know what you are thinking.

[matt335672]

Process is as follows. I haven't tested this, but it should be more or-less-there.

• Log in using an RDP client as normal.
• Use the DISPLAY value to work out the firewall port to open. This is 5900+$DISPLAY, so for DISPLAY=:10, you'll open port 5910.

The VNC password is auto-generated by XRDP to prevent drive-by logins. You'll need to set it to something you know to connect to the VNC server.

First locate the password file which is being used by the VNC server. You can either do this with ps-ef or by looking in ~/.vnc where the file will be called something along the lines of sesman_passwd-$USER@$HOSTNAME:$DISPLAY. I'll call this $VNCPASSWDFILE below.

To change the VNC passwd:-

$ cd ~/.vnc
$ cp  $VNCPASSWDFILE vnc.sav
$ vncpasswd $VNCPASSWDFILE
<enter a password you know>

You should then be able to connect direct to Xvnc from a client on a remote machine using the password you've specified.

If you need to reconnect over RDP to the same session for comparison:-

$ cd ~/.vnc
$ mv vnc.sav $VNCPASSWDFILE

I hope I've explained that clearly enough. Let me know if not.

[philayres]

I'll try this early next week and get back to you. Thanks for the info.

[lcarcamo1526]

@philayres did you found any solution?

[philayres]

I didn't try the suggestions for VNC yet. But it is almost certainly related to latency, since most of my users are not complaining and are much more local to the server. I hope to try the suggested VNC test soon.

…

On Tue, Aug 11, 2020 at 10:05 PM Luis Carcamo ***@***.***> wrote: @philayres <https://github.com/philayres> did you found any solution? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1625 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACH7WSOPINKIQA6TBGHT6LSAGXDHANCNFSM4OME2Y3A> .

-- *Phil Ayres - Consected LLC* Email: *phil.ayres@consected.com <phil.ayres@consected.com>* Phone and SMS: *+1-617-794-2330* | *+44 (0)7393 892 699* Google Duo: search *+1 617-794-2330*

[unihon]

It is very slow, I hope it can become a better project

[gueug78400]

Hi team,

Firstly, a great many thanks for this project and the time you spent on it!!!!
I was really happy to see that I could use RDP with Gnome, XFCE is not my thing.

I encounter refreshing issues when I add a background with high definition images, when I use videos in Firefox, but it seems the lighter the window is the less I got those refreshing issues. To me compression or internal communication between two services would be the root cause. UDP would perhaps accelerate communication between client and RDP server but something is wrong between XRDP and XORG. I tried a lot of combination, with cyphering and without, bitmap ... 24 bits, ... This issue reminds me of all DIVX issues back in 1998 when you could see one image out of three and the video was not smooth.

Something else, login authentication bugs sometimes.

Furthermore, I wanted to use GNS3 and SSH is not possible for me ... I don't want to use Windows for this in regards to different internal network stack issues.

Regards and thanks!!

@gueug78400

[rjkip-seequent]

Changing tcp_nodelay to false and tcp_send_buffer_bytes to 262144 greatly improved throughput. Images don't slow everything to a crawl anymore. Thanks!

It is still a little bit disturbing that this can lead up to 3.6MiB/s continuous thoughput when working in a website with an image background (Microsoft AD, Trello board, etc.).

[jacqueshenning]

Hi @philayres
Trust you are well.
Have you managed to resolve this?

My scenario is different, Ubuntu Desktop 20.04 VM running XRDP behind a pfSesne Firewall, being accessed over a VPN.
Both sides are over 100Mbps Fibre lines, latency to remote host over VPN is a constant 21ms.
Super slow refresh, I have tried tuning the settings as described in this post but none have made any difference.

Apologies if I am hijacking, I can delete and create a new issue.

[philayres]

@jacqueshenning I don't think you are hijacking. Sounds like an extra data point for diagnosing the issue. Is it particularly bad for you with a textured background (or at least not block colours)?

I wish I understood X better. It seems to me that an app that has a textured background seems to perform the run length encoding of layers above, whether the background was showing or was already overlaid. But that is just what it feels like to me, without any real understanding about the mechanics of X and the RDP encoding.