Hi all,
Asking for help please troubleshooting an issue with x2go.
Inbound connections stopped working after I updated the kernel and nvidia drivers on my centos workstation (the x2go server).
On the client, the error message says "Connection failed. Failed to start X2Go Agent session with ID account-_____. X2Go Agent terminated unexpectedly. Aborting session startup."
[If I attempt a connecion again the client shows a previous apparently pre-existing connection, but I cannot resume it.]
On the server, the session log says "Error: Aborting session with 'Unable to open display 'nx/nx,options=...". I have attempted uninstalling and reinstalling x2goserver from epel.
This symptoms are similar to an issue Mihai Moldovan resolved for Qianqian Fang in April: https://www.mail-archive.com/x2go-user@lists.x2go.org/msg02809.html
However, I do not have the offending line set in LD_LIBRARY_PATH which solved the issue for Qianqian.
#Contents of LD_LIBRARY_PATH (this is from env after ssh-in).
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/[...]/caffe/distribute/lib/::/usr/lib:/usr/lib/python2.7
I also attempted commenting out this setting from .bashrc, and that did not fix it.
I've copied below answers to some questions Mihai asked Qianqian. The only strange thing I noticed is that Mihai asked for the contents of /usr/lib64/nx/X11 , and this is empty. The library Qianqian listed in there is located at /usr/lib64 on my server.
Some more details below, answering questions that were made on this earlier, similar troubleshooting request.
Best regards, thanks for any help, Ale
# Please make sure that nx-libs 3.5.99.16 is installed on the server side:
Package nx-libs-3.5.99.16-1.el7.x86_64 already installed and latest version
# Client up to date:
x2goclient is already the newest version (4.1.1.1-2).
# Perhaps this is useful, these lines are also from env:
QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/1 QT_GRAPHICSSYSTEM_CHECKED=1 QTLIB=/usr/lib64/qt-3.3/lib QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
# Contents of /usr/lib64/nx/X11
$ ls /usr/lib64/nx bin X11
$ ls -lah /usr/lib64/nx/X11 total 0 drwxr-xr-x. 2 root root 6 Aug 18 14:28 . drwxr-xr-x. 4 root root 26 Aug 18 14:28 ..
$ ls -lah /usr/lib64/nx/bin total 0 drwxr-xr-x. 2 root root 36 Aug 17 21:28 . drwxr-xr-x. 4 root root 26 Aug 18 14:28 .. lrwxrwxrwx. 1 root root 16 Aug 17 16:09 nxagent -> /usr/bin/nxagent lrwxrwxrwx. 1 root root 7 Aug 17 21:28 x2goagent -> nxagent
$ ls -lah /usr/bin/nxagent -rwxr-xr-x. 1 root root 4.5M Jul 18 17:47 /usr/bin/nxagent
# Where I found the related libraries
$ ls -lah /usr/lib64/*NX*
lrwxrwxrwx. 1 root root 18 Aug 18 14:15 /usr/lib64/libNX_X11.so.6 -> libNX_X11.so.6.3.0 -rwxr-xr-x. 1 root root 1.3M Jul 18 17:47 /usr/lib64/libNX_X11.so.6.3.0
You seem to be missing a libX11 link to libNX_X11. Please run ldd /usr/bin/nxagent and post the result.
Uli
Alec Bayo <alec.bayo@gmail.com> schrieb am Mo., 20. Aug. 2018, 19:48:
Hi all,
Asking for help please troubleshooting an issue with x2go.
Inbound connections stopped working after I updated the kernel and nvidia drivers on my centos workstation (the x2go server).
On the client, the error message says "Connection failed. Failed to start X2Go Agent session with ID account-_____. X2Go Agent terminated unexpectedly. Aborting session startup."
[If I attempt a connecion again the client shows a previous apparently pre-existing connection, but I cannot resume it.]
On the server, the session log says "Error: Aborting session with 'Unable to open display 'nx/nx,options=...". I have attempted uninstalling and reinstalling x2goserver from epel.
This symptoms are similar to an issue Mihai Moldovan resolved for Qianqian Fang in April: https://www.mail-archive.com/x2go-user@lists.x2go.org/msg02809.html
However, I do not have the offending line set in LD_LIBRARY_PATH which solved the issue for Qianqian.
#Contents of LD_LIBRARY_PATH (this is from env after ssh-in).
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/[...]/caffe/distribute/lib/::/usr/lib:/usr/lib/python2.7
I also attempted commenting out this setting from .bashrc, and that did not fix it.
I've copied below answers to some questions Mihai asked Qianqian. The only strange thing I noticed is that Mihai asked for the contents of /usr/lib64/nx/X11 , and this is empty. The library Qianqian listed in there is located at /usr/lib64 on my server.
Some more details below, answering questions that were made on this earlier, similar troubleshooting request.
Best regards, thanks for any help, Ale
# Please make sure that nx-libs 3.5.99.16 is installed on the server side:
Package nx-libs-3.5.99.16-1.el7.x86_64 already installed and latest version
# Client up to date:
x2goclient is already the newest version (4.1.1.1-2).
# Perhaps this is useful, these lines are also from env:
QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/1 QT_GRAPHICSSYSTEM_CHECKED=1 QTLIB=/usr/lib64/qt-3.3/lib QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
# Contents of /usr/lib64/nx/X11
$ ls /usr/lib64/nx bin X11
$ ls -lah /usr/lib64/nx/X11 total 0 drwxr-xr-x. 2 root root 6 Aug 18 14:28 . drwxr-xr-x. 4 root root 26 Aug 18 14:28 ..
$ ls -lah /usr/lib64/nx/bin total 0 drwxr-xr-x. 2 root root 36 Aug 17 21:28 . drwxr-xr-x. 4 root root 26 Aug 18 14:28 .. lrwxrwxrwx. 1 root root 16 Aug 17 16:09 nxagent -> /usr/bin/nxagent lrwxrwxrwx. 1 root root 7 Aug 17 21:28 x2goagent -> nxagent
$ ls -lah /usr/bin/nxagent -rwxr-xr-x. 1 root root 4.5M Jul 18 17:47 /usr/bin/nxagent
# Where I found the related libraries
$ ls -lah /usr/lib64/*NX*
lrwxrwxrwx. 1 root root 18 Aug 18 14:15 /usr/lib64/libNX_X11.so.6 -> libNX_X11.so.6.3.0 -rwxr-xr-x. 1 root root 1.3M Jul 18 17:47 /usr/lib64/libNX_X11.so.6.3.0
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Thanks for the reply Uli. Pasted output below
Also, I was scratching my head some more after sending the previous email. Would I run the two lines below to add these links manually? Or should I do something more/else?
ln -s /usr/lib64/libNX_X11.so.6 /usr/lib64/nx/X11/libX11.so.6 ln -s /usr/lib64/libNX_X11.so.6.3.0 /usr/lib64/nx/X11/libX11.so.6.3.0
Best, A.
$ ldd /usr/bin/nxagent linux-vdso.so.1 => (0x00007ffe21fc2000) libtirpc.so.1 => /lib64/libtirpc.so.1 (0x00007fa9d1cd4000) libX11.so.6 => /lib64/libX11.so.6 (0x00007fa9d1995000) libXcomp.so.3 => /lib64/libXcomp.so.3 (0x00007fa9d166c000) libXcompshad.so.3 => /lib64/libXcompshad.so.3 (0x00007fa9d1460000) libXrender.so.1 => /lib64/libXrender.so.1 (0x00007fa9d1254000) libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007fa9d104e000) libXfont.so.1 => /lib64/libXfont.so.1 (0x00007fa9d0e1a000) libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007fa9d0c16000) libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007fa9d0a13000) libXdmcp.so.6 => /lib64/libXdmcp.so.6 (0x00007fa9d080d000) libNX_X11.so.6 => /lib64/libNX_X11.so.6 (0x00007fa9d04c9000) libXext.so.6 => /lib64/libXext.so.6 (0x00007fa9d02b7000) libz.so.1 => /lib64/libz.so.1 (0x00007fa9d00a1000) libm.so.6 => /lib64/libm.so.6 (0x00007fa9cfd9e000) libXpm.so.4 => /lib64/libXpm.so.4 (0x00007fa9cfb8c000) libxml2.so.2 => /lib64/libxml2.so.2 (0x00007fa9cf822000) libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007fa9cf578000) libpng15.so.15 => /lib64/libpng15.so.15 (0x00007fa9cf34d000) libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fa9cf0f8000) libc.so.6 => /lib64/libc.so.6 (0x00007fa9ced2a000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fa9ceadd000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fa9ce7f5000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fa9ce5c1000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fa9ce3bd000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fa9ce1a3000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa9cdf86000) libxcb.so.1 => /lib64/libxcb.so.1 (0x00007fa9cdd5e000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fa9cdb5a000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa9cd852000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa9cd63c000) libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007fa9cd439000) libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007fa9cd22d000) libXtst.so.6 => /lib64/libXtst.so.6 (0x00007fa9cd027000) libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007fa9ccd81000) libfontenc.so.1 => /lib64/libfontenc.so.1 (0x00007fa9ccb79000) /lib64/ld-linux-x86-64.so.2 (0x00007fa9d25bd000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fa9cc953000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fa9cc744000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fa9cc540000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fa9cc327000) libXau.so.6 => /lib64/libXau.so.6 (0x00007fa9cc122000) libXi.so.6 => /lib64/libXi.so.6 (0x00007fa9cbf12000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fa9cbcea000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fa9cba88000)
On Mon, Aug 20, 2018 at 1:09 PM, Ulrich Sibiller <ulrich.sibiller@gmail.com> wrote:
You seem to be missing a libX11 link to libNX_X11. Please run ldd /usr/bin/nxagent and post the result.
Uli
Alec Bayo <alec.bayo@gmail.com> schrieb am Mo., 20. Aug. 2018, 19:48:
Hi all,
Asking for help please troubleshooting an issue with x2go.
Inbound connections stopped working after I updated the kernel and nvidia drivers on my centos workstation (the x2go server).
On the client, the error message says "Connection failed. Failed to start X2Go Agent session with ID account-_____. X2Go Agent terminated unexpectedly. Aborting session startup."
[If I attempt a connecion again the client shows a previous apparently pre-existing connection, but I cannot resume it.]
On the server, the session log says "Error: Aborting session with 'Unable to open display 'nx/nx,options=...". I have attempted uninstalling and reinstalling x2goserver from epel.
This symptoms are similar to an issue Mihai Moldovan resolved for Qianqian Fang in April: https://www.mail-archive.com/ x2go-user@lists.x2go.org/msg02809.html
However, I do not have the offending line set in LD_LIBRARY_PATH which solved the issue for Qianqian.
#Contents of LD_LIBRARY_PATH (this is from env after ssh-in).
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/[...]/caffe/ distribute/lib/::/usr/lib:/usr/lib/python2.7
I also attempted commenting out this setting from .bashrc, and that did not fix it.
I've copied below answers to some questions Mihai asked Qianqian. The only strange thing I noticed is that Mihai asked for the contents of /usr/lib64/nx/X11 , and this is empty. The library Qianqian listed in there is located at /usr/lib64 on my server.
Some more details below, answering questions that were made on this earlier, similar troubleshooting request.
Best regards, thanks for any help, Ale
# Please make sure that nx-libs 3.5.99.16 is installed on the server side:
Package nx-libs-3.5.99.16-1.el7.x86_64 already installed and latest version
# Client up to date:
x2goclient is already the newest version (4.1.1.1-2).
# Perhaps this is useful, these lines are also from env:
QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/1 QT_GRAPHICSSYSTEM_CHECKED=1 QTLIB=/usr/lib64/qt-3.3/lib QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
# Contents of /usr/lib64/nx/X11
$ ls /usr/lib64/nx bin X11
$ ls -lah /usr/lib64/nx/X11 total 0 drwxr-xr-x. 2 root root 6 Aug 18 14:28 . drwxr-xr-x. 4 root root 26 Aug 18 14:28 ..
$ ls -lah /usr/lib64/nx/bin total 0 drwxr-xr-x. 2 root root 36 Aug 17 21:28 . drwxr-xr-x. 4 root root 26 Aug 18 14:28 .. lrwxrwxrwx. 1 root root 16 Aug 17 16:09 nxagent -> /usr/bin/nxagent lrwxrwxrwx. 1 root root 7 Aug 17 21:28 x2goagent -> nxagent
$ ls -lah /usr/bin/nxagent -rwxr-xr-x. 1 root root 4.5M Jul 18 17:47 /usr/bin/nxagent
# Where I found the related libraries
$ ls -lah /usr/lib64/*NX*
lrwxrwxrwx. 1 root root 18 Aug 18 14:15 /usr/lib64/libNX_X11.so.6 -> libNX_X11.so.6.3.0 -rwxr-xr-x. 1 root root 1.3M Jul 18 17:47 /usr/lib64/libNX_X11.so.6.3.0
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
On Mon, Aug 20, 2018 at 8:17 PM, Alec Bayo <alec.bayo@gmail.com> wrote:
Thanks for the reply Uli. Pasted output below
Also, I was scratching my head some more after sending the previous email. Would I run the two lines below to add these links manually? Or should I do something more/else?
ln -s /usr/lib64/libNX_X11.so.6 /usr/lib64/nx/X11/libX11.so.6 ln -s /usr/lib64/libNX_X11.so.6.3.0 /usr/lib64/nx/X11/libX11.so.6.3.0
No, this would just be wrong!
$ ldd /usr/bin/nxagent linux-vdso.so.1 => (0x00007ffe21fc2000) libtirpc.so.1 => /lib64/libtirpc.so.1 (0x00007fa9d1cd4000) libX11.so.6 => /lib64/libX11.so.6 (0x00007fa9d1995000) ... libNX_X11.so.6 => /lib64/libNX_X11.so.6 (0x00007fa9d04c9000)
An here's the problem. You should never see both libs in the ldd output. NX brings libNX_X11 but does not directly require it (linker-wise). Instead it requires a libX11.so.6. There must be a link in an NX related dir (I don't know the exact path on CentOS/RHEL and cannot look that up atm). That link must be named libX11.so.6 and point to libNX_X11.so.6.
It is a trick that is required to ensure the other X11 libs (that are required by NX) who require a libX11.so.6 themselves use the NX variant of the lib instead of the one from the system (which would break NX).
Now try objdump -x /usr/bin/nxagent. Grep for RUNPATH in the output and you will find the directory where to put the missing link. In that directory run
ln -s /lib64/libNX_X11.so.6 libX11.so.6
This should fix it.
It would be interesting to know the reason for the missing link.
Uli
Thank you so much Uli, that fixed it!
I'd be happy to report further if there's anything that I can look up as to why the link might have been missing.
I don't know if it will be of general/debugging interest, however, because my workstation's yum is not the most healthy state: I've followed some bad advice from dodgy internet "how tos" in the past, e.g. installing a kernel beyond the level supported by CentOS, which I've backtracked from. Each kernel change requires reinstalling Nvidia drivers.
Currently, yum update generates the following errors, which might be related:
yum update
Loaded plugins: fastestmirror, langpacks, versionlock Loading mirror speeds from cached hostfile
- base: reflector.westga.edu
- elrepo: reflector.westga.edu
- epel: mirror.team-cymru.com
- extras: centos.mirrors.tds.net
- nux-dextop: mirror.li.nux.ro
- updates: centos.servint.com Resolving Dependencies --> Running transaction check ---> Package xorg-x11-drv-nvidia.x86_64 1:352.99-1.el7 will be updated ---> Package xorg-x11-drv-nvidia.x86_64 1:396.44-1.el7 will be an update --> Processing Dependency: nvidia-kmod >= 1:396.44 for package: 1:xorg-x11-drv-nvidia-396.44-1.el7.x86_64 ---> Package xorg-x11-drv-nvidia-devel.x86_64 1:352.99-1.el7 will be updated ---> Package xorg-x11-drv-nvidia-devel.x86_64 1:396.44-1.el7 will be an update ---> Package xorg-x11-drv-nvidia-gl.x86_64 1:352.99-1.el7 will be updated ---> Package xorg-x11-drv-nvidia-gl.x86_64 1:396.44-1.el7 will be an update ---> Package xorg-x11-drv-nvidia-libs.x86_64 1:352.99-1.el7 will be updated ---> Package xorg-x11-drv-nvidia-libs.x86_64 1:396.44-1.el7 will be an update --> Finished Dependency Resolution Error: Package: 1:xorg-x11-drv-nvidia-396.44-1.el7.x86_64 (cuda) Requires: nvidia-kmod >= 1:396.44 Installed: 1:nvidia-kmod-352.99-2.el7.x86_64 (@cuda) nvidia-kmod = 1:352.99-2.el7 Available: kmod-nvidia-390.67-1.el7_5.elrepo.x86_64 (elrepo) nvidia-kmod = 390.67-1.el7_5.elrepo Available: kmod-nvidia-390.77-1.el7_5.elrepo.x86_64 (elrepo) nvidia-kmod = 390.77-1.el7_5.elrepo You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
Thanks again, best, A.
On Mon, Aug 20, 2018 at 1:43 PM, Ulrich Sibiller <uli42@gmx.de> wrote:
On Mon, Aug 20, 2018 at 8:17 PM, Alec Bayo <alec.bayo@gmail.com> wrote:
Thanks for the reply Uli. Pasted output below
Also, I was scratching my head some more after sending the previous email. Would I run the two lines below to add these links manually? Or should I do something more/else?
ln -s /usr/lib64/libNX_X11.so.6 /usr/lib64/nx/X11/libX11.so.6 ln -s /usr/lib64/libNX_X11.so.6.3.0 /usr/lib64/nx/X11/libX11.so.6.3.0
No, this would just be wrong!
$ ldd /usr/bin/nxagent linux-vdso.so.1 => (0x00007ffe21fc2000) libtirpc.so.1 => /lib64/libtirpc.so.1 (0x00007fa9d1cd4000) libX11.so.6 => /lib64/libX11.so.6 (0x00007fa9d1995000) ... libNX_X11.so.6 => /lib64/libNX_X11.so.6 (0x00007fa9d04c9000)
An here's the problem. You should never see both libs in the ldd output. NX brings libNX_X11 but does not directly require it (linker-wise). Instead it requires a libX11.so.6. There must be a link in an NX related dir (I don't know the exact path on CentOS/RHEL and cannot look that up atm). That link must be named libX11.so.6 and point to libNX_X11.so.6.
It is a trick that is required to ensure the other X11 libs (that are required by NX) who require a libX11.so.6 themselves use the NX variant of the lib instead of the one from the system (which would break NX).
Now try objdump -x /usr/bin/nxagent. Grep for RUNPATH in the output and you will find the directory where to put the missing link. In that directory run
ln -s /lib64/libNX_X11.so.6 libX11.so.6
This should fix it.
It would be interesting to know the reason for the missing link.
Uli
-
Alec Bayo -
Ulrich Sibiller -
Ulrich Sibiller