Hi,
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can't do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don't know where to start on this one since I don't know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent) PID STARTED RSS USER 4668 Fri Nov 8 08:35:35 2019 60896 user1 5743 Wed Dec 4 15:44:41 2019 249456 user2 14728 Thu Dec 5 10:17:25 2019 34926120 user3 19408 Fri Dec 20 08:46:30 2019 199720 user4 48797 Wed Oct 2 15:34:44 2019 297680 user5 50693 Thu Dec 12 18:24:24 2019 317796 user6 54561 Thu Dec 19 09:35:34 2019 28977804 user7 56244 Fri Jan 17 08:19:29 2020 170856 user8 60386 Thu Nov 14 09:23:37 2019 146416 user9
johannes@compute-server:~$ apt show x2goserver Package: x2goserver Version: 4.1.0.3-0~1708~ubuntu18.04.1
- Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
Thanks for quick reply.
Answers to your questions below. I hope it gives some clue. Please let me know what else I can try.
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
dpkg -l nx-libs: no packages found <--- this seems strange?
x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64
nxagent 2:3.5.99.20-0~20190425 amd64 Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
Nothing out of the ordinary there I think. No processes with high memory usage.You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
I'll try to think of a way to test this.What are you running inside the session?
No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc.— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 17, 2020 13:25 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
I have now set up a logging of the memory use over time. I hope this will give some data so we can understand the error. Hopefully I can correlate the memory use increase with user actions.
Ulrich, were you able to figure out if x2go can be run with valgrind?
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: x2go-user <x2go-user-bounces@lists.x2go.org> On Behalf Of Johannes Töger Sent: Friday, January 17, 2020 16:28 To: Ulrich Sibiller <uli42@gmx.de> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks for quick reply.
Answers to your questions below. I hope it gives some clue. Please let me know what else I can try.
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
dpkg -l nx-libs: no packages found <--- this seems strange?
x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64
nxagent 2:3.5.99.20-0~20190425 amd64 Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
Nothing out of the ordinary there I think. No processes with high memory usage.You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
I'll try to think of a way to test this.What are you running inside the session?
No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc.— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 17, 2020 13:25 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Sorry, I forgot to check, but probably not today. But can you please answer my questions in the meantime?
Basically you must replace the x2go binary with a little wrapper that use valgrind to start the original binary with the original arguments.
Uli
On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
I have now set up a logging of the memory use over time. I hope this will give some data so we can understand the error. Hopefully I can correlate the memory use increase with user actions.
Ulrich, were you able to figure out if x2go can be run with valgrind?
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: x2go-user <x2go-user-bounces@lists.x2go.org> On Behalf Of Johannes Töger Sent: Friday, January 17, 2020 16:28 To: Ulrich Sibiller <uli42@gmx.de> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks for quick reply.
Answers to your questions below. I hope it gives some clue. Please let me know what else I can try.
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
Nothing out of the ordinary there I think. No processes with high memory usage.You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
I'll try to think of a way to test this.What are you running inside the session?
No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc.— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 17, 2020 13:25 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
OK, thanks.
Sending the answers again, maybe they were cut by the mailing list before.
Question: What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Answer: dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64
Question: Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? Answer: Nothing out of the ordinary there I think. No processes with high memory usage.
Question: You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? Answer: I don't know. I started logging the memory usage over time, so when the problem arises next time I can correlate with log events and ask the users what they have been doing at certain times etc.
Question: What are you running inside the session?fdasfafad Answer: No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc. Maybe a browser here and there to look up stuff (Firefox).
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Monday, January 20, 2020 15:45 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Sorry, I forgot to check, but probably not today. But can you please answer my questions in the meantime?
Basically you must replace the x2go binary with a little wrapper that use valgrind to start the original binary with the original arguments.
Uli
On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
I have now set up a logging of the memory use over time. I hope this will give some data so we can understand the error. Hopefully I can correlate the memory use increase with user actions.
Ulrich, were you able to figure out if x2go can be run with valgrind?
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: x2go-user <x2go-user-bounces@lists.x2go.org> On Behalf Of Johannes Töger Sent: Friday, January 17, 2020 16:28 To: Ulrich Sibiller <uli42@gmx.de> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks for quick reply.
Answers to your questions below. I hope it gives some clue. Please let me know what else I can try.
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
Nothing out of the ordinary there I think. No processes with high memory usage.You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
I'll try to think of a way to test this.What are you running inside the session?
No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc.— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 17, 2020 13:25 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
On Tue, Jan 21, 2020 at 9:39 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
OK, thanks.
Sending the answers again, maybe they were cut by the mailing list before.
I have now found it, don't know how I managed to miss it. Sorry.
Question: What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Answer: dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64
Ok, we are at 3.5.99.22 at the moment (see https://sunweavers.net/blog/node/97) and close to 3.5.99.23 although x2go is not having packages for that yet. So please try http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/
(do not use the current nightlies, they miss a patch which fixes regular crashes)
Uli
Question: Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? Answer: Nothing out of the ordinary there I think. No processes with high memory usage.
Question: You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? Answer: I don't know. I started logging the memory usage over time, so when the problem arises next time I can correlate with log events and ask the users what they have been doing at certain times etc.
Question: What are you running inside the session?fdasfafad Answer: No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc. Maybe a browser here and there to look up stuff (Firefox).
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Monday, January 20, 2020 15:45 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Sorry, I forgot to check, but probably not today. But can you please answer my questions in the meantime?
Basically you must replace the x2go binary with a little wrapper that use valgrind to start the original binary with the original arguments.
Uli
On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
I have now set up a logging of the memory use over time. I hope this will give some data so we can understand the error. Hopefully I can correlate the memory use increase with user actions.
Ulrich, were you able to figure out if x2go can be run with valgrind?
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: x2go-user <x2go-user-bounces@lists.x2go.org> On Behalf Of Johannes Töger Sent: Friday, January 17, 2020 16:28 To: Ulrich Sibiller <uli42@gmx.de> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks for quick reply.
Answers to your questions below. I hope it gives some clue. Please let me know what else I can try.
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
Nothing out of the ordinary there I think. No processes with high memory usage.You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
I'll try to think of a way to test this.What are you running inside the session?
No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc.— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 17, 2020 13:25 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
Great, I see in the blog that you did a lot of work on fixing memory leaks. I'll try to install the .22 version.
But isn't it strange that I can't find or have an nx-libs package installed? What's going on there? Is the packaging for Ubuntu 18.04 LTS done in some other way?
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Tuesday, January 21, 2020 10:39 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Tue, Jan 21, 2020 at 9:39 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
OK, thanks.
Sending the answers again, maybe they were cut by the mailing list before.
I have now found it, don't know how I managed to miss it. Sorry.
Question: What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Answer: dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64
Ok, we are at 3.5.99.22 at the moment (see https://sunweavers.net/blog/node/97) and close to 3.5.99.23 although x2go is not having packages for that yet. So please try http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/
(do not use the current nightlies, they miss a patch which fixes regular crashes)
Uli
Question: Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session? Answer: Nothing out of the ordinary there I think. No processes with high memory usage.
Question: You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect? Answer: I don't know. I started logging the memory usage over time, so when the problem arises next time I can correlate with log events and ask the users what they have been doing at certain times etc.
Question: What are you running inside the session?fdasfafad Answer: No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc. Maybe a browser here and there to look up stuff (Firefox).
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Monday, January 20, 2020 15:45 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Sorry, I forgot to check, but probably not today. But can you please answer my questions in the meantime?
Basically you must replace the x2go binary with a little wrapper that use valgrind to start the original binary with the original arguments.
Uli
On Mon, Jan 20, 2020 at 2:26 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
I have now set up a logging of the memory use over time. I hope this will give some data so we can understand the error. Hopefully I can correlate the memory use increase with user actions.
Ulrich, were you able to figure out if x2go can be run with valgrind?
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: x2go-user <x2go-user-bounces@lists.x2go.org> On Behalf Of Johannes Töger Sent: Friday, January 17, 2020 16:28 To: Ulrich Sibiller <uli42@gmx.de> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Thanks for quick reply.
Answers to your questions below. I hope it gives some clue. Please let me know what else I can try.
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
dpkg -l nx-libs: no packages found <--- this seems strange? x2goserver-x2goagent 4.1.0.3-0~1708~ubuntu1 amd64 nxagent 2:3.5.99.20-0~20190425 amd64Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
Nothing out of the ordinary there I think. No processes with high memory usage.You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
I'll try to think of a way to test this.What are you running inside the session?
No real pattern here for good and bad sessions. Matlab, Python/machine learning stuff, NVIDIA CUDA-accelerated machine learning programs etc.— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 17, 2020 13:25 To: Johannes Töger <johannes.toger@med.lu.se> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Fri, Jan 17, 2020 at 11:56 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
We are using x2go for a common compute server at our department. We have users both on the local LAN and connections to the other side of Europe that work really well. Being able to close the client and have computations still running in a GUI environment is great.
However, I think we have encountered a bug. Some (but not all) of the long-running x2goagent processes start taking a lot of memory, ~30 GB. This adds up quickly so that the other users can’t do their work. Below you can see how user3 and user7 have x2goagent processes with large memory use (RSS column).
How can I go about debugging this? I have quite a bit of Linux experience, but I don’t know where to start on this one since I don’t know much about how x2go works internally.
Server is on Ubuntu Server 18.04.1 LTS, (x2goserver 4.1.0.3-0~1708~ubuntu18.04.1). Clients are on Windows 10, version. 4.1.2.0.
johannes@compute-server:~$ ps -o pid,lstart,rss,user $(pgrep x2goagent)
What version of x2goagent resp. nxagent are you using? dpkg -l nx-libs?
Can you please run xrestop within such a session and check if there are numbers that are far away from those of a fresh session?
You probably cannot answer that due to your usage scenario, but anyway: does this also happen when you do not disconnect/reconnect?
What are you running inside the session?
You could run x2goagent with valgrind to identify memory leaks but I am unsure if that is working on a stock installation ( I don't do the packaging for Debian/Ubuntu so I don't know if the deliveres binaries allow for decent analysis/output). I'll check that this weekend.
Uli
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
On Tue, Jan 21, 2020 at 11:24 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
Great, I see in the blog that you did a lot of work on fixing memory leaks. I'll try to install the .22 version.
But isn't it strange that I can't find or have an nx-libs package installed? What's going on there? Is the packaging for Ubuntu 18.04 LTS done in some other way?
No, this is a mistake from my side. The project is called nx-libs and produces a number of packages. You can replace the relevant x2go packages by the arctica ones.
Uli
Just a little update: I have tried to integrate valgrind into the x2go setup but I failed. I'll try again...
In your initial bug report you showed some measurement where you accumulated the memory per user. Can you verify that this amount is is tied to x2goagent processes?
Uli
On Tue, Jan 21, 2020 at 11:43 AM Ulrich Sibiller <uli42@gmx.de> wrote:
On Tue, Jan 21, 2020 at 11:24 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
Great, I see in the blog that you did a lot of work on fixing memory leaks. I'll try to install the .22 version.
But isn't it strange that I can't find or have an nx-libs package installed? What's going on there? Is the packaging for Ubuntu 18.04 LTS done in some other way?
No, this is a mistake from my side. The project is called nx-libs and produces a number of packages. You can replace the relevant x2go packages by the arctica ones.
Uli
On Thu, Jan 23, 2020 at 2:52 PM Ulrich Sibiller <uli42@gmx.de> wrote:
Just a little update: I have tried to integrate valgrind into the x2go setup but I failed. I'll try again...
In your initial bug report you showed some measurement where you accumulated the memory per user. Can you verify that this amount is is tied to x2goagent processes?
Forget that, I had missed the grep..
Uli
On 1/23/20 2:52 PM, Ulrich Sibiller wrote:
Just a little update: I have tried to integrate valgrind into the x2go setup but I failed. I'll try again...
I guess I should try to compile x2go myself, but typically ASAN (which then also runs LSAN) is much faster than valgrind. A bit tricky, though, to get the reports for daemon processes.
Ok, that's fine, too! I am curious about the results!
Uli
PS: Well, Johannes started the thread and I tried to help him, I am a bit confused that now you (Bernd) kick in. Are you a colleague of Johannes or are you also seeing that problem?
On Thu, Jan 23, 2020 at 3:23 PM Bernd Schubert <bs_lists@aakef.fastmail.fm> wrote:
On 1/23/20 2:52 PM, Ulrich Sibiller wrote:
Just a little update: I have tried to integrate valgrind into the x2go setup but I failed. I'll try again...
I guess I should try to compile x2go myself, but typically ASAN (which then also runs LSAN) is much faster than valgrind. A bit tricky, though, to get the reports for daemon processes.
On 1/23/20 3:41 PM, Ulrich Sibiller wrote:
Ok, that's fine, too! I am curious about the results!
Uli
PS: Well, Johannes started the thread and I tried to help him, I am a bit confused that now you (Bernd) kick in. Are you a colleague of Johannes or are you also seeing that problem?
Ah sorry, no, I'm just reading the thread. I had similar issues in the past with x2go, but right now long running sessions work pretty well. I'm also coding all the time and the compiler sanitizers are great help and work so much faster than valgrind - I just tried to point that out. Sorry for confusion.
- Bernd
Thanks to you both for getting involved in this.
I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help.
Link to plot: https://imgur.com/a/lDqZ2zd
As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: x2go-user <x2go-user-bounces@lists.x2go.org> On Behalf Of Bernd Schubert Sent: Thursday, January 23, 2020 16:11 To: Ulrich Sibiller <uli42@gmx.de> Cc: x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On 1/23/20 3:41 PM, Ulrich Sibiller wrote:
Ok, that's fine, too! I am curious about the results!
Uli
PS: Well, Johannes started the thread and I tried to help him, I am a bit confused that now you (Bernd) kick in. Are you a colleague of Johannes or are you also seeing that problem?
Ah sorry, no, I'm just reading the thread. I had similar issues in the past with x2go, but right now long running sessions work pretty well. I'm also coding all the time and the compiler sanitizers are great help and work so much faster than valgrind - I just tried to point that out. Sorry for confusion.
- Bernd
x2go-user mailing list x2go-user@lists.x2go.org https://lists.x2go.org/listinfo/x2go-user
On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
Thanks to you both for getting involved in this.
I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help.
Link to plot: https://imgur.com/a/lDqZ2zd
As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip.
OK, well, do you think you could compile an own version of x2go?
Uli
Yeah, I could absolutely compile my own version. I think it might be better to try with the .deb packages you linked first though.
I need to ask my users for a window where I can shut down everything first, so might take a few days before I can organize this.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Thursday, January 23, 2020 19:43 To: Johannes Töger <johannes.toger@med.lu.se> Cc: Bernd Schubert <bs_lists@aakef.fastmail.fm>; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
Thanks to you both for getting involved in this.
I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help.
Link to plot: https://imgur.com/a/lDqZ2zd
As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip.
OK, well, do you think you could compile an own version of x2go?
Uli
Well, from MPOV you do not need a shutdown. Just update the nx packages. The next new connection will use the new nxagent.
Let me now when you want to compile an own version, so that I can provide some instructions for sanitizer integration. (I need to figure that out again since I mainly use valgrind).
Uli
On Fri, Jan 24, 2020 at 8:35 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
Yeah, I could absolutely compile my own version. I think it might be better to try with the .deb packages you linked first though.
I need to ask my users for a window where I can shut down everything first, so might take a few days before I can organize this.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Thursday, January 23, 2020 19:43 To: Johannes Töger <johannes.toger@med.lu.se> Cc: Bernd Schubert <bs_lists@aakef.fastmail.fm>; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
Thanks to you both for getting involved in this.
I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help.
Link to plot: https://imgur.com/a/lDqZ2zd
As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip.
OK, well, do you think you could compile an own version of x2go?
Uli
Hi,
Yesterday I found some time to upgrade the packages. I did the following:
- Downloaded all packages, (.22, amd64, 18.04 versions) from http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/
- sudo dpkg -i *deb to install everything
- Asked users to end sessions and reconnect.
After this, the x2go memory use is <= 350 MB for each user and no longer growing (last 18 hours). So problem fixed from my point of view! Thanks for all your help.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 24, 2020 09:44 To: Johannes Töger <johannes.toger@med.lu.se> Cc: Bernd Schubert <bs_lists@aakef.fastmail.fm>; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Well, from MPOV you do not need a shutdown. Just update the nx packages. The next new connection will use the new nxagent.
Let me now when you want to compile an own version, so that I can provide some instructions for sanitizer integration. (I need to figure that out again since I mainly use valgrind).
Uli
On Fri, Jan 24, 2020 at 8:35 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
Yeah, I could absolutely compile my own version. I think it might be better to try with the .deb packages you linked first though.
I need to ask my users for a window where I can shut down everything first, so might take a few days before I can organize this.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Thursday, January 23, 2020 19:43 To: Johannes Töger <johannes.toger@med.lu.se> Cc: Bernd Schubert <bs_lists@aakef.fastmail.fm>; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
Thanks to you both for getting involved in this.
I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help.
Link to plot: https://imgur.com/a/lDqZ2zd
As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip.
OK, well, do you think you could compile an own version of x2go?
Uli
Great to hear that! This closes one point on my list ;-)
Uli
On Tue, Jan 28, 2020 at 9:37 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
Hi,
Yesterday I found some time to upgrade the packages. I did the following:
- Downloaded all packages, (.22, amd64, 18.04 versions) from http://packages.arctica-project.org/ubuntu/pool/main/n/nx-libs/
- sudo dpkg -i *deb to install everything
- Asked users to end sessions and reconnect.
After this, the x2go memory use is <= 350 MB for each user and no longer growing (last 18 hours). So problem fixed from my point of view! Thanks for all your help.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Friday, January 24, 2020 09:44 To: Johannes Töger <johannes.toger@med.lu.se> Cc: Bernd Schubert <bs_lists@aakef.fastmail.fm>; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
Well, from MPOV you do not need a shutdown. Just update the nx packages. The next new connection will use the new nxagent.
Let me now when you want to compile an own version, so that I can provide some instructions for sanitizer integration. (I need to figure that out again since I mainly use valgrind).
Uli
On Fri, Jan 24, 2020 at 8:35 AM Johannes Töger <johannes.toger@med.lu.se> wrote:
Yeah, I could absolutely compile my own version. I think it might be better to try with the .deb packages you linked first though.
I need to ask my users for a window where I can shut down everything first, so might take a few days before I can organize this.
— Johannes Töger
Associate Senior Lecturer Cardiac MR Group Department of Clinical Sciences Lund, Clinical Physiology Lund University, Sweden
-----Original Message----- From: Ulrich Sibiller <uli42@gmx.de> Sent: Thursday, January 23, 2020 19:43 To: Johannes Töger <johannes.toger@med.lu.se> Cc: Bernd Schubert <bs_lists@aakef.fastmail.fm>; x2go-user@lists.x2go.org Subject: Re: [X2Go-User] Long-running x2goagent, high memory usage
On Thu, Jan 23, 2020 at 7:35 PM Johannes Töger <johannes.toger@med.lu.se> wrote:
Thanks to you both for getting involved in this.
I've been swamped with other things, so not much to report. I did manage to plot the x2goagent memory use over time though. Maybe this can help.
Link to plot: https://imgur.com/a/lDqZ2zd
As you see, several users have a constant memory use. However, 3 of them have a constantly increasing memory use. So not isolated incidents that leak, but rather a continuous drip.
OK, well, do you think you could compile an own version of x2go?
Uli
-
Bernd Schubert -
Johannes Töger -
Ulrich Sibiller