On 02/04/2011 04:14 PM, Gerry Reno wrote:

On 02/02/2011 09:48 PM, Gerry Reno wrote:

...

On 02/02/2011 09:35 PM, Gerry Reno wrote:

...

I'm seeing the following errors trying to connect to a new machine running server version 3.0.99 using the browser client:

Feb 3 02:20:33 newmachine su[16393]: pam_unix(su:session): session opened for user ubuntu by (uid=0) Feb 3 02:20:33 newmachine sudo: ubuntu : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/touch /root/firstlogin_done Feb 3 02:20:33 newmachine sshd[16319]: Accepted publickey for ubuntu from XXX.XXX.XXX.XXX port 51610 ssh2 Feb 3 02:20:33 newmachine sshd[16319]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0) Feb 3 02:20:33 newmachine sshd[16320]: Accepted publickey for ubuntu from XXX.XXX.XXX.XXX port 51609 ssh2 Feb 3 02:20:33 newmachine sshd[16320]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0) Feb 3 02:20:33 newmachine sshd[16441]: error: bind: Address already in use Feb 3 02:20:33 newmachine sshd[16441]: error: channel_setup_fwd_listener: cannot listen to port: 30008 Feb 3 02:20:33 newmachine sshd[16234]: pam_unix(sshd:session): session closed for user ubuntu Feb 3 02:20:33 newmachine su[16393]: pam_unix(su:session): session closed for user ubuntu Feb 3 02:20:33 newmachine sshd[16524]: error: connect_to localhost port 30007: failed. Feb 3 02:20:33 newmachine sshd[16232]: pam_unix(sshd:session): session closed for user ubuntu Feb 3 02:20:33 newmachine sshd[29561]: channel 2: open failed: connect failed: Connection refused Feb 3 02:20:33 newmachine sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/touch /root/firstlogin_done Feb 3 02:20:33 newmachine sudo: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/touch /root/firstlogin_done Feb 3 02:20:33 newmachine sshd[16233]: pam_unix(sshd:session): session closed for user ubuntu

The client reports: Resuming Aborting Error message appears:

The remote proxy closed the connection while negotiating
the session. This may be due to the wrong authentication
credentials passed to the server.

Finished

I ran a check for sessions and there is only one existing session.

I've triple-checked the credentials and logged in on command line using them.

Here's what we have for x2go processes:

ubuntu 3219 1 0 Feb02 ? 00:00:50 /usr/lib/x2go/x2goagent -dpi 96 -D -auth /home/ubuntu/.Xauthority -geometry 1005x552+113+232 -name X2GO-ubuntu-51-1296672519_stDGNOME_dp24 :51 ubuntu 3690 1 0 Feb02 ? 00:00:00 /bin/bash /usr/bin/x2goruncommand 51 3219 ubuntu-51-1296672519_stDGNOME_dp24 30008 gnome-session nosnd D root 15119 1 0 Feb02 ? 00:00:04 /usr/bin/perl /usr/sbin/x2gocleansessions

So there is an existing session. I got detached from it when the browser crashed.

It seems as though the client should either automatically attach to the session or present the user with the list of sessions to take some action.

Regards, Gerry

Has anyone found out how to prevent this "remote proxy closed the connection" error from happening when a session becomes detached and a user tries to resume the session?

We are seeing this problem quite often and every once in a while you can get it to resume but mostly this session is unrecoverable and must be manually terminated - which is creating much sysadmin work.

Regards, Gerry

It looks as though NX proxy is having difficulty attaching to the session socket:

Feb  3 02:20:33 newmachine sshd[16441]: error: bind: Address already
in use
Feb  3 02:20:33 newmachine sshd[16441]: error:
channel_setup_fwd_listener: cannot listen to port: 30008

That's the only thing that stands out in the logs.

Regards, Gerry

On 02/04/2011 04:43 PM, Mike Gabriel wrote:

Hi Gerry,

On Fr 04 Feb 2011 22:14:28 CET Gerry Reno wrote:

...

Has anyone found out how to prevent this "remote proxy closed the connection" error from happening when a session becomes detached and a user tries to resume the session?

We are seeing this problem quite often and every once in a while you can get it to resume but mostly this session is unrecoverable and must be manually terminated - which is creating much sysadmin work.

Test the following:

Whenever this error occurs, restart your X2go server's SSH daemon. My guess is that you can resume the session once SSH is restarted.

I have read about weird port forwarding habits of the SSH daemon... I also meet these problems with PyHoca-GUI.

Greets, Mike

Ok, the plot thickens.

I took a look at netstat both prior-to and after we get this error:

# normal state # we can successfully resume a session

# netstat -a | grep 300
tcp        0      0 localhost:30008         *:*                    
LISTEN    
tcp        0      0 localhost:30009         *:*                    
LISTEN    
tcp6       0      0 ip6-localhost:30008     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30009     [::]:*                 
LISTEN    

# # after I forced a session to detach

# netstat -a | grep 300
tcp        0      0 localhost:30008         *:*                    
LISTEN    
tcp        0      0 localhost:30009         *:*                    
LISTEN    
tcp        0      0 localhost:33552         localhost:30037        
TIME_WAIT 
tcp6       0      0 ip6-localhost:30008     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30009     [::]:*                 
LISTEN    

# # if we just wait for a while then reattempt to resume the session it succeeds

# netstat -a | grep 300
tcp        0      0 localhost:30008         *:*                    
LISTEN    
tcp        0      0 localhost:30009         *:*                    
LISTEN    
tcp6       0      0 ip6-localhost:30008     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30009     [::]:*                 
LISTEN    

So whatever is causing this TIME_WAIT is what appears to have a role in the error.

Regards, Gerry

On 02/04/2011 06:09 PM, Gerry Reno wrote:

On 02/04/2011 06:01 PM, Gerry Reno wrote:

...

On 02/04/2011 04:43 PM, Mike Gabriel wrote:

...

Hi Gerry,

On Fr 04 Feb 2011 22:14:28 CET Gerry Reno wrote:

...

Has anyone found out how to prevent this "remote proxy closed the connection" error from happening when a session becomes detached and a user tries to resume the session?

We are seeing this problem quite often and every once in a while you can get it to resume but mostly this session is unrecoverable and must be manually terminated - which is creating much sysadmin work.

Test the following:

Whenever this error occurs, restart your X2go server's SSH daemon. My guess is that you can resume the session once SSH is restarted.

I have read about weird port forwarding habits of the SSH daemon... I also meet these problems with PyHoca-GUI.

Greets, Mike

Ok, the plot thickens.

I took a look at netstat both prior-to and after we get this error:

# normal state # we can successfully resume a session

# netstat -a | grep 300
tcp        0      0 localhost:30008         *:*                    
LISTEN    
tcp        0      0 localhost:30009         *:*                    
LISTEN    
tcp6       0      0 ip6-localhost:30008     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30009     [::]:*                 
LISTEN    

# # after I forced a session to detach

# netstat -a | grep 300
tcp        0      0 localhost:30008         *:*                    
LISTEN    
tcp        0      0 localhost:30009         *:*                    
LISTEN    
tcp        0      0 localhost:33552         localhost:30037        
TIME_WAIT 
tcp6       0      0 ip6-localhost:30008     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30009     [::]:*                 
LISTEN    

# # if we just wait for a while then reattempt to resume the session it succeeds

# netstat -a | grep 300
tcp        0      0 localhost:30008         *:*                    
LISTEN    
tcp        0      0 localhost:30009         *:*                    
LISTEN    
tcp6       0      0 ip6-localhost:30008     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30009     [::]:*                 
LISTEN    

So whatever is causing this TIME_WAIT is what appears to have a role in the error.

Regards, Gerry

What's interesting is that it presents the menu and the process is shown as "running" so then your "suspend" it and "resume" it and you still get the "remote proxy closed the connection" error and NX proxy shows "aborting" but it does resume the session. Very confusing to users but at least they get their session back.

Regards, Gerry

The TIME_WAIT occurs on the server because apparently there is no keep_alive which would make the client responsible for closing the connection. I didn't see any TCPKeepAlive=yes in the ssh options which may be accounting for TIME_WAIT showing up on the server.

Regards, Gerry

Hi Gerry,

On Sa 05 Feb 2011 00:09:52 CET Gerry Reno wrote:

What's interesting is that it presents the menu and the process is shown as "running" so then your "suspend" it and "resume" it and you still get the "remote proxy closed the connection" error and NX proxy shows "aborting" but it does resume the session. Very confusing to users but at least they get their session back.

By what I read the issue really occurs around the port forwarding
request. I try to sketch the mechanism...

o client: sets up a forwarding tunnel to the server (for nxproxy traffic which in our discussed scenario) o server: thinks the tunnel is already/still set up (by the former request) and denies the tunnel

 -> here PyHoca-GUI tries to push a cancel_port_forward request to the SSH
    daemon which sometimes works but sometimes doesn't

o the x2goclient does not realize the failed tunnel (PyHoca-GUI
btw. does and reports an error) o thus, x2goclient continues as planned o client: call of server-side x2goresume-session script (which fails because of the missing port forwarding tunnel) o client: from x2goresum-session script the x2goresume script is called (which updates the database and states that the new session is running although the session resuming failed)

The problem is that old port forwarding requests on the SSH server do
not get properly cancelled by the clients. If a request does not get
canceled the port is blocked/busy and another SSH client instance
cannot request a port forwarding on this server port.

Hope that sheds more light on the situation... If you can find an SSHd
option that could fix this behaviour this would be really great!!!

Greetings, Mike

--

DAS-NETZWERKTEAM mike gabriel, dorfstr. 27, 24245 barmissen fon: +49 (4302) 281418, fax: +49 (4302) 281419

GnuPG Key ID 0xB588399B mail: mike.gabriel@das-netzwerkteam.de, http://das-netzwerkteam.de

freeBusy: https://mail.das-netzwerkteam.de/freebusy/m.gabriel%40das-netzwerkteam.de.xf...

On 02/05/2011 09:45 AM, Gerry Reno wrote:

On 02/05/2011 05:25 AM, Mike Gabriel wrote:

...

Hi Gerry,

On Sa 05 Feb 2011 00:09:52 CET Gerry Reno wrote:

...

What's interesting is that it presents the menu and the process is shown as "running" so then your "suspend" it and "resume" it and you still get the "remote proxy closed the connection" error and NX proxy shows "aborting" but it does resume the session. Very confusing to users but at least they get their session back.

By what I read the issue really occurs around the port forwarding request. I try to sketch the mechanism...

o client: sets up a forwarding tunnel to the server (for nxproxy traffic which in our discussed scenario) o server: thinks the tunnel is already/still set up (by the former request) and denies the tunnel

-> here PyHoca-GUI tries to push a cancel_port_forward request to

the SSH daemon which sometimes works but sometimes doesn't

o the x2goclient does not realize the failed tunnel (PyHoca-GUI btw. does and reports an error) o thus, x2goclient continues as planned o client: call of server-side x2goresume-session script (which fails because of the missing port forwarding tunnel) o client: from x2goresum-session script the x2goresume script is called (which updates the database and states that the new session is running although the session resuming failed)

The problem is that old port forwarding requests on the SSH server do not get properly cancelled by the clients. If a request does not get canceled the port is blocked/busy and another SSH client instance cannot request a port forwarding on this server port.

Hope that sheds more light on the situation... If you can find an SSHd option that could fix this behaviour this would be really great!!!

Greetings, Mike

Right now I'm working only with the browser-client. I'm intentionally crashing the browser and then trying to resume the detached session. So from the client perspective it's a whole new client process that doesn't know anything about an existing tunnel.

What I'm thinking is that when this happens the server is not receiving an ACK and is putting the connection into TIME_WAIT state whose interval is defined as 2*MSL where MSL is the packet lifetime. Most networks have MSL set to 30 seconds these days. I think it used to be 120 seconds. So it looks like TIME_WAIT will last from 1 to 4 minutes depending upon the server network.

So what happens is if the user just waits long enough then they can resume the detached session. If they keep trying at intervals less than TIME_WAIT then it seems they never get reconnected.

I'm not sure what type of experiments need to be constructed to account for all the scenarios that can happen between the different types of clients and the server. But we need to think about that. And then see if we can manually construct these scenarios just using terminals and try to tweak the settings until we have good behavior under each scenario. Then try to determine how to get the client, nxproxy and the server to implement those behaviors.

Regards, Gerry

Been working with this problem some more today.

Here is what I observed:

I detach the client by killing the browser. On the client:

$ ps -ef | grep x2go
greno    24577 24393  0 21:08 ?        00:00:00 x2goclient
--embed-into=54525953 --config=/home/greno/.x2go/ssh/gen/x2go.cfg-eB8dER
greno    24592     1  0 21:08 ?        00:00:00
/home/greno/.mozilla/firefox/dmbbyyox.default/extensions/x2goplugin@obviously-nice.de/x2goclient/x2goclient
--dialog ok --caption NX - 51 --message The remote proxy closed the
connection while negotiating?the session. This may be due to the
wrong authentication?credentials passed to the server.? --local
--parent 24590 --display :0.0

On the server:

# netstat -a | grep 300
tcp        0      0 localhost:30005         *:*                    
LISTEN    
tcp        0      0 localhost:30006         *:*                    
LISTEN    
tcp6       0      0 ip6-localhost:30005     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30006     [::]:*                 
LISTEN    

Start a new browser-client session which aborts (twice on one try) with the remote proxy closed the connection error messages. On the client: $ ps -ef | grep x2go

greno    24577 24393  0 21:08 ?        00:00:01 x2goclient
--embed-into=54525953 --config=/home/greno/.x2go/ssh/gen/x2go.cfg-eB8dER
greno    24592     1  0 21:08 ?        00:00:00
/home/greno/.mozilla/firefox/dmbbyyox.default/extensions/x2goplugin@obviously-nice.de/x2goclient/x2goclient
--dialog ok --caption NX - 51 --message The remote proxy closed the
connection while negotiating?the session. This may be due to the
wrong authentication?credentials passed to the server.? --local
--parent 24590 --display :0.0
greno    24627 24577  0 21:11 ?        00:00:00 [x2goclient] <defunct>
greno    24638 24577  0 21:11 ?        00:00:00 [x2goclient] <defunct>

On the server:

# netstat -a | grep 300
tcp        0      0 localhost:30005         *:*                    
LISTEN    
tcp        0      0 localhost:30006         *:*                    
LISTEN    
tcp        0      0 localhost:59060         localhost:30004        
TIME_WAIT 
tcp        0      0 localhost:59058         localhost:30004        
TIME_WAIT 
tcp6       0      0 ip6-localhost:30005     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30006     [::]:*                 
LISTEN    

Wait 5 minutes (ServerKeepAliveInterval)

Try to Reconnect: first we get abort status showing and the remote proxy closed the connection message but it is immediately followed by resuming and running status and the session is reconnected. On the client:

$ ps -ef | grep x2go
greno    24577 24393  0 21:08 ?        00:00:01 x2goclient
--embed-into=54525953 --config=/home/greno/.x2go/ssh/gen/x2go.cfg-eB8dER
greno    24592     1  0 21:08 ?        00:00:00
/home/greno/.mozilla/firefox/dmbbyyox.default/extensions/x2goplugin@obviously-nice.de/x2goclient/x2goclient
--dialog ok --caption NX - 51 --message The remote proxy closed the
connection while negotiating?the session. This may be due to the
wrong authentication?credentials passed to the server.? --local
--parent 24590 --display :0.0
greno    24627 24577  0 21:11 ?        00:00:00 [x2goclient] <defunct>
greno    24638 24577  0 21:11 ?        00:00:00 [x2goclient] <defunct>
greno    24672 24577  0 21:12 ?        00:00:00 [x2goclient] <defunct>

On the server:

# netstat -a | grep 300
tcp        0      0 localhost:30005         *:*                    
LISTEN    
tcp        0      0 localhost:30006         *:*                    
LISTEN    
tcp        0      0 localhost:34849         localhost:30006        
ESTABLISHED
tcp        0      0 localhost:47652         localhost:30004        
ESTABLISHED
tcp        0      0 localhost:30006         localhost:34849        
ESTABLISHED
tcp        0      0 localhost:30004         localhost:47652        
ESTABLISHED
tcp        0      0 localhost:47649         localhost:30004        
TIME_WAIT 
tcp6       0      0 ip6-localhost:30005     [::]:*                 
LISTEN    
tcp6       0      0 ip6-localhost:30006     [::]:*                 
LISTEN    

So when we reconnect instead of 2 TIME_WAITS we only get one and the other 30004 connection goes into ESTABLISHED state.

Also the client seems to be severely misbehaving in that it is sending TWO x2goresume-session requests back-to-back. This probably is hosing up the database as well as the connection processing. And that probably accounts for the double defunct processes we see on the client as well as the double TIME_WAIT connections we see on the server.

Regards, Gerry

On 02/04/2011 04:54 PM, John A. Sullivan III wrote:

<snip> We have a related nightmare of a problem with a user running Windows XP on a very old computer who can almost never successfully connect. Sometimes the session starts and then never completes. We can see the "running" session in the database but we know the session has not started because the memory in the vserver is only 20MB - i.e., no KDE. The same user is constantly getting incorrect password errors to the point that OSSEC locks them out of any communication with the X2Go Server. This is no matter how carefully we enter the password.

<snip> - John

We have this problem with some XP as well. And still unresolved. We're not sure if it's a slow network, or the OS, or combination.

Regards, Gerry

On 02/04/2011 05:37 PM, John A. Sullivan III wrote:

On Fri, 2011-02-04 at 17:04 -0500, Gerry Reno wrote:

...

On 02/04/2011 04:54 PM, John A. Sullivan III wrote:

...

<snip> We have a related nightmare of a problem with a user running Windows XP on a very old computer who can almost never successfully connect. Sometimes the session starts and then never completes. We can see the "running" session in the database but we know the session has not started because the memory in the vserver is only 20MB - i.e., no KDE. The same user is constantly getting incorrect password errors to the point that OSSEC locks them out of any communication with the X2Go Server. This is no matter how carefully we enter the password.

<snip> - John

We have this problem with some XP as well. And still unresolved. We're not sure if it's a slow network, or the OS, or combination.

<snip> The majority of our users are running XP and only this one has a problem. The Internet connection is fine. We've been assuming it is because it is a ten to twelve year old computer. Are all your problematic ones old and slow? Thanks - John

Yes, old P4 machines.

Regards, Gerry