package: x2gobroker version: 0.0.4.0-0~1038~ubuntu16.04.1 priority: enhancement
We encountered another corner case very annoying. The whole broker stack does its work quite well when everything works as expected (which is mostly the case). A failing server for example is perfectly handled, loadchecker simply disable this server from the list and thus, broker sends users to remaining live servers. Problems are beginning when your servers misbehaves.
Loadchecker mainly checks :
- liveness ("ping")
- ssh acess
- load/memory values
This is okay but not enough to be sure that a server can handle incoming connections, for example, for our needs, we should make sure that :
- a user can authenticate on a server (Active Directory authentication through PAM/Winbind)
- some mounts are correct (/home folders, other user mounts or shares)
- some networks are available
- some mandatory services are running
- ...
In our recent problems, for unknown reasons, some servers were having troubles joining the Active Directory domain and thus, user auth was failing. On the broker/loadchecker side, the server is perfectly working and this is even one of the best performing server since it is empty so it allways tries to redirect incoming connections to this server and since users can't auth, it fails and blocks every new connections. With extended checks, this server would have been considered offline from the loadchecker point of view and thus, it would have just been out of the load balancing.
Since those checks are pretty "specific", it would be great to have some kind of "extended check" feature where a directory "extra-check.d" folder would exist on the x2gobroker setup in which we could drop some scripts that would be executed by the x2gobroker user on the remote servers with just an "ok" or "ko" value. As soon as you get a "ko" value, the server would be considered unavailable and then removed from load balancing until the next "ok" check. Script could be of any language with supported intepreter installed on the server (could be bash, perl, python or anything, you'll just need the interpreted to be installed but this is the administrator responsibility).
DISCLAIMER: This e-mail is private and confidential and may contain proprietary or legally privileged information. It is for the intended recipient only. If you have received this email in error, please notify the author by replying to it and then destroy it. If you are not the intended recipient you must not use, disclose, distribute, copy, print or rely on this e-mail or any attachment. Thank you