[X2Go-Dev] Bug#516: Bug#516: [PATCH 1/5 v2] Create Windows and fix drawing issues on Big Endian 64bit systems

[Mihai Moldovan]

Multiple endiannes issues were setting incorrect event masks when creating and
drawing X11 windows.

This time, a smaller integer has been casted to a bigger one and passed to some
function actually setting its value.

This meant, that garbage from stack was attached to the smaller integer value,
putting unknown memory into the lower bytes of the bigger integer.

Fix this by creating a big, initialized temporary variable, let the function do
its magic on that one and pass the value back to the smaller variable -- and
cross your fingers the smaller variable can hold it without overrunning. (The
last bit is a design issue we can't really fix and has been around even before
this patch.)


This fixes window creation and redrawing/updating issues on Big Endian 64bit
systems.
-------------- next part --------------
diff --git a/nx-X11/programs/Xserver/hw/nxagent/Screen.c b/nx-X11/programs/Xserver/hw/nxagent/Screen.c
index 104d431..44c6cb4 100644
--- a/nx-X11/programs/Xserver/hw/nxagent/Screen.c
+++ b/nx-X11/programs/Xserver/hw/nxagent/Screen.c
@@ -1665,7 +1665,10 @@ N/A
 
     attributes.background_pixel = nxagentBlackPixel;
 
-    nxagentGetDefaultEventMask((Mask*)&attributes.event_mask);
+    /* Assume that the mask fits in int... broken on Big Endian 64bit systems. */
+    Mask tmp_mask = attributes.event_mask;
+    nxagentGetDefaultEventMask(&tmp_mask);
+    attributes.event_mask = (int)tmp_mask;
 
     attributes.colormap = nxagentDefaultVisualColormap(nxagentDefaultVisual(pScreen));
 
diff --git a/nx-X11/programs/Xserver/hw/nxagent/Window.c b/nx-X11/programs/Xserver/hw/nxagent/Window.c
index 0a8e8c2..5ffd24d 100644
--- a/nx-X11/programs/Xserver/hw/nxagent/Window.c
+++ b/nx-X11/programs/Xserver/hw/nxagent/Window.c
@@ -335,7 +335,10 @@ FIXME: We need to set save under on the real display?
 
   if (mask & CWEventMask)
   {
-    nxagentGetEventMask(pWin, (Mask*)&attributes.event_mask);
+    /* Assume that the mask fits in int... broken on Big Endian 64bit systems. */
+    Mask tmp_mask = attributes.event_mask;
+    nxagentGetEventMask(pWin, &tmp_mask);
+    attributes.event_mask = (int)tmp_mask;
   }
   #ifdef WARNING
   else
@@ -2921,7 +2924,10 @@ FIXME: Do we need to set save unders attribute here?
 
   if (mask & CWEventMask)
   {
-    nxagentGetEventMask(pWin, (Mask*)&attributes.event_mask);
+    /* Assume that the mask fits in int... broken on Big Endian 64bit systems. */
+    Mask tmp_mask = attributes.event_mask;
+    nxagentGetEventMask(pWin, &tmp_mask);
+    attributes.event_mask = (int)tmp_mask;
   }
   #ifdef WARNING
   else
@@ -3382,7 +3388,10 @@ void nxagentSetTopLevelEventMask(pWin)
 
   if (nxagentOption(Rootless) && nxagentWindowTopLevel(pWin))
   {
-    nxagentGetEventMask(pWin, (Mask*)&attributes.event_mask);
+    /* Assume that the mask fits in int... broken on Big Endian 64bit systems. */
+    Mask tmp_mask = attributes.event_mask;
+    nxagentGetEventMask(pWin, &tmp_mask);
+    attributes.event_mask = (int)tmp_mask;
 
     XChangeWindowAttributes(nxagentDisplay, nxagentWindow(pWin), mask, &attributes);
   }
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4265 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.x2go.org/pipermail/x2go-dev/attachments/20140621/8a079397/attachment.bin>