Before this the visibility was changed if the window pointed to a
display without taking into account that it might not own it. Fix #4805