Many memory leaks that look like x11 are just leaks of programs using x;
what you should test is running the system for a few days with X on but o
nly
a login screen! 
I do support changing the idea of replacing the graphics driver if the er
ror
persists.
Niels