No protocol specified, Could not initialize class sun.awt.X11.XToolkit [Oracle install via VNC]

Do not run vncserver as a root, a don't do 'su oracle'. Start VNC sever as a plain user, start runInstaller within his VNC session.