By unstable he means that your gpu can only operate up to a certain frquency until it will no longer work. Your gpu is going beyone this limit, and will start to have problems.
Btw, I really wouldn't overclock anything on an alienware, just because you really don't know about the oem parts in there. But A modest like 900 core overclock should be fine.
P.S. this is coming from someone who knows their pgu will operate at 1050mhz but instead goes at 1 ghz so I don't have to deal with instability.