Theres a bit of a problem with the question (this is why you don't use MIF lol).The dot product can only be negative if the angle between the two vectors is obtuse.
Ignoring this: so (I'm assuming they meant the angle between the vectors is 150 degrees).
Next let . Then again squaring both sides: using the second definition of the dot product: . You now have two simultaneous equations solve for x and y and your done.
@ALOZZZ - Remember that, in squaring (as Qeru has done), there is the possibility of an extra (and invalid) solution having been introduced. Thus, you need to be careful in deciding whether the solution to the problem is one or both or neither of the solutions of the quadratic. The approach is correct and squaring is necessary, it just means that care is needed to make sure your solution is valid.
In this case, the dot product must be negative (as ) and so the solution of the problem must satisfy .
Both of these solutions satisfy and so both are valid.