Accurate scoring

Id like to track a user score after they drag a series of objects to the correct location. Ive come up with the following way but this solution increases and decreases the score simply if the mouse is clicked on the object. Does anyone know of a way to prevent this?

Ive attached a sample to clarify what I mean.