Sort array of colors

I have an array of colors that i want to sort from light to dark. currently I am reading them in as uints and sorting the array decending using colors.sort(Array.DESCENDING); It almost works the way I want it to but there are always a couple of dark colors mixed in with lighter ones( black in a range of yellows for example). Anyone know of a good way to compare rgb values?

Thanks