Designing for the Minority?

Should we design sites so that everyone can view them, or just the significant majority. Example, I was asked to redesign a perfectly good site design with very readable text (to all concerned, bar one). The one stated the text would be hard to read for people who have their contrast and colours set wrongly (and very wrongly it turns out when i tried it later) on their screen?

Was I right to say that designs dont need changing to cater for these people or not?