Your logic somewhat works when it comes to fields with a low labour supply. Businesses can't exactly afford to be picky if the number of workers capable of doing the tasks you require them for is heavily limited, especially if there are several businesses competing against each other in that field. It doesn't work for other markets that have a high labour supply, where businesses in that sector can afford to act off of any discriminatory biases that recruiters and bosses may have. You may also get situations where, even for businesses with a more limited labour supply, businesses may choose to exclude one worker for the purpose of retaining other workers or keeping other workers happy.
I'd say that the arguments against having a laissez-faire attitude towards employment and workplace discrimination are way stronger on issues such as race, gender and class though.