I did a blog post a while back in which I made the mistake of using what I thought at the time was an innocuous example of rejecting a work because the characters were unlikeable. These two words, “unlikeable characters,” set off a bit of a sh**storm, if you’ll excuse my language.
What many of you tend to forget is that this business is subjective, and the likeability of characters, the likeability of people, is subjective. In the comments section you listed a number of examples of characters you felt were unlikeable but clearly worked in literature, one of whom was Lisbeth Salander from the Stieg Larsson series. I didn’t find her unlikeable at all. I thought she was damaged, odd, interesting, and intriguing. Not unlikeable. My opinion, of course.
For me likeability tends to coincide with one-dimensional. A character being unlikeable usually means she has no redeeming qualities, and usually even unlikeable people have a redeeming quality or two, something that gives them more depth. Lisbeth Salander, for example, has a vulnerability that gives her an intriguing dimension. And I find that true of any successful but unlikeable characters, typically they have qualities that make them likeable, or at least intriguing. Maybe they have a brilliant mind or damaged past. Either way, we desperately want to know more about them, want to spend more time with them, despite the fact that they repel us.
The other thing to consider when writing an unlikeable character is what genre you’re writing. In romance, for example, it’s really hard to write an unlikeable hero. Sure he can be damaged and yes, he can definitely have flaws, but your readers, along with your heroine, need to fall in love with him, so we need to see the good side of him too. And that’s just one example.
Certainly if an agent or editor tells you she didn’t find your character likeable, you can ignore what she says, assuming it’s subjective, and find someone who does. Or you could take a close look at your character to see if possibly it’s that she needs that thing (or two) to give her more depth and vulnerability.