For me it's a case of whether it grips me or not in the first few pages (sometimes first few lines even). If I'm not gripped by the plot or the characters or something, then I simply give up and move on.
I used to read a book right the way through to the bitter end (I was in English Literature for far, far, far too long). However, one day I had an 'awakening' and realised life is too short to read books I don't want to read, so I stopped doing that.
I honestly can't remember any book I've stopped reading because I've never read it long enough for it to impinge on my memory for long enough.
The best example of a non-must read because of studies book I ploughed on with until the bitter end was The Curious Incident of the Dog in the Night-time. In fact it may have been this book that woke me up and made me realise I didn't have to carry on reading a book I wasn't enjoying.
Again, I can't call it a 'horrible' book per se; it was just a book I didn't enjoy, a book that made me uncomfortable and a book which had some subject matters which made me cringe. However, just because I disliked it, it doesn't make it horrible.