While it's true that the story's the important thing, and that's what sells a book, and while there are a lot of bestselling books that are really badly written but readers don't care because they love the story and the characters, if you don't have good mechanics, it becomes a lot more difficult to sell a book. Anything that distracts an editor from the story is a bad thing, and if the editor is mentally editing while reading a submission, she's less likely to fall in love with the story. You improve your chances by using the right words in the right way.
Here are a few things to look for that I think are common errors, judging by Internet posts, e-mails and manuscripts I've either judged or critiqued:
They're vs. their
"They're" is the contraction for "they are." "Their" is a possessive pronoun. An easy way to remember the difference is that the apostrophe fills in for missing letters.
It's vs. its
Similar to the above. "It's" is the contraction for "it is." "Its" with no apostrophe is the possessive pronoun.
You're vs. your
Again, like the above. Use "you're" when you mean "you are" and "your" when you're talking about something belonging to someone.
In general, you don't use an apostrophe before the s when making something plural. If you're making a plural noun possessive, you add the apostrophe at the end of the word.
Repeated, ongoing problems with any of the above will cause most editors to scream in pain, and they may stop seeing anything other than the errors in your manuscript, so they won't even notice if you have a good story.
Affect vs. effect
There are some exceptions for specialized uses, but generally "affect" is a verb. "Effect" is the noun. The effect is the result when someone affects something. "Affect" may be a noun in psychology, and "effect" may be a verb meaning "to bring about," but if you aren't writing psychological papers and have any confusion between affect and effect, avoid these uses and stick to the "affect=verb, effect=noun" rule.
Accept vs. except
"Accept" means to take something that's being offered. "Except" means "other than." So, I accept the job offer, except I refuse to work weekends.
Rein vs. reign
This can get confusing because the literal meanings for both involve control. "Rein" is the mechanism for controlling a horse, while "reign" is what a king does. When you say someone has been given "free rein," it means letting the horse go where it wants without trying to control it using the reins, although technically and literally it wouldn't be entirely incorrect to think of "free reign" as someone being allowed to rule without checks and balances. However, the expression did come from horsemanship, so "rein" is correct (and that's what the Associated Press style book says is to be used). The same applies to other related figures of speech, like "seize the reins" or "rein in."
This term has become so misused that it's become a running joke on the sitcom Parks and Recreation. "Literally" is supposed to mean that this thing that is usually considered to be a figure of speech is actually true in this case. What most people mean when they say "literally" is "figuratively," though the "figuratively" can generally be taken to be implied unless there's a chance for confusion from someone taking you literally. So, we might say figuratively that it's raining buckets when it's just raining hard, but if there's an explosion at the bucket factory, we might say that it's literally raining buckets. If there's been a vampire attack, we might say that the victim has literally been bled dry. Otherwise, we usually take that as a figure of speech that means someone has taken every last cent from them. But if we've recently been talking about vampire attacks, we might make sure to add the "figuratively" when talking about someone being bled dry in the money sense to make sure everyone knows we're not talking about another vampire attack.
When someone is "livid with rage," she's pale, not red in the face. It means the color has drained out of the face.
This literally means to kill one in ten. If you say that the Black Death decimated the population of Europe, you're actually understating the case. It's often used to mean "kill a lot of people," but publishing people tend to be word nerds, and while you might not get dinged for the "kill a lot" use, if you use "decimate" properly, you may get bonus points.
"Troop" vs. "Troupe"
A "troupe" is generally a group of performers, while "troop" is used for discussing the military. Misuse of this can cause massive giggle fits at the mental image. Or it did for me in a manuscript I critiqued once, so much so that I still remember this one. Then again, Busby Berkley did start his career designing military drills …
Of course, nobody's perfect and editors aren't going to reject you because you've put an apostrophe where one doesn't belong. But today's publishing climate is very competitive and you don't want to shoot yourself in the foot with preventable errors. They're less likely these days to buy a manuscript that requires a lot of editing, and a lot of word usage errors in the opening pages signals that your book might require a lot of work to get it in shape. I'd also caution you about how well you edit any other writing you have in public, like blog posts or tweets. Editors have been known to Google authors, and while they know that the Internet is more casual and autocorrect sometimes inserts errors into tweets, if your blog posts show that you don't know the difference between common things like "it's" and "its," then they may have doubts. Readers may also wonder about a writer who has issues with basic grammar.