I helped with a couple of writing workshops this summer, which meant I was reading beginnings of novels, and that has me thinking a lot about what makes a beginning work. Here are some suggestions:
1) Save the backstory for later
I think this is the number-one problem with most beginner manuscripts. Writers feel like they need to explain the entire background of their world, exactly how it works, what the history is, and the main character's entire history before the story can start. This bogs down the beginning. It's better to show all this through action as the protagonist interacts with his/her world or notices what's wrong about it that needs to be fixed. And no, characters having a conversation about the history of their world is not a way to get around this, especially if both of them know the history already. Give us just enough information to understand the inciting incident and how it affects the protagonist. We don't need to know about the hero's childhood and relationship with his parents or about the war that was fought a century ago unless that's absolutely critical to understanding the inciting incident and how the hero reacts to it. Unanswered questions and curiosity can keep readers turning pages, as long as there's eventually some kind of payoff.
2) Give your protagonist a goal
Not just a story goal, though that should come as soon as possible in the book, but some goal, wish, or need that exists before the story kicks off and that would have been there even if this particular story hadn't happened. I think this is a big reason that villains are so often more popular than the heroic characters. You know from the start that the villain wants something, while the hero is often just reacting to the villain's actions without actually wanting anything. Having a goal can also create additional conflict for the protagonist because that initial goal might conflict with the story goal, and he has to make a choice.
The initial goal might be something that's focused and expanded by the story goal -- when we meet Luke Skywalker in the original Star Wars, he wants to go to the Academy with his friends, mostly because he wants to get off that sandbox of a planet and have adventures. Then after he meets Obi-Wan Kenobi, his story goal is to get the Death Star plans to the Rebels so they can destroy it. So it fits his initial goal of getting off the planet and having adventures, but in a more purposeful way. The initial goal can also be contradicted by the story goal. In the movie version of The Wizard of Oz, Dorothy initially wants to go "somewhere over the rainbow." She gets that very quickly, realizes she doesn't really want it, after all, and her story goal is to get back home. And the initial goal might be in conflict with the story goal. You see that a lot in romances, where the heroine initially just wants a promotion at work or to get a date with the guy she likes, and then she meets the hero and is torn because she likes him, but being with him requires her to reconsider her other goal.
It's also good if the protagonist's story goal is positive and proactive, rather than reactive. Instead of the goal being to stop the villain, which puts the villain in the driver's seat, there's a specific plan for stopping the villain that has nothing to do with the villain's actions. Going back to Star Wars, the goal is to get the plans to the Rebels and find a weakness rather than the goal being simply to stop the Empire, which would have had them running about, reacting to everything Darth Vader did.
3) Avoid flashbacks
I was surprised by how many manuscripts I critiqued this summer that started with some big, exciting incident and then flashed back to the start of the story. This is a common TV series storytelling structure that I think has become overused even on TV, and I don't think it works as well in a book. If you're using it, you should have a very good reason for doing so that's integral to the story, and you have to be careful about how you use it.
On TV, they can get away with starting with a big, exciting scene with a cliffhanger ending, then after the opening credits flashing up the "24 hours earlier" caption and going back to the beginning because unless you've recorded the show, you're stuck watching as it goes. In a book, it's easy to just skip ahead to see how that cliffhanger comes out. In a TV series, usually the viewer already knows the characters pretty well and knows what to expect of them, and that opening scene is usually shocking because it's something we know those characters would never do (like the cop hero shooting his partner), so we want to keep watching to find out what the real story is. In the opening of a novel, unless the novel is part of a series, we don't know enough to know whether or not this is shocking behavior for these characters, so it loses impact. This kind of opening is also bad about losing all energy once you leave that opening scene and flash back to the beginning. It's easy to put the book down then because everything that comes afterward seems boring in comparison. Plus, I think too many writers use the exciting opening as an excuse to then start the main story a lot earlier and fit in a lot of backstory before getting to the inciting incident, like if they start with a bang that gives them a license to be more leisurely about starting, and that makes matters even worse. And I suspect that this looks to editors and agents like a ploy to send the most exciting part of the book while sticking to the request for the first chapter or first three chapters.
Unless you really know what you're doing and you have a solid story reason for doing so, it's better to start at the beginning, as close to the inciting incident as possible, and find a way to make that opening scene interesting or exciting in its own right.