As I contemplate what needs to be rewritten in The New Project, I've found myself pondering some common advice from all those how-to books on writing I read. In most of the books (so this isn't just one author's quirky rule), they talk about what makes a scene good, and discuss the ideal structure/format: The active character in the scene needs to have a scene goal that's a subset or a step on his story goal, and he should try multiple ways to achieve that goal in the scene, with none of them working, so that he has to keep trying something new. The scene should end in some kind of disaster -- either not achieving his scene goal or achieving it and having that actually be bad for his overall progress. Then he reacts to that disaster and develops a new goal that leads into the next scene. The hero should never get exactly what he wants (unless it turns out to be bad for him) until the very end of the book.
Now, that does mean you've got scenes with lots of tension, drama and conflict, but if you really did that in every scene, I think you'd have a boring book because nothing would ever really happen. There would be no forward momentum. Yes, things are supposed to get worse for your characters as the story progresses, but that doesn't mean they can't make forward progress or ever achieve a scene goal. Since I like Star Wars examples, consider the scene where Luke and Obi-Wan go to the cantina. Their goal is to find a ship and pilot to take them to Alderaan. And they do. There's some tension in that there's first the guy who wants to kill Luke, then Han drives a hard bargain, and then they have to hurry out of there before the Imperial troops catch them, but they achieve their goal without even having to resort to plans A, B, C, D, etc. I don't think you could call the outcome a disaster, as they get what they want and Han and Chewbacca turn out to be valuable allies. I guess the only "disaster" is more personal, in that Obi-Wan would have probably had a longer life, and they'd have both had somewhat easier lives if they'd never found a pilot to take them into the action, but in story terms it's not a disaster at all, in that achieving that scene goal moved them a step closer toward achieving their story goal, which was to get the plans of the Death Star to someone who could use them (and, ultimately, destroy it, but that's an escalation of their goal that they aren't considering at that point).
Actually following this writing advice, you'd get Luke and Obi-Wan not finding any pilot at all in the first cantina, so they'd have to go to another one, and then the first pilot they found wouldn't want to take them, so they'd have to try another, and they'd have to come up with some way to persuade him, but then it would be a disaster because he'd turn out to be a crook who'd cheat them, force them into slave labor, or maybe inform on them to the Empire. And there goes half the movie with no forward momentum toward solving that pesky Death Star problem.
So, it would seem that, every so often, you have to let your heroes achieve their scene goals simply as a way to get them into the action and so you can focus the bulk of your story on aspects that actually matter. It's just as bad to have a deus ex machina that hinders your heroes unnecessarily as it is to have it help. You only want conflict and obstacles keeping your hero from something when they matter or mean something in the broader sense of the story. You don't want a marching band strolling through the scene, aliens landing, a bomb set by someone who doesn't have anything to do with the story, or any other random event that exists solely to keep the hero from achieving his scene goal.
Of course, the reason I was thinking about this was that I realized I had a scene where my hero achieved his scene goal pretty easily, and that made me wonder if I'd done something wrong, or if it was okay. I rationalized that it was all about getting to that point, so delaying it would be pointless, and achieving that goal is going to massively complicate his life. But then I realized that this is something the bad guy wouldn't want him to achieve, so he might have something in place to stop the hero, and I don't really have a Threshold Guardian in this story. This is the hero's first big test, so maybe it should be more difficult. Or, perhaps the difficulty should come later.
Another good Star Wars example might be the scene where Luke and Han rescue Princess Leia. They have a plan for getting into the detention area, and it totally works. They deal with the obstacles quickly and easily and are able to get to the Princess and get her out of her cell. Then that's where the obstacles come up, as they have trouble getting her out of there. I think that's actually a clever bit of writing because you're expecting the rescue itself to be hard, and when it goes smoothly, you almost relax. And that's when things suddenly take a turn for the worse as they reach the part for which they don't have a plan.
Perhaps, as usual, the trick is in knowing the "rules" well enough to know when and where to break them deliberately as a way of subverting expectations. The audience may not consciously know the rules, but they're aware of how things tend to go in books and movies. It can be fun playing with that -- make the scenes where the audiences tenses and expects things to be hard go easily, then make the next part harder.
It also seems to be that where you need the goal ending in disaster is more at the end of a sequence rather than each individual scene, so that scenes can work out okay as long as they lead to that disaster. Back to my first Star Wars example, the scene about finding a ship is part of the sequence that begins with Luke deciding to go with Obi-Wan to Alderaan. The goal in this sequence is to get to Alderaan, and first they have to get to Mos Eisley, which they do without any snags like running into Sandpeople. They face the obstacle of the stormtroopers on the edge of town, which they get past with Jedi mind tricks. They get to the cantina and find Han and Chewie, and hire them as pilots. They have to evade more stormtroopers to get on board the ship and take off. They evade Imperial ships and get into hyperspace. Then they get to Alderaan and -- disaster! -- it isn't there anymore, and they get taken on board the Death Star. Then they have to come up with an entirely new goal and plan for how to get the Death Star plans to someone who can use them. In a sense, all those successes in individual scenes leading to that point are mini-disasters, in that they wouldn't have run into that major disaster if they hadn't had those successes.
There's also a layering of problem/solution, so that they're never completely in the clear, even when things are going their way. When they get safely to Mos Eisley and get past the stormtroopers, they still need to find a ship and pilot. As soon as they find a ship and pilot, they learn that the stormtroopers are onto them. As soon as they escape on the ground, they have ships coming after them in space.
I still don't know what I'm going to do with this scene in my book, though. I do know that part of my problem is that after all that careful outlining and development I did, I seem to have gotten carried away and skipped some parts that I think will intensify that scene once I get there. Note to self: When you make an extensive outline, it might help to check it every so often as you write.