I think the biggest factor in making a series work is planning. If there's even the slightest possibility that your book could have at least a single sequel, it helps to be thinking about it up front, since if your book is successful, the publisher will likely suggest a sequel, and a series is a good way to build a readership.
One thing to think about is the kind of series that might work best for this world/cast/story.
There's the serial saga kind of series, with one big story broken up into multiple books, like the Song of Ice and Fire series by George R.R. Martin (known on TV as A Game of Thrones). Each individual book may not follow any of the structure of a novel and may not have a resolution. In fact, it may even have a cliffhanger ending. This kind of series might be considered to have a closed ending because when the story's over, it comes to a definite conclusion. But if your characters are still alive, it's possible to start another saga in the same world or to tell more adventures. Readers pretty much have to start with the first book and continue to the last one to get the whole story. Otherwise, it would be like reading a few chapters from the middle of a novel.
There's the more episodic kind of series, in which each book is a self-contained story, but there may be character and plot arcs that stretch across books. You see this a lot in mystery series, where each book covers a particular case, but the personal life, character development and relationships of the main character build and grow across books. A reader could pick up any book in the series and follow the main plot, but readers who start with book one will appreciate the character parts of the story more, and readers will keep picking up each book not so much for the main plots but to see what's going on with the characters.
You can also do a hybrid of these, as in the Harry Potter series or the first five books of my Enchanted, Inc. series, where there's one main villain with the same big-picture goal, but each book tells a mostly self-contained story about a particular facet of the villain's scheme, and that story is resolved in that book, while the villain lives to scheme again.
Or there's the series model common in the romance genre, where there's a large cast of recurring characters and each book focuses on a different pair of characters. The best friend of the main characters in book one may be the hero of book two, with the characters from book one taking a supporting role. We get to see what's going on in the lives of characters from earlier books while focusing on someone else. This has been done with a group of friends, a military unit, a family, or a town. I had a friend writing a family series who ran out of family members she'd established and started having to dig up long-lost cousins.
Depending on the genre, it is possible to switch series modes -- start with an episodic series that becomes more of a serial saga as the big-picture plot starts to eclipse the "case of the week" story, or close out the saga and then tell episodic adventures. I haven't seen it done that I know of, but it may even be possible to do the romance-style series until you reach the point that a particular character wants to take over, and then to continue the series focusing on that character. Still, it helps to know up front if you're going to need to have more of the main story to tell, a reason for this character to keep getting into trouble and having adventures, or if you're going to need to find the story for a secondary character. What further development does each character need to go through? Are the relationships set at the end of book one or evolving? What is there in this world that could continue to cause problems?
If you're doing a saga, it really helps to plot it in advance as though it was a novel -- who's the big bad, what's his goal, what's his plan, what are the major turning points, where will everyone end up, how will you break it into books. It would be really hard to write this kind of series as a seat-of-the-pants writer since you're essentially publishing your earlier chapters before you've written the rest of the book, and that means you're stuck with what you've established.
If you're doing an episodic series, you need to make sure you've set up a reason for your characters to keep having adventures. Not only does their situation need to lend itself to things happening, but the characters need to have a motivation to keep getting themselves in trouble. It's easy if it's their job -- they're a sword for hire, a cop, a detective. It's harder if it's an ordinary person who keeps getting swept into trouble.
In the romance model, make sure there are other characters you want to work with who will intrigue readers. You can always introduce more people along the way, but there needs to be someone in book one who's ready to have a story told in book two.
Some other hints for working with a series:
- Keep a master list of your characters, their roles and their major traits, both physical (eye and hair color) and otherwise (tics, quirks, interests). This really, really comes in handy later. For some weird reason, it's easier to keep track of all this stuff in something you're obsessed with as a reader than in something you're writing, and it helps to have an easy reference.
- I've heard of writers who set themselves a challenge for each book -- like using a different genre model, focusing on a particular aspect of the main character's personality, developing something in the key relationship. Readers may not even notice that this is going on, but it's something that keeps the writer interested and makes the writing fresh.
- Really dig to explore various aspects of the main characters or the story -- what would they never do? Find a way to make them do it. What could never happen? Find a way to make it happen. What do they value most? Take it away. What do they think they value least? Take it away and see what happens. A lot of readers get frustrated with a series character who never seems to grow or learn anything, in spite of everything she's been through.
- Put the characters in different settings -- make your main character a fish out of water, or if your character is already a fish out of water, move her to her home turf and switch roles with the other characters. Take everyone out of their comfort zone occasionally.
- Beware the exploding cast. It's easy in a long-running series to have so many recurring characters that it's hard to fit them all in. Sometimes it helps to change the setting or find a situation that allows you to focus on one smaller group of people while the others sit on the sideline. Then in the next book you can find something for that group to do while the others get a break. This all gets really tricky in a saga with a big cast of key characters rather than one main hero because readers have their favorites and will lose interest if their favorites are offstage too long.
- Finally, don't be afraid to end it. If you've been doing it right, your readers will beg for more, but don't keep going if you're getting tired of it, if you don't really have any good ideas or if there's something else you'd rather be writing. It will show in your work, and then your readers will be disappointed. As they say in show biz, leave 'em wanting more, and unless you killed everyone, you can always revisit that world if you find yourself inspired.