Way back when, when I first decided I wanted to be a writer, someone said something very interesting to me. It was something I'd never heard of before, and was really shocked to hear it then -
"It's costs a lot to get a book published, you know."
Now, I was about 17 or so at the time, and though I would find I had a lot to learn about publishing, the thought of having to pay to have a book published never really crossed my mind. Isn't it the other way around? That's what I thought, but hey - I was 17 and still had a lot to learn...
Well, fast forward x amount of years and I do know better. No. You don't pay to have Avon or Harlequin or Kensington publish your book. You write a great manuscript, and they pay you.
When it costs you to have a book published, you've been published by a vanity press (publisher). Plain and simple. Or, if your book comes out in one format (i.e. -electronic) and there's a setup fee for a print copy of that same book - that's a subsidy publisher. Any time your money flow toward the publisher, you are paying to play and that's vanity publication.
Now, that's not to say it's wrong. In certain situations, for certain niches, vanity publishing is perfectly acceptable. Maybe you wrote a family history and want to give it out in book form at the upcoming family reunion. Well, vanity publishing is fine in this case.
But, if you're a serious writer, hoping to have a career in the field, don't confuse it with commercial publishing. Agents and editors do not consider a vanity publication to count as a legitimate writing credit. A vanity-published book will not gain you entrance into certain writers' organizations - or, might exclude you from the published author factor of certain organizations (such as RWA's PAN - Published Author Network.)
In fact, depending on what vanity publisher you use **coughcough** PublishAmerica **coughcough** , it could, in fact, hurt you. Hurt your credibility as writer. And that's because certain vanity publishers try to give the impression that they are a traditional publisher (which, in fact, was a term that never even existed until a certain vanity publisher - see above mentioned coughs - coined it).
So, bottom line is, if you aren't serious about a writing career, or your book's target audience is so limited that is isn't commercially viable (but not necessarily a bad book, per se) a vanity publisher may be the way to go. But, if you are serious about a writing career, remember this one golden rule, courtesy author James MacDonald - "Money flows toward the writer. Not the other way around."