NOS tubes are the most expensive initially, but over the long-haul may represent a good investment. Newly manufactured tubes can sound good as well, if carefully chosen. However the quality of new tubes can vary greatly. Don't take anything for granted, as not even all new tubes arrive in working condition (especially some rectifier types). In general, you should experience longer tube-life with NOS tubes.
Good, used tubes are another option. But, be careful here, Most people who sell used good tubes just plop them in a tube checker, and if the checker says the tube is good, they sell it as a "good" used tube.
The only foolproof way to determine if an audio tube will perform well is to use it in an actual live circuit. Many tubes that check good on a tube checker can sound bad in an actual live amp. A tube checker cannot tell you if the tube might be noisy. Also, a tube checker cannot predict how a particular tube is going to sound your amp.
Use common-sense tube troubleshooting techniques. If you suspect a tube problem, you can take the tubes and swap them from channel-to-channel (at least on stereo circuits) to see if the problem stays in the same channel, or switches channels when you swap the tubes. Be sure and label the tubes with a "Sharpie" marker and notate your changes carefully.
Visually inspect your tubes. A tube that is in good working order usually has a clean and clear look to it. Any tube in which the getter (the normally silver coating on the inside of the glass envelope) has turned frost-white has begun to lose vacuum and should be discarded and replaced immediately.
Sometimes a tube will exhibit a blue "glow," when in use. This blue or "purple" haze can be vague or quite pronounced. It is caused by gas leakage in the tube. When electrons pass through the gas, ionization occurs. This is the beginning of the end for the tube, but it will usually continue to work for some time.
Coming soon: "How important is it to match or grade tube sets?"
Highly recommended web viewing: "The Cool Sound of Tubes," By Eric Barbour, as published in the IEEE Spectrum, August 1998 edition.