In VoIP Carrier networks, as in other complex systems, there are two types of problems. I'll call these "Preppy" problems, and "Gambling" problems.
Preppy problems occur when you're at the limits of achievable quality within the tolerable costs.
-- A physical device fails.
-- A generally good algorithm has a memory leak.
-- A hacker finds a way to exploit a defect in your firewall.
-- VoIP through a link with guaranteed prioritization for voice packets occasionally drops voice packets when the link isn't saturated.
Gambling problems happen when you have large risks, but you're typically just lucky.
-- VoIP across the Internet has occasional audio quality.
-- You use the word "password" as your password, and don't usually get attacked.
It's important to make a distinction, because solving them is the responsibility of two different groups:
The Preppy problems are the domain of scientist, researchers, and product developers. Their goal is to push the boundaries of quality, reliability, and robustness.
The Gambler problems are the domain of every engineer or business owner. They are the risks you're taking when you run a business and assemble someone else's components into a product. For example, occasional bad voice quality due to packet drops across the Internet are Gambler problems, because there are engineering ways to prevent that problem.