The Universe Today takes a look at the satellites of the "dwarf planet" Pluto, Charon, Hydra and Nix. The last two smaller moons were only recently discovered by the Hubble Space Telescope in 2005 and named the next year. The currently accepted theory of Pluto moon formation is much like that of the Earth's; a major impact event of two objects chipped Charon from Pluto, with the other satellites remaining as leftovers from the collision.
However, the orbits of the two smaller moons lead this theory into question. If they were the result of debris from the impact event, their orbits would almost certainly be irregular and elliptical. However, they are not only both in a near regular circular orbit, they are in resonance with Charon, with Nyx orbiting Pluto 4 times for every Charon orbit and Hydra 6, mcuh like the Jupiter system moons of Io, Europa and Ganymede. The question then remains if they could have simply formed from a cold disk of dust that may have remained from the impact. Again, however, the modelling researchers have conducted of the impact event leads us to believe such a dust disk would be very compact around Pluto - far inside the orbits of the two small moons.
Another possibility is that the two moons formed quite close to Pluto but later migrated due to gravitational interaction between Pluto and Charon, but the simulations conducted so far don't seem to support that idea either. Thus, we are left with the only other solution; that they formed outside of the Pluto-Charon system and were likely captured at some later point as Pluto orbited the Sun.
While the research will continue, it is likely that no definitive answer will be forthcoming until the New Horizons probe visits Pluto in 2015. And perhaps not even at that time.