Been too busy to post for a while, but this was fairly quick to implement, and I dare say it's a pretty cool effect. The grotesque distortions remind me a bit of a Chris Cunningham video.
It works in (somewhat sluggish) real time. The face detection is a bit glitchy, leading to a few popping artifacts now and then, which could be easily remedied with a Kalman filter. A further improvement would be to use optical-flow-based interpolation so you wouldn't get that sort of zebra-stripe effect during rapid movements, but that would kill frame rate dead. But it would be just the thing for high-quality, offline processing.