Green Screen Handbook by Jeff Foster
I recently read The Green Screen Handbook: Real World Production Techniques by Jeff Foster.
The first part of the book has a lot of interesting (though not necessarily practical) history and trivia about compositing in general, and color keying specifically. But the real meat of the book is after that.
Jeff doesn’t just show you how to use After Effects and other software packages to do real-world green screen work (though he does do that). He also shows how to do all of the crucial things that you must do before you even turn on your post-production software—or even turn on your camera.
The good folks at Sybex/Wiley, who publish the book, have made a couple of chapters freely available on their website so that you can check the book out. These two chapters are certainly worth a read.
In Chapter 4, “Basic Compositing Techniques”, Jeff explains the basics of mattes, color keying, and rotoscoping, and he introduces Keylight, which is a core tool for color keying work in After Effects.
In Chapter 16, “Fixing Problem Green Screen Shots”, Jeff shows how to deal with poor lighting, color spill, reflections, and other banes of color keying work. He shows how to use garbage mattes, hold-out mattes, and a variety of other tools and techniques.