Cancellation is covered nicely here:
http://www.betteraudio.com/geolemon/Phasing/Phasing.htm
Anyways....the main problem with mixing sub sizes isn't the cancellation issues (you can have cancellation anytime you have more than 1 speaker playing the same frequency).....but rather the necessity of it.
#1; Unless done "right" (which can work, by the way), the two simply won't sound good together. They'll have different frequency responses, different output levels, etc etc...which overall will result in a very poor, muddy, unfocused sound.
#2; With the quality of todays subwoofers; there really is no need for it. The "main" reason people do it is because they are working under the misconception that they need a 10" sub for the "punchy bass" and a 15 for the "low end".....all in one setup. Well, pretty much any quality subwoofer will be able to satisfactorily play the entire subbass bandwidth by itself, regardless of diameter....there is no need to break it up between drivers by frequency range.
Now, how can you make it work?? As someone mentioned; with careful tuning of the crossover so that they play different frequency ranges, using them in separate boxes and different alignments, etc etc.....But, IMHO this is pretty much a waste of resources, time, effort and real estate because, as I said, any quality subwoofer will play the entire subbass region just fine.