It's entirely possible that there's a non-skeevy explanation, and also we have no way of knowing how many actors were auditioned/seriously considered and how many of them were of color (or just how far "consideration" of PJ playing the part actually went in reality). At the same time, though, yeah, it would've been cool to have a non-white Doctor as more than just a theoretical exercise, and it would suck--though hardly be unprecedented thinking--if either Team Cardiff or the Beeb decided they "needed" another white guy of a certain age bracket.