same reason why not all dinosaurs were megafauna, and in fact nowadays birds are the survivors, I guess. Many such cases of death by being swarmed by numbers, quantity over quality. Being bigger only means you're a bigger target.
Besides, I know that Tokugawa does speak about Pickle in terms of humanity in general, but (my headcanon is) that he was unique even among his peers, a mutant, a sort of "Yujiro Hanma" of his people. Or maybe he's not homo at all and we just look similar, all the rest have died in the asteroid impact (when the food disappear and they adapted by losing muscle mass).