In Chapter 1 of The Origin, near the end, he writes:

Pigeons, I may add, can be propagated in great numbers and at a very quick rate, and inferior birds may be freely rejected, as when killed they serve for food. On the other hand, cats, from their nocturnal rambling habits, cannot be matched, and, although so much valued by women and children, we hardly ever see a distinct breed kept up; such breeds as we do sometimes see are almost always imported from some other country, often from islands.

It's amazing what cat fanciers have done with Felis Cattus in the last 150 years, innit? Cats don't vary nearly as widely as dogs do, but then dog breeding has haad a few thousand years' head-start.