It's possibly because when the church was built 1000 years ago, there were a lot of stones lying around that were gathered up to use in the walls. Cottages would have been wooden framed and wattle and daub. Later buildings tend to be made from stone all cut from the same quarry, so it's more uniform.

At least that's what happened in the little Welsh town where I live. The oldest building has examples of all the different local stones whereas the stones in the church are all the same, as are the stones in the bank.