I don't think it is necessarily fear of unknown change, either. It can be that people understand that there will be less to go round if there are more people in the country. And there are possibly/arguably less units of housing than can house everyone who needs housing. Overcrowding generally and competition for fewer unskilled jobs can also create fear.
Therefore the fear is based on quite concrete worries.
Jenny