Second homes are damaging communities and fuelling a housing crisis in the South West, the National Housing Federation has warned.
The federation (NHF) fears the high number of second homes in the region is pushing up house prices and pricing local people out of the market.
It also claims the large proportion of second homes that are empty is becoming detrimental to local businesses struggling to survive outside the holiday season.
Cornwall became the second homes capital of the country earlier this week when new figures were released by the Office of National Statistics.
The statistics published for the first time showed that 22,997 people listed a second home in the unitary authority area in the 2011 census – more than for any other council area in England and Wales.
The authority also recorded the greatest number of holiday homes, according to the nationwide report, with some 10,169 used for holidays.
Dan Rogerson, Liberal Democrat MP for North Cornwall, is pushing the Government to restrict the number of second homes in the region by changing planning laws.
"There are too many second homes in Cornwall. Many people come for two or three weeks a year and spend a lot of money but not during the rest of the year.
"The knock-on effect of high numbers of second home owners is to drive up housing prices and increase housing waiting lists."
Mr Rogerson said: "We need an acceptable number of second homes according to the area. Often places with a big chunk of second homes also have a long waiting list for social housing. I will need support from the other parties if we are going to have the power to make anything change."
In Devon, almost 35,000 people usually living elsewhere listed a second home in the county.
The South Hams – with 7,672 second homes – was in the top 20 local authorities for the number of people with a second address, of which 3,738 counted it as a holiday destination. In East Devon there were 7,227 second homes.
One second-home owner in Devon for 16 years, who did not want to be named, said high numbers of second homes was "the lesser of two evils".
He said: "I understand why people hate us as the character of many communities is changing.
"But there is a lot of wrongly directed anger and frustration towards us when in many cases we help to restore towns and bulk up the tourism industry.
"Local authorities should take the blame for destroying thriving local businesses by giving planning permission to large supermarkets out of town."
Sarah Wollaston, Conservative MP for Totnes, said people shouldn't "demonise" second home owners.
But she admitted there was a problem with totally uninhabited properties in some parts of the South Hams where there were "no lights on". Hugo Swire, Conservative MP for East Devon, said it was a "lazy argument" to blame second home owners for people struggling to get on the housing ladder.
"We need a more flexible housing market, involving more affordable housing and encouraging Rent to Buy."
Ben Bradshaw, Labour MP for Exeter, said: "Second homes are a red herring for a chronic shortage of affordable housing in the South West."
The NHF warned the lack of new homes being built will send rents soaring by up to 48% across parts of the South West by 2020.
Home Truths 2012 found the cost of renting an average home is expected to surge to £1,064 in some areas, leaving thousands forced to rely on housing benefits.
Jenny Allen, lead manager for the federation in the South West, said: "The high number of second homes across the South West has helped push prices up, taking them out of reach for local people.
"If families and young people in the region are priced out of their local villages, it can have a hugely damaging impact on community life, with village shops, schools and pubs closing as a result.
"Unless we build more affordable homes in the South West, local people will continue to be priced out of rural areas, and the countryside will increasingly become a place for the well-off to enjoy at weekends."
More than 10,000 of those second homes were listed as holiday homes across the eight authorities in Devon.
In total, 130,055 people usually living elsewhere listed a second home in the South West region.
