When choosing a place to relocate, people take all sorts of factors into account. School systems, the job market, the housing market and even the scenery all take center stage in the discussion. But there’s one issue that often gets left out and can affect your finances more than all the other factors combined: tax rates.

But do tax rates really differ that much from state to state? Is it really something people should be taking into account when deciding on where to make their new home?

The short answer: a resounding yes. Which state you live in can mean thousands of dollars saved or lost each year. Read on for a list of which states to avoid. (For related reading, see: States With the Biggest (and Smallest) Tax Burden.)

New Jersey

The home of Atlantic City and Bruce Springsteen is also home to some of the worst tax rates in the country. Not only does the Garden State have one of the worst income tax rates at 8.97%, but it also has one of the highest average property tax rates at 2.38%.

Ohio

Flyover country typically has some of the lowest tax rates, but Ohio seems to be the exception to the rule. The Buckeye State is number 4 in the country for income taxes, and its average property tax rate is 1.55%.

Vermont

One of the smallest states in the Union has one of the biggest tax burdens for its residents. Vermont has an income tax rate of 8.95% and a property tax rate of 1.71%—some of the highest averages in the country. (For related reading, see: Overall Tax Burden by State.)

Rhode Island

The state that only takes up 1,212 square miles places a premium on its land—the average property tax rate is 1.67%. The income tax rate doesn’t help either; it’s high at 9.9%.

New York

It should be no surprise that living in the Empire State is pricey, but did you know how expensive it can truly be? The average effective property tax rate is 1.5% in New York while income tax rates are 8.82%.

Wisconsin

The Badger State has harsh winters AND high tax rates. Wisconsin is number 10 in the country for income taxes and has a property tax rate of 1.97%. So much for cheap living in the Midwest, right?

Maine

Living in a state full of fresh seafood has its benefits, but also its disadvantages. Maine has a property tax rate of 1.27% and its highest income tax rate is 7.15%—number 11 in the country. But it may be worth it for the crab cakes alone. (For related reading, see: The Best States to Retire to for Tax Reasons.)

Minnesota

The Land of 10,000 Lakes could also be called the state with the Great Tax Rates. Minnesota ranks number 6 in the country for state and local taxes and its top income bracket has a tax rate of 9.85%.

Massachusetts

The state of the Kennedys, Harvard University and the New England Patriots has a lot to offer tourists. Residents, on the other hand, only get high taxes. Its property taxes are slightly above average at 1.18%, but it ranks in the top 10 for personal income tax collection.

The Bottom Line

Fleeing one of these states to avoid taxes might be an extreme measure, but this list can at least give some insight into where not to move if you want to avoid a significant tax burden. But if you live in one of these states and want to reduce the taxes you pay, talk to a financial advisor or tax expert. They can pinpoint where your burden is highest and help guide you through the process of reducing it. (For related reading, see: Investing in Property Out of State.)

Want to learn how to invest?

Get a free 10 week email series that will teach you how to start investing.

Delivered twice a week, straight to your inbox.