On the “fear factor” level, few scenarios match the nightmare of falling behind on your taxes and having to deal with the weight, might and power of the IRS to square your tax ledger.

Make no mistake, the IRS is an institution that tosses nickels around like manhole covers – and they will come after you for any back taxes owed to Uncle Sam. In fact, the IRS collected $31 billion in late taxes in 2013.

With that kind of cash on the table, the IRS has a separate enforcement and collections division that focuses strictly on recouping back-tax money owed by individuals and businesses.

And the agency is not afraid to unleash its powers of enforcement on late payers.

“I think the main reason that people fall behind in filing their taxes is that they are feeling afraid,” notes Robert Seltzer, a Los Angeles-based Certified Public Accountant (CPA) and tax specialist. “This fear is based on their not being able to pay the balance. Otherwise, people are flakey and disorganized and not filing their returns on a timely basis is a symptom of that.”

But late payers should know what really annoys the IRS – and avoid those habits.

File Even if You Can’t Pay

“I explain to my clients that late filing penalties are much higher than late payment penalties,” Seltzer adds. “Part of this explanation includes letting them know that the IRS is much more forgiving with people who can’t pay as opposed to non-filers and that payment plans are available. I also tell them that knowing what they owe has to be better than not knowing.”

Other tax specialists agree.

“File anyway,” says consumer finance expert Kevin Gallegos, vice president of Phoenix operations with Freedom Financial Network. “Even if you owe back taxes, file your 2013 return,” he says. “The penalty for failing to file a tax return is 10 times higher than the penalty for failing to pay taxes.”

It’s also important to get a big picture grasp of your emotions, your health and your state of mind when you’re dealing with a back taxes issue.

So let’s take this time to analyze your step-by-step moves when the IRS comes calling, looking for some of your hard-earned cash.

Communicate Right Now 
Step one might as well be Rule One: Do not put off calling or contacting the IRS about your situation. The more you put it off, the more it will cost you in interest and penalties on what you owe. The IRS will even appreciate your forthright attitude in coming to them to work something out. Don’t fear it, embrace it; you will not go to jail. But, neither should you shirk your responsibility, because that will get you into deep trouble with the IRS.

Contact a Reputable Tax Specialist
At this stage of the game, you need professional help. You may already have your own tax accountant, and that’s fine, as long as he or she has done a quality job. Above all else, don’t wait to get help. Years of back taxes gain interest and the numbers may be significantly higher the longer you wait. Consequently, you need to begin paying on these amounts quickly to minimize the total interest you will have to pay.

Work out a Payment Plan 
The IRS will sit down with you and work out payment arrangements to which you will have to adhere in order to pay what you owe them, plus interest and any penalties. Sometimes they do this via the mail and phone. You might not even have to visit an IRS office. They are aware of what you owe, and will give you the opportunity to begin to make payments to them. When you submit a request to the IRS for a payment plan, you will have a better chance of success if you:

  • Let the IRS know you’ll pay the debt off within five years – but ideally within two years.
  • Aim higher. The monthly payment you negotiate with Uncle Sam should be equal to or higher than what the government believes it can garner from you from you from a negotiated agreement initiated by the IRS.
  • The regular (usually monthly) tax payment you introduce to the IRS should be tied to existing IRS criteria. For example, you should subtract household living expenses from your total income. Then cut a check for the difference to the IRS.

Also, be creative about catching up on your back taxes.

“Tap all your resources,” advises Freeman. “In a worst-case scenario, tax debt can be paid with a low-interest credit card, a home equity line of credit or a loan from a family member. While not highly recommended, these solutions can be cheapest in the long run because they help the taxpayer avoid the IRS penalty fees (but not interest).”

Stick to Your Payments 
Another key rule - do not fail to make your payments on time to the IRS. This one organization will stop at nothing to get its money. If you violate the terms of your payment arrangements, the IRS will attach and seize property that you own, including bank accounts and even the mortgage on your home. If you speak with them, however, in the event that you are having problems making your payments, you will likely be able to work through it.

Watch out for Scams
Most taxpayers have seen those television commercials where someone, who claims to have once been an IRS employee, can get your IRS debt settled for pennies on the dollar. Each person’s circumstances are different and just because someone else had a major portion of his or her back taxes wiped out does not mean that you will have the same experience. If you choose to contact one of these businesses, you might want to first do some research about that company. The IRS is a good place to start asking questions, as the agency knows what it’s willing to do and whom it’s willing to work with. 

Another good tip: stop digging. “Start paying current taxes first and then work on old taxes,” says Daniel Morris, a senior partner at Morris & D’Angelo, a San Jose, California-based accounting firm. “And get professional help if you’re unsure – it’s always a good idea to have a tax professional review your case.”

The Bottom Line

Falling behind on your tax obligations is the financial equivalent of a root canal. Take the pain away by following the above tips, and get square with Uncle Sam.

Want to learn how to invest?

Get a free 10 week email series that will teach you how to start investing.

Delivered twice a week, straight to your inbox.