If your wallet is groaning from too many credit cards, you may have given some thought to cancelling one (or more), especially if you're paying a hefty annual membership fee for a rewards card that, given your spending habits, isn’t generating much in the way of rewards.

But breaking up with your credit card, while easy to do, can be a painful misstep.

“There are very few reasons to cancel a credit card,” says credit expert John Ulzheimer, who has worked at Equifax, one of the big three credit bureaus, and FICO, the firm that runs the calculus that determines credit scores used by the most lenders. “If you want to break up with a credit card you need to be cognizant of what trouble this seemingly benign action is going to cause.”

Credit Score Impact

The main risk when you cancel a credit card is the potential negative impact on your credit score. In the land of FICO credit scoring, your debt-to-credit utilization ratio accounts for 30 percent of your score. That’s a mouthful of a term, but it’s a simple math equation: the sum of all your credit card balances divided by the total credit limit on your cards. There is no magic ratio you want to shoot for, but lower is better.

The problem with cancelling a card is that you immediately lose that available credit limit.  “Even if your spending doesn’t change, your credit score could fall because your utilization rate goes way up,” says Ulzheimer.

Read more about credit cards

“If your utilization rates jumps from 30 percent to 60 percent you could see your credit score fall 50 points, or more,” says Ulzheimer. That’s a severe haircut that could cost you plenty more than the annual fee.

FICO credit scores range from 300 to 850. A score of 760 puts you in good shape to qualify for the best offers, if you expect to be in the market to buy home or car insurance, or are planning to take out a loan. Recently, someone with a FICO score of at least 760 might be able to qualify for a 3.6 percent interest rate on a 30-year fixed rate mortgage. If a higher credit utilization ratio caused the 760 to drop to 710, the borrowing cost would rise to more than 3.8 percent. On a $300,000 mortgage that could add about $40 a month to your mortgage payment, or nearly $500 a year. 

Credit History Consequence

A less worrisome side effect of cancelling a card is the impact on the part of your FICO score that looks at the length of your credit history. This accounts for 15 percent of your FICO credit score. When you cancel a card, that history stays on your credit report—and thus is part of your FICO credit score calculations-for another 10 years. As long as you have another card that you keep for the next 10 years, this part of your score should not be seriously impacted a decade from now when the cancelled card is no longer factored into your length of credit history.

While no annual fees are obviously preferable, Ulzheimer suggests a big-picture perspective is useful when deciding if you should cancel a card. “Paying $79 a year to have a credit line of $10,000 to $15,000 is a pretty cheap way to have access to credit. Do you really need to cancel a card for that?”

Cost-Cutting Tactics

If the effects on your credit of closing an account make you leery, try these steps.

Ask for a fee waiver. If your main problem with keeping a card is the annual fee, Matt Schulz, senior industry analyst at creditcards.com, recommends calling the bank and asking for it to be waived, or at least lowered. In a recent survey, Schulz said that just one in four cardholders bothered to try and negotiate loan terms. Yet more than 80 percent of people who asked for their annual fee to be lowered or waived were successful. “People don’t realize what power they have to negotiate,” says Schulz.

Ask for a card downgrade. If there’s no give on lowering your fee, Schulz recommends asking the bank if it has another card you could switch to that doesn’t charge an annual fee. It won’t offer the same rewards or perks, but if you were using the card infrequently, that’s not a problem. The advantage of doing this rather than cancelling is that your credit history moves from the old card to the new card and you maintain your credit limit.

Shred it. If you don’t want the temptation of having more credit within reach, Ulzheimer suggests shredding a card, rather than cancelling. “You keep the credit limit, but can’t use the card.” If you change your mind down the line, just call up the card issuer, say you lost the card and you’ll have a replacement card mailed to you lickety-split.

If You Decide to Cancel

Sometimes cancelling makes sense. For instance, divorcing couples with joint credit cards should aim to close down those accounts. Or maybe you’ve come to the realization that a $450 membership fee for American Express Platinum or Citi Prestige, is a bit too steep to swallow if you’re not getting value from the service and rewards, and your request for a reduction/waiver was rebuffed.

If you do need to cancel a card, aim to offset the loss in that credit limit with additional credit on other cards. The easiest move may be to call up your remaining card/cards and ask for your credit limit to be increased. Or apply for a new credit card that will give you a similar credit limit to the card you are cancelling but charges no annual fee.

Opening a new card might cause a small temporary ding to your credit score. “I wouldn’t worry about it,” says Schulz. “It’s not nothing, but it’s nothing like the impact of having your credit utilization rate go way up when you cancel a card, which can have a far more negative impact.”