Only you and your financial advisor, family, accountant, etc. can answer the "should I?" question because there are many more factors that aren't in the assumptions you included, and a lot of them relate to your own gut feel.
It would be easy if paying off a mortgage were ONLY just another investment. Then you could decide whether investing in that house is better or worse than putting the money in cash, an income property, gold, the market, bonds, etc. But a home has a lot more aspects, including being one of the biggest "musts" in retirement - after all, you'll need somewhere to live.
If the housing market stays pretty good, you will have parked those funds and saved the interest costs, which are always a nasty way to erode your savings. And you can always access the money again in an emergency with a new mortgage - as long as the bank doesn't discriminate against you on credit as a retiree and the market hasn't fallen apart. Meanwhile, you can also enjoy our home debt free.
Let's split ourselves into three people for a second. Flash forward 23 years. Person 1 did what you're contemplating, and the house is worth more, less or the same, but is still in good shape and there is one less bill to worry about. Person 2 put the money in the market and lost 70% of it, still has the mortgage and doesn't have the cash either. Person 3 bought a second house, fixed it up, sold it for a 50% profit (or bought an apartment, or started a small home business, etc.), still paid off the house and had some nice extra retirement cash or an enjoyable small business. If both Person 2 and 3 sound like a pain, paying of your mortgage may be the right choice for you.
Interest and inflation are both silent retirement killers, and controlling interest should provide peace of mind and be a great return at the savings of 6.5% - minus, perhaps a 3% "guaranteed" opportunity cost and an inflation adjustment.
If you lay out all the factors to your accountant and family, your gut will give you the best direction. Many people who pay off their mortgages feel it's the smartest thing they've ever done. That has to do with how they feel, not a bunch of numbers!