The best investment is energy efficiency. My house is in Northern NH (Berlin Gorham area) and was built in 1988. A lot has changed with respect to energy efficiency in 34 years. NH did not have a model energy building code and builders tended to cut a lot of corners on the energy side to make a place look good as most folks didnt pay attention to energy costs. Even when NH put in model energy codes, there is no statewide enforcement and many smaller towns ignore it. If you have not done so get a home energy audit from Eversource (Public Service of NH) its the best $100 you can spend as they will give you list of energy reduction improvements to your home and they will pay for half the improvements up to $6,000. There is also going to be new federal dollars for home energy improvements. The upgrades are ranked by payback so they do the lowest hanging fruit first. Its likely they can drop you heat load by 20% or more. Typically, they seal lot of air leaks all over the house to begin with and usually spend a lot of time in the basement sealing the sills.
Homes tended to be built with way too much glass in the eighties as glass sold homes. A major improvement in home comfort is to put in cellular blinds with side tracks.
https://symphonyshades.com/22-side-tracks. The ones with side tracks can double the R value of a typical double pane window. More importantly they really cut down on drafts and radiant heat loss at night. I bought mine from the company I linked to as they are in VT. They are not cheap so the financial payback is long but the comfort improvement of being able to sit next to window on a cold night without a cold draft and the heat being sucked out of you is worth it. I do recommend the heavy corded loop option as its more durable. Alternatively for far less money folks buy sheets of iso board foil faced foam, cut it tight to fit the window opening and cover with fabric from Walmart. It works but its likely they get installed once in the fall and left all winter, plus you need a place to store them in the summer. Be careful on the side tracks, many places sell cellular blinds but far fewer sell the side tracks, there is big difference in performance, withouh side tracks the cold air tends to leak around the sides.
Now that you have reduced your heating load I guess I need to answer your original question. Sad to say there are no great options and they all will cost money to implement. Global warming concerns are coming to a head, contrary to what a politician will tell you, fossil energy costs are going to trend up but its going to be a roller coaster ride getting there and this winter is the perfect storm with the Ukraine war driving fossil prices sky high. As much as this winter could be super high prices, many big energy companies are betting that in a couple of year the prices could drop big time which is why they are not drilling.
One option to consider to get off the roller coaster is if there is bulk pneumatic pellet delivery in your area (broken link removed). If you have the space, you can install a pellet boiler and bulk pellet storage tank. A truck similar to a heating oil tanker stops by every month or so in the winter (depends on heat load and storage capacity and blows pellets into your tank). Its probably a 15 to 20K install. NH used to have a good rebate for these installations paid for by Regional Greenhouse Gas Initiative (Called REGGI) money, but the legislature cut funding the residential program. Until the legislature swings back to the democrat I dont expect the incentive to be renewed.
Be careful of lumping heat pumps together as a generic class, there are two distinct type of heat pumps. The old style air source while house heat pumps with ducts tended to be energy hogs as they werent designed for efficiency. Many switched to electric resistance heat when it went below freezing. Thus not a great option. There is new "cold climate mini split" technology that works well for moderate temps down to around 10 to 20 F, yes they will go down to lower temps, possibly -12 F but the efficiency drops down to electric heat and the heat produced is not great (lots of slightly warm air being blown around). Still 20 degrees and above they are potentially twice to 4 times more efficient for heating and also a lot more efficient for cooling as there are no ducts. Mini splits are space heaters, like a pellet stove they heat and cool the space they are in. Ducts leak and frequently run through unconditioned spaces so a lot of the heating and cooling get lost on the way to a room. What most people do is pick one room to heat in the winter with mini split and then keep the rest of the house at a lower temp. Odds are you will have occasional cold snaps in your area for below 20 F so you need some sort of backup but 95% of the year its hassle free heat.
A possible backup that has been forgotten about is a vented "kerosene" heater. (broken link removed) Contrary to name the new designs can be run off of low sulfur heating oil or diesel in pinch(the old ones needed to run on Kerosene). They are a space heater like a pellet stove but can be run off a bulk fuel oil tank so home deliveries can be arranged. They need yearly service like an oil furnace. There are dealers in NH, use this link to look one up (broken link removed) Many folks use them with minisplits as primary heating sources but for homes with basements they may need some sort of supplemental heat to keep pipes from freezing (less likely if the energy audit was done. There are similar propane units but unless you a have huge propane tank installed, heating oil storage takes up a lot less space than an equivalent btu content of propane. I think that is important in NH as propane deliveries can sometimes be an issue in cold weather in NH.
I used to be a big advocate of net metered solar with minisplit heat pumps but the state has watered down the net metering program for new installations to the point where I would need to look closely at the numbers as they are a far less attractive. REGGI used to fund an incentive for solar installations but thank the current legislature for not funding it.
If your time window is 10 years or longer in the house IMHO the pellet boiler may be the best option but you need to look at if there is going to be reliable pellet delivery. The Berlin/Gorham area gets supplied by a big distibutor in Bethel Maine but unsure what options are in your area.