I've never tried running for extended periods without the blower since it seems quite noticeable it doesn't put out much heat into the room without. But your reasoning wouldn't work on my stove... unless I'm trying to burn wet wood my air control is all the way to min all times once its through startup until I'm getting ready for reload.
I'd agree with your observations for short-term fan use, but over a longer term I've found that by not using the fan I get much more heat into the surrounding brickwork which results in a steadier heat output. I know that I'm the the luxurious position of having an oversized insert, so I don't need to get every last BTU out of the burn, but I really don't think I'm losing much heat by not using the fan. Once my room is over 70, the fan goes off.
Hmmm I do often forget about the thermal mass of the huge brick work in my great room, it does get slightly warm close to the insert but it just doesn't seem to get warm enough to do much good. Maybe I will have to test shutting the blower off just to see, but I can't imagine the slightly warm bricks being able to balance my poor insulation and whatever drafts are in the house. What I am more concerned with is if more heat is being wasted up the flue if a blower isn't used. But I'm just guessing there, I have never measured the exit gas temps. My insert is probably undersized though, so I am usually in the position of needing everything its got!
