Bathroom exhaust vent

There should be a flapper door at the fans exhaust that closes on it's own when the fan is off. Is the fan box in the attic? If it is, make sure it is covered with insulation.

I installed a bathroom fan last winter and could see no reason why it wouldn't be ok to insulate it. I caulked around the box opening, stuffed insulation around the sides and over the top.

Yeah, my second floor bath fan vent has a gravity damper in it. The only thing is that when we have windy days my damper will flap back and forth with the wind. The wind drafts the damper open and closed.
The ducting for the vent is sleeved in insulation, I don't get worried about heat loss there, it's just the flapping flapper that kills me.

I would think if you have warm air leaking out of the bathroom exhaust fan your bathroom would be plenty warm since the warm air from the stove would have a path leading right through the bathroom to the vent opening. All that warm air flowing through the room would have to heat the room up.

If I read your post correctly, you have a damper somewhere in the exhaust duct? Most bathroom exhaust fans also have a damper built in the unit itself at the point where it connects to the duct. Ti determine if it really is the exhaust fan just tape a piece of plastic temporarily over the face of the exhaust and see if that makes a difference.

I suspect your problem is simply that it is difficult to get air to exchange out of a small space through a single, small door.

Make sure the fan is installed correctly. You may have some sort of trim piece covering where the unit actually cuts through the ceiling plane. The gypsum board should be sealed tightly to where the unit cuts through your ceiling.
I had a very cold bathroom for a while caused by: 1. backdraft damper in fan was permanently stuck open, and the exhaust pipe did not have a backdraft damper where it exited the roof. 2. fan was installed in oversized hole, so cold air was directly entering the bathroom around the fan itself. 3. tub drain has a hole cut in floor above an uninsulated cold basement. Stack effect pulled cold air up under tub through the hole (space under tub connects over to plumbing chase connecting into to cold attic space). Cold cast iron tub helped make bathroom cold.

FYI...be very careful about covering an existing fan with insulation. The fan must be designed for insulation contact/coverage. Even my new exhaust fan specified in the manual that it could only be covered to a certain depth by insulation.

Try a small fan sitting on the floor near the bathroom blowing towards the stove (on low speed) and see if that makes a difference. It should get the warm air circulating.
Jim