Negative pressure from fans pulling cold air in through HVAC ducts

I am using a couple of fans to blow cold towards my stove. This is generally working well. However, today I noticed that in the two rooms with fans blowing air out of them toward the stove, cold air is being drawn in through the HVAC cold air return.

Cold air is coming "backwards" through the system and into the room at a pretty good rate. I assume it is cold because it is being pulled through ducts in the rather cold basement. When I turn the fans off, the backward draft out of the cold air return stops, so it appears that the negative pressure created by the fan is the culprit. Anyone else have this problem?

The cold air returns are not equipped with shut offs. (The warm air vents are). I'd also rather not close them off because I do use the central heat system. It kicks on during very cold nights and when I am away from the house for too long in the day. (My stove is quite small.)

Ideas, folks? Thanks.

The warm air part of the system is metal duct work. The cold air part is some kind of composite material (not shiny). The cold air part of the system is definitely not very air tight in the basement. I could try to improve that, but it would take a good bit of work and I am not sure how much it would help unless I also insulated all of it because they run through the cold basement. The basement is much colder than it used to be because the furnace down there doesn't run much (and leak heat).

I turned the fan speed down, as suggested. That seems to have slowed the back-draft a lot. We'll see if the heat still gets moved around enough. I'll also ponder the other options some more (covering the vents and sealing/insulating the ducts). Thanks for the comments so far.