You need to remove the baffle every time you clean the stack! If you pull the pin and drop the baffle down, you will find your problem. All the crap the sweep brushed down the stack, which is now laying on the baffle top around the stove outlet, prolly clogging it mostly. Either that or your cap is clogged already.
. I am not having any problems with my secondary air tube(I don't think) but after reading this I have a question. I have cleaned my chimney and removed the baffle before doing that. But I did not even think about soot going down the secondary air tube. Can I check that tube by just pushing something down it?