Tile Chimney Diameters: Correct Size?

We have a split block chimney stack that contains the second story fireplace flue and a downstairs fireplace flue. Upstairs the flue is a 12" by 10" brown tile liner. This fireplace never smoked and now has a Merrimack insert with 6" ss flue insert. Works fine, but with some smoking when loading new fuel. Downstairs in the master bedroom is the small fireplace with its own 8" square tile liner. This fireplace has ALWAYS smoked into the room. It has glass doors when we keep shut and sealed (sometimes smoke from the upstairs fireplace vents down into the bedroom). So we rarely burn wood in downstairs. A friend who used to install fireplaces said the 8" flue may be too large for the relatively small firebox. He suggested a 5" ss liner installed. The distance from firebox to exit at the roof is probably 20 feet. Is his assessment correct? And can a liner be placed and somehow sealed at the bottom, without breaking out the damper?