Part of your problem is just the inherent loss of heat that is going to come with an exterior chimney. Personally, I would prefer a stove outside of the fireplace rather than an insert, to reduce that loss.
Obviously you want to maximize your insulation to minimize heat loss. It would seem to me that if you had an insulated, airtight, well-sealed blockoff plate at the level of the old damper, then nearly all the heat ABOVE that plate would be generated by the liner itself. If the liner was insulated right down to the blockoff, then the heat should be minimal and the loss should not be too great, even though some heat certainly would build up and be lost through transfer from the brick to the outdoors.
Reclaiming heat or preventing its loss from a fully insulated liner does not seem like a worthwhile effort, in the smoke chamber or anywhere else. On the other hand, if heat is being lost from the actual stove, and that heat is not coming into the living space, you have a problem. But I don't know ways to approach that beyond: 1) insulated blockoff plate to contain most heat below damper area; 2) insulated fireplace to prevent transfer to exterior, and 3) blower/fans to circulate air and transfer heat into the living space.
