Speed is the enemy when loaded, or any time for that matter. The heavier I am, the slower I go.
The truck stops pretty well when loaded. You can hardly tell it's that full.
Trailers are a different animal. They can bite you if you turn your back.
Different strokes for different folks. Given the caveats of properly maintained, functional, and correctly adjusted trailer brakes, I actually feel safer with a heavy trailer than an overloaded truck. Trouble is, so many trailers are NOT well-maintained. Also, in PA we are forced to have brakes on ALL axles of a any trailer with 3500 lb. axles or greater, so my tandem stops pretty damn fast. Very few trailers ship with this configuration by default, and few states require it, trailer dealers in PA are always adding this as a required dealer upgrade.
On the subject of having your brake controller properly adjusted the load, it should be just a few tenths of a volt shy of locking up at 25 mph, when you push the proportional lever to full max without touching the brake pedal. If not properly adjusted, then yes... that trailer could bite you.
I will grant you that the one time I was forced to pull the tandem loaded to the max with concrete block, behind my lite 1/2 ton truck in heavy snow on un-treated roads, I was really nervous about having the trailer come around on me in a stopping situation or rounding a bend in the road. However, it actually pulled, cornered, and stopped just fine. I guess guys pull 55,000 lb. trailers with 25,000 lb. semi tractors in the snow everyday, not sure why I was so nervous about it, but it was an experience I'm not anxious to repeat.