The mistake is treating a jumper dress like a jumper that happens to be long, and sizing it the same way you'd size a normal knit, which is how you end up with something that sits two inches above the knee and clings in a way that no amount of tugging at the hem fixes.
A good one earns its stretch back every time, which sounds like a small thing until you've owned the other sort. Cheap knitwear dresses go baggy at the seat within a few wears because the yarn has no memory in it, so by week three you've got a dress that bags behind and clings in front, which is the worst possible combination. Wash it the way the label actually means, not the way you wash everything else in a hurry, because a hot wash will felt the fibres and you'll get a dress a size smaller and twice as thick. Lay it flat to dry, always, hanging it will give you sleeves down to your knuckles and a hem that's suddenly asymmetric for no reason you can explain. Bobbling happens to every jumper dress eventually, but it happens fast and badly to the cheap acrylic ones, especially under the arms and across the bottom where a bag strap or a chair rubs all day. A proper wool or wool blend bobbles slower and shaves off easily with one of those little fabric combs, so it's worth keeping one in a drawer rather than pretending it won't happen.
As for what it clashes with, anything else chunky. A jumper dress with thick tights, a padded coat and a scarf on top can tip you into looking like you're wearing all your clothes at once rather than one good outfit. Thin tights or bare legs with boots does more for it than another layer ever will.
On fit, the shoulders and upper arms are where it goes wrong, not the waist, which is where most of us check first out of habit. If the shoulder seam sits even slightly down your arm, the whole dress will look like it belongs to someone else, however nice the knit is. Try it sitting down, not just standing in the mirror, because a jumper dress that's fine standing can ride up alarmingly once you sit. The difference between a decent one and a poor one is mostly in the knit itself, a close, even stitch that doesn't go see through across the chest when you stretch your arms up, proper ribbing at the cuffs that keeps its bounce, and a hem that's deliberately finished rather than just the knitting stopping.
It will never be a smart dress, whatever it's styled with, and the sooner you stop asking it to be one, the happier you'll both be.
Worth a look next: silver dresses and sweater outfits.