The outfit that sold me on joggers properly was not a lazy Sunday one. It was a pair in a heavy grey marl, straight leg, worn with a plain white shirt tucked in and old trainers, running to the school gate and then straight on to a coffee with a friend afterwards. Nobody looked twice. That was the moment I stopped thinking of them as something you only wear when nobody's going to see you.
Where they fall apart is when you try to dress them up the way you'd dress up any other trouser. A silky cami and heels on top of joggers looks like you got halfway through getting changed and gave up, and a structured blazer with a very slouchy tracksuit bottom usually just looks like two outfits arguing. The combination that works is one plain, slightly fitted thing up top, a shirt, a simple jumper, a fitted tee, and let the joggers be the relaxed half of the outfit rather than fighting for attention with something equally loose. Same goes for the full tracksuit set. The matching top and bottom already does a lot of visual work, so anything else loud near it, a big logo jacket, a print scarf, just clutters it.
Over months, the difference between a good pair and a bad one shows up at the cuffs and the knees before anywhere else. Cheap ribbing on cuffs and ankles loses its grip within a few washes and ends up hanging rather than sitting close, which changes the whole shape of the trouser. I wash mine inside out on a cool setting and skip the tumble dryer where I can, because heat is what flattens the fleece lining and turns a soft pair bobbly and thin looking by the second month. A set that's still holding its cuffs and still soft on the inside after a winter of proper wear is worth more than one that looked slightly nicer on the first day.
Where they're the wrong choice is anywhere you need to look like you made an effort without a story attached, a work meeting, a smarter lunch, anything with a dress code you haven't been told but can feel. No amount of good tailoring in the trouser changes what a tracksuit communicates in a room like that, and trying to override it with jewellery or heels just reads as mismatched rather than smart.
The one mistake to avoid is buying a size up thinking it'll feel more relaxed. It just drags at the knee and pools at the ankle, and a jogger that's genuinely cut generously in the right places needs your actual size, not the next one up, to hang properly.