If you're stood there holding three completely different dresses and asking which one counts as "casual", the honest answer is that it depends what your week actually looks like, not what the hanger says. A jersey midi that skims rather than clings, a shirt dress you can undo a button on, a smock shape with pockets deep enough for your phone and keys, all of that counts. The test I use is whether you could sit on the floor with a kid or a dog and not spend the whole time tugging at the hem. If you can, it's casual. If you're wincing, it's smart dressed down, which is a different thing entirely.
This kind of dress earns its stripes on the days with no real shape to them, the supermarket run, the school pickup, a long lunch that turns into an afternoon, a festival where you need pockets and no zips to fight with. It's brilliant for spring and early autumn especially, when you want to look like you made an effort without actually making one. Where it falls apart is anything that needs real structure underneath it, a work presentation, a funeral, anything where you need to stand up straight and look like you mean business. A soft jersey dress will not do that job, and asking it to just makes you look like you got dressed in a hurry, which, fairly often, you did.
Don't pair a casual dress with anything trying too hard elsewhere, a stiff blazer kills the whole point of it, and so do heels that need concentrating on. Trainers, flat boots, a denim jacket thrown over the top, that's the register it wants. Chuck a structured bag or proper tailoring into the mix and you've got two outfits fighting each other rather than one that works.
In the shop, forget how it hangs on the rail and check what happens when you move. Sit down in it. Reach up as if you're getting something off a shelf. See if the fabric is thick enough that you're not thinking about what's underneath by lunchtime, and check the hem doesn't ride when you walk fast, because you will walk fast, you'll be late for something. Cheap jersey goes thin and see through within a few washes, so hold it up to the shop light before you buy, not after. A good one will still have body to it the second winter you own it. It still won't get you through anything that needs a blazer.