It comes from the idea of getting a wheel stuck in mud. You just don't seem to get anywhere when that happens - the wheels just keeps spinning but going nowhere. So, when you say you are "in a rut", it means you are just spinning or doing stuff but not getting anywhere or nothing is changing.