Because the writers simply didn't understand the characters.
Unfortunately she was boiled down to usually being the voice of reason, very sensible, and occasionally saying y'all. AJ wasn't written terribly in later seasons, but it felt like they reigned in all her quirks such as being competitive and badass. She just became "the sensible down to earth one" in most cases.
Probably had her character watered down the most. She went from being curious, dorky and a bit of a know-it-all to just being, again, voice of reason character who loves BOOKS BOOKS BOOKS! and sometimes freaks out over tiny details. Twilight has always been a bookworm but in the earlier seasons we saw why she loved books so much, and believed that she had a deep interest in reading and learning. In later seasons she just likes books because.
similar to Twilight, the writers turned her into a very one-note character who just loves fun and parties and adds comic relief to every scene. They forgot why she loves fun and parties, and what that means for her character. The worst example was "the last laugh" where Pinkie has no idea what her purpose in life is outside of throwing parties, despite the fact she sung an entire fucking song about it in Season 2.
To be honest, you could argue Dash was always the most inconsistently written outside of particular episodes. She was always a bit of a cunt, but in the earlier seasons they made more effort to show that despite that she is loyal and actually very skilled as a flyer. In later seasons she just became the "WHOA SO COOL, YAWN THIS IS BORING" character without any of her positive qualities to balance that out.
As you say, Rarity and Fluttershy were probably handled the best. Rarity was generally written okay even in later seasons, her dialogue became a bit more stale but her character developed fairly well. Fluttershy became more assertive, which is a good thing, but at the same time lost a lot of the humor of her character in the early seasons.