As our bike parts have matured some standards have not. Gears, brakes, headsets, tire widths, materials have all evolved. I’m a wheel builder so I’ll stick to what I know and not even comment on the whole bottom bracket dysfunction.
A wheel profile is a triangle formed on 2 sides by the spokes and on the third side by the width of the hub flanges. As the flange length changes so does the wheel stability. In the case of the rear wheel gears have been added from 7 to 8 and hub flanges got closer together. Then spacing got closer together for 9 and for 10 speed. In the meantime we added a disc brake rotor to the nondrive side further pushing the hub flanges closer together. Besides chains getting narrower and wearing out faster, shifting getting more finicky because the gears are spaced closer together creating less margin of error, and dirt not shedding because tolerances are so close, wheels are less stable because the hub flanges are closer together. Mountain bike wheels also got bigger around making the spokes side of the triangle longer and the wheel less stable. Mountain bikes used to be 26” now there is mostly 27.5” and 29er. Gravel and road wheels are 700 or the same diameter as 29er but now they sport disc brakes and more gears too.
Thankfully hub spacing is changing. The old 130mm road 135mm mountain frame spacing has mostly been replaced. Lots of it with 142mm. Be careful though some manufacturers just made spacers for their existing hub shells. This doesn’t change the width of the hub flanges so it doesn’t increase wheel stability. Then there is also boost and now super boost which from what I have measured are increasing flange distance. This wider stance helps with wheel stability.
The advancement of thru axles is helping wheel stability also. Bigger axle, more clamping surface = less flex. Campy did a great job designing the QR but wheels have changed. That wheel profile triangle has changed. We all like our mega gearing range, disc brake, 29er wheels. The bike industry just needs to take a moment to catch up to itself where we can have a standard hub spacing again. Till then a pair of hand built wheels is still the best upgrade you’ll make.