I get it. Frequency response-wise, yes, you can make things sound similar with a frequency response profile. You can absolutely make two different speakers measure the same on an RTA. But in no way does that tell the whole story. You could probably record a sound that originated from a trumpet and compare it to a synthetically created trumpet sound that measures the same as far as frequency response but the character will easily be picked off by the human ear for which one is the simulation. This is in the time domain and is part of the "character" I'm referring to.
Remember that rise and decay thing you referenced? Drivers also store and release energy in different ways from each other. You can only see this in a waterfall plot (some can pick it off in the impedance plot as well). Some are ultra-transparent with their incredible rise and decay times(ribbons, planars) and some are thick sounding without much fine detail being released (woofers that play somewhat high but have resonances and inductance issues). Not many people compare drivers in this regard, using the waterfall plot. Best example I could hope to use as an illustration is a theoretical setup in which two floor standing speakers were mic'd and proven to be providing the same frequency response on axis (we'll stay there) and they are matched in every possible regard to play evenly in level and all that jazz. One is a true ribbon wideband full range driver and the other is a conventional wide band full range driver (for simplicity sake). One of those is going to possess a rise and decay time faster than the other. Thus, the presentation will be more refined and more transparent. One will reveal smaller changes elsewhere in the system that the other may not be able to resolve those details well enough to alert you (or sound different). Drivers also have different polar responses and will fill a space with energy differently because if it. Different speakers have different reactive profiles and will cause an amplifier to present/struggle in different ways because of it.
So many other things and this is really so tricky to discuss because it is ALL about hearing the difference. This is what it's like when you heat a difference in amplifiers that are delivering roughly the same power. And it is by no means a striking or glaring difference. It's more of a subtlety and not likely something you would hear with a short term comparison. It's usually something you notice only once it's gone (if that makes sense). Worse, if you've always had a speaker that doesn't offer the best performance as far as being truly transparent, you were never very likely to hear a difference in the presentation of amplifiers. The subtleties may have always been masked by the limitations of the driver because it was always the limiting factor in system performance. There was a time when I was unaware of this, too. It's something you're much more likely to notice if you had been rolling around with the same gear for years and years, and you then changed speakers that have the same reported frequency response, and really do sound similar in their presentation, but yet there's something to your ears that is lacking. Could simply be the modes traveling in the cone causing energy storage that masks finer details and you don't find it to be as good as the speakers you were just running. Or if you had been rolling around with the same gear for years and years, and then finally switched over to a new amplifier that was proven to deliver the same wattage. You didn't change your drivers or your DSP settings (if applicable) but you somehow heard a difference. The amplifier may not be built the same and can't cope with the reactive profile of the driver in the same fashion as the original amplifier did. The presentation might be slightly different simply due to the "grip" the amplifier has over the driver(s). Best way I can describe that is getting 20 watts from a 25 watt amplifier versus getting 20 watts from 250 watt amplifier. The bass character will not sound the same inside of that 20 watt limit between those two amplifiers. I know, I know; that's loaded. But try it.