As Andy B says, standardisation is the key. Just think how difficult it is with AA, AAA, C, D, PP3, PP9s at the moment, and that's before you get into button cells/watch batteries.
Until/unless everyone used the same battery pack, then you risk turning up in the middle of nowhere for a battery swap and the garage haven't got your 'type' in stock. Presumably they'd remove your battery, charge it, and put your battery back in, which won't be 5 minutes.
You've also got the problem that a small city car might have a 33KW battery, whereas a luxury long distance car has a 100KW battery, so even once standardised, they would need to be made up of (say) multiple 25KW or 33KW sub assemblies.