This is one of those bugbears that frustrate me too.
Good news though! From what I’ve heard, it’s something the upcoming Trends feature will fix! It should be rolling out to more users soon. Whilst it can’t replace everything summary does just yet, it does handle these situations much better, and will hopefully evolve over time to incorporate more of what Summary does in a much better way.