I think you're making an assertion that's not necessarily backed up by facts.

Do you know how the dictionary stores keys internally? Ad you sure it's not copying the struct/key out as a reference type? Do you know how fast the struct and its values can be hashed?

Be interesting to actually measure and time the two different approaches.