I don’t think you’ll be able to see/ or measure any performance penalty from having so many kerning pairs. However a lot of kerning pairs is usually (not always) an indication, the font is poorly spaced or has other design flaws. Can you please post a screenshot of your font in action?