We have shown experimentally that dispersion due to an intra-cavity electromagnetically induced transparency or gain medium reduces the sensitivity of the cavity resonance frequency to a change in its length by a factor which is inversely proportional to the group index. Since the group index under atomic coherence can be made extremely large, the sensitivity of a long path length optical cavity can be reduced significantly. This can help in constructing highly frequency-stable cavities for various potential applications without taking additional measures for mechanical stability. The results also establish indirectly the opposite effect of increased sensitivity that can be realized for a negative dispersion corresponding to a group index close to a null value. These effects are discussed in the context of potential sensitivity enhancement of a rotation sensor.