
Have got somebody under your skin
Negative Connotation
Often implies annoyance or displeasure rather than physical irritation. 
Emotional Impact
Describes something causing ongoing emotional agitation, not just one-time annoyance. 
Persistent Effect
The phrase emphasizes that the effect remains for a considerable time. 