
You had me worried
Emotional Connection
This phrase often shows a speaker's deep concern for the other person, indicating a close relationship. 
Intensity Varies
The phrase's intensity can vary based on intonation and context, from mild concern to extreme anxiety. 
Not Literally Worried
Sometimes used hyperbolically to mean 'you caught my attention' rather than actual worry. 