On the sly
Definition 1 of 1
Pro Tip 1/3Cultural Context
The phrase often implies disapproval or something not socially accepted. 
He was dating her on the sly because his parents didn't approve.
Pro Tip 2/3Slight Disapproval
'On the sly' might suggest that the action is slightly wrong or naughty. 
She'd been eating sweets on the sly, despite her diet.
Pro Tip 3/3Sneakiness
Implies cleverness or sneakiness in avoiding detection. 
Tom completed his project on the sly to surprise his teammates.