The cards/odds are stacked against you
Definition 1 of 1
Pro Tip 1/3Metaphorical Use
Used metaphorically to describe an unfavorable situation, not only involving literal cards or games. 
Despite being the most qualified, the cards were stacked against her because of company politics.
Pro Tip 2/3Idiomatic Nuance
Understand it's an idiom implying unfairness or difficulty, not just numerical disadvantage. 
He tried to change the policy, but the odds were stacked against him by the administration.
Pro Tip 3/3Non-Literal Odds
The 'odds' may involve non-quantifiable factors like bias, not just calculable probabilities. 
She fought for the promotion even though the odds were stacked against her due to office biases.