The most frustrating aspect of poker for many players is dealing with variance, where even correct decisions can result in losses due to bad luck. This unpredictability can test emotional resilience and lead to frustration, especially during prolonged downswings. Additionally, encountering slow play, disruptive opponents, or feeling outplayed in key moments can add to the challenge, making poker both mentally and emotionally demanding.