Pilloried for the decisions they give and castigated for the ones they don't, referees do not have an easy job.
Even with the technology today, there are lot of controversies.
There was a major one today in the Six Nations as Wayne Barnes awarded James Lowe's try based on the evidence given.
However, another angle was later shown that appeared to show the Ireland wingers foot in touch.
Over the following slides, we look at 5 most debated refereeing decisions from down the year's.
1. Speargate on the Lions Tour
The first game of the Lions tour of New Zealand was swept into controversy following a horrific spear tackle on Captain Brian O’Driscoll, which ended his tour within minutes. The tackle, with O’Driscoll picked up by two players in the ruck, was against the rules, but the ref waved play on.
The moment caused a major knock-on effect in World Rugby, tightening up restrictions on spear tackling.
2. Mark Cueto's try
Matthew Tait sliced open the Boks defence and hurtled towards the line. Andy Gomarsall moved it quickly to Mark Cueto on the left wing who dived through the tackle of Danie Rossouw and over the line.
Alain Rolland called for the TMO. It was tight. Impossibly tight. No try.
England were instead awarded a penalty which Wilkinson slotted, and roused by Cueto’s diving effort they had the momentum.
But the Springbok nut was impossible to crack. Montgomery’s boot kept a huffing and puffing England at bay, guiding his side to a 15-6 win.
South Africa were deserved World Champions, England the plucky, dethroned former kings.
But if Cueto’s ghost try had stood, it all could have been so different.
3. Kick it, Bernard
During a game between New Zealand and Australia in September 2022, Foley received a warning from the French referee for wasting time when he delayed kicking the ball into touch.
Leaked sky cam footage showed that as Foley was running down the clock, the referee stopped play and told him to hurry up before restarting the clock and again urging Foley to continue.
As Foley's teammates frantically gestured for him to get rid of the ball, the impatient referee intervened and gave the All Blacks a scrum put-in. New Zealand then scored through Jordie Barrett on the right flank to secure a 39-37 victory.
4. Sam Warburton's red card
Sam Warburton, a Welsh rugby legend, received a red card in the first half of the 2011 Rugby World Cup semi-final against France.
Warburton, who played as an open side flanker, was shown a red card by referee Alain Rolland in the 18th minute of the match for a tackle on French winger Vincent Clerc that Rolland deemed worthy of a straight red.
The decision was met with outrage from commentators and pundits, but both Warburton and Wales head coach Warren Gatland, while expressing their disappointment, remained gracious in defeat after France secured a 9-8 victory in Auckland.
This controversial decision, which many believe cost Wales a place in the final against the All Blacks, remains one of the most debated calls in the history of rugby.
5. Wayne Barnes - NZ v France 2007 RWC
Many Kiwis still haven't gotten over Barnes' performance in the 2007 RWC quarter-final against France.
Barnes missed a clear forward pass from Damien Traille to Freddie Michalak in the lead up to a France try.
It led to the All Blacks' earliest ever exit from a World Cup.