Can someone clarify the rule...

On free kicks? It seems the generally accepted course of events nowadays is free kick is won, opposition player sprints over to where the free kick is to be taken from, stands there for as long as possible, argues with referee then slowly walks backwards to about 5 yards away preventing both a quick free kick being taken and usually preventing it being taken in the preferred direction. Is it just me that gets annoyed by this? Frustrates me so much, surely it should be like rugby, if you concede a free kick onus should be on you to retreat. By the time the free kick is taken usually there's no advantage at all, and it just slows the game down so much. Rant over.