The very next commit introduces a new feature to show git hunks, which
seems to have a negative impact on performance when moving the cursor
around, or adding/removing text within lines which are highlighted by
diff-hl while diff-hl-flydiff-mode is enabled.
So for now I'm pinning diff-hl to the commit right before this was
introduced.