Growth degree classification for finitely generated semigroups of integer matrices