It could be him showing you that he has had enough and is bored. How many repitions do you do before he takes the mat?
Holly did this when I was teaching her to retrieve a dumb-bell, if I asked for too many repititions she would run off with it (and start to chew it which is not good at all) so I learn't to only ask for a maximum of 4 retrieves (although I'd try to end on a good one so we might actually do just 3 if the 3rd is good, or a 5th if the 4th was not as good as I know she can do) and then move on to something else.
She would be fine if we then came back to the retrieve and did a few more.
She loves the retrive now, but I still only ask for a couple of repititions (which is fine since she only needs to do it once in competitions).
It's always tempting to try "one more time", but maybe you need to take a break after just a couple of repititions and do something else for a while.
Or you could have a leash on him and as soon as he grabs the mat he can't get away and you can take it off him and end the fun.