First of all, what OS do you have? Not very many OSs supports both SQL 2008 R2 and SQL 2017, and if it supports SQL 2008 R2, the OS is quite old by now. So that alone could mandate moving to a new server. Or do you consider upgrading the OS in place as well?
Another factor is hardware - it may be time to move to new hardware. Then again, if SQL Server runs on a virtual machine, that's a not an issue, as you simply can move the VM to a new host any time you like.
I used to be in favour of moving to new an instance, as that permitted me to keep the old instance as a reference, in case of performance regressions which is not unheard of with upgrades. But this has changed with Query Store. Here is a good approach no matter which path you take: enable Query Store once you are SQL 2017. Keep the compat level at 100 for a week or two. Then flip the switch. If you get performance issues, you can force the old plan through Query Store and then investigate that query in more detail. There is also tooling for SSMS for this process.
There is one undeniable advantage with an in-place upgrade: no need to move jobs, logins etc.
So this is not a definite answer, but I leave it up to you. But I hope that there is some food for thought.