My last contract was very restrictive in terms of IPR. Basically if I spent any time on personal projects in work, or using work equipment in any way, the IPR for the development work belonged to them. I also had to give back all information that the company owned on departure. I also needed written permission to engage in any external activity that could cause a conflict of interest with my employer.

However it is down to them to prove that you have material (your development) which they have a right to own. If someone else has told them this, it's basically hearsay, which means that unless you've admitted in verbally or in writing you can maintain a dignified silence and it's unlikely they'll be able to do much about it.