You can certainly apply permissions to a custom set of AD users to a report or folder, but as far as I know you cannot create a custom group within SSRS and add users to it. SSRS native mode uses Windows security so you would have to either create a custom AD security group and add the members there or create a security group local to the SSRS machine and add the AD users there.