You can not restore a SQL Server 2008 backup to a SQL Server 2005 system regardless of the compatibility mode of the database.
If you need to move a database from SQL Server 2008 to SQL Server 2005 you will probably need to do the following:
1) Script the database in SQL Server 2008 with NO 2008 features
2) Run that script on the SQL Server 2005 system
3) Use SSIS to transfer the data from 2008 to 2005