The duplicate value is listed in column ID or in column OrigID. So you also have to delete the rows where the duplicate value is listed in the OrigID column.

delete FROM #Table
WHERE ID in ( SELECT r1.id FROM #Table r1
  INNER JOIN #Table r2
  ON r1.ID= r2.OrigID
  WHERE r1.Col3 = r2.Col3
  )
OR OrigID in ( SELECT r1.id FROM #Table r1
  INNER JOIN #Table r2
  ON r1.ID= r2.OrigID 
  WHERE r1.Col3 = r2.Col3
  )