For security and sanity, I would put them all on the same domain. However, this is not absolutely necessary.

SQL Replication will work using SQL Authentication. You can configure your transactional replication using SQL Logins and it should work fine. Since you are using a shared snapshot folder, you will probably have to configure folder access allowing permissions to "Everyone" (hence your security problem).