There are definitely some drawbacks here. You know what groups and individuals have access, but there is no way within SQL Server who are members of a Windows group. I believe that there are some PowerShell scripts/modules out there that get that information. Here's a link to a blog post that includes session information from a SQLSaturday session that included getting permissions using PowerShell.