My company has our own flavor of Agile. When we initially started brainstorming about adopting Agile last year, one thing we all agreed is that the first initial sprints will be design sprints. We try to come up with conceptual/logical design (alteast the core entities and relationships) in the design sprints. As we start getting into the development sprints, we flush out the detail design.

We completed four projects so far. Seemed to be working fine so far. It is not that scary as I initially thought.

Good luck