Home Forums Data Warehousing Strategies and Ideas Need help with system architecture for real-time data integration RE: Need help with system architecture for real-time data integration

  • Quick feedback on "The architects want the warehouse to receive all its data through the message bus like the individual services are currently" situation. I think this is a pretty bad idea. Data Warehousing systems are usually tuned to ingest data by bulk imports, or at least "micro-batches" for closer-to-real-time reporting scenarios. With the one-record-at-a-time inserts (per message), the record will need to contribute net new data to dimensions and facts, this is gonna be a severe overhead on a data warehouse. Perhaps the messages can be queued up in an OLTP-tuned environment, something like AWS Aurora, which can ingest 6+ million records per minute, then micro-batched into the data warehouse. That's my 5 cents on the subject.