Operational and analytical use cases all face the same problem: they are unable to reliably access relevant, complete, and trustworthy data from across their organisation. Instead, each use case typically requires cobbling together its own means for accessing data. ETL pipelines may provide a partial solution for data access for data analytics use cases, while a REST API may serve some ad hoc data access requests for operational use cases.
However, each independent solution requires its own implementation and maintenance, resulting in duplicate work, excessive costs, and similar yet slightly different data sets.
There is a better way to make data available to the people and systems who need it, regardless of whether they're using it for operations, analysis, or something in between. It involves rethinking those archaic yet still commonly used ETL patterns, the expensive and slow multi-hop data processing architectures, and the "everyone for themselves" mentality prevalent in data access responsibilities. It's not only a shift in thinking but also a shift in where we do our data processing, who can use it, and how to implement it. In short, it's a shift left. Take the same work you're doing (or will be doing) downstream, and shift it left (upstream) so that everyone can benefit from it.
A data lake is typically a multi-hop architecture, where data is processed and copied multiple times before eventually arriving at some level of quality and organisation that can power a specific business use case. Data flows from left to right, beginning with some form of ETL from the source system into a data lake or data warehouse.
The medallion architecture is the most popular form of multi-hop architecture today. It is divided into three different medallion classifications or layers, following the Olympic Medal standard: bronze, silver, and gold. Each of the three layers represents progressively higher quality, reliability, and guarantees, with bronze being the weakest and gold being the strongest.
The medallion architecture and, indeed, all multi-hop architectures have serious problems. The complete article examines why.
This content is an excerpt from a recent InfoQ article by Adam Bellemare, "The End of the Bronze Age: Rethinking the Medallion Architecture".
To get notifications when InfoQ publishes content on these topics, follow "Architecture and Design", "Event-Driven Architecture", and "Cell-based Architecture" on InfoQ.
Missed a newsletter? You can find all of the previous issues on InfoQ.
Sponsored
|
|
Help us tailor InfoQ's content to better serve software architects like you. Share your insights on the technologies and tools shaping your work by completing a quick, three-minute survey. Your input will directly influence the topics we cover, ensuring our content remains relevant and valuable. As a thank-you, you'll have the chance to win one of five complimentary video access passes to QCon London (Apr 7–9, £435 value). The survey is anonymous, and all responses remain confidential.
Take the InfoQ 2025 Technology Adoption Survey, sponsored by InfoQ.com
|
|
Upcoming Events
InfoQ and QCon: For practitioners, by practitioners
QCon London 2025 (April 7-10): The full schedule is live
Explore 125+ technical talks from senior software practitioners on AI, platform engineering, security, and more. See the full agenda, plan your experience, and save with early pricing before March 11. Explore the schedule & register today.
InfoQ Dev Summit Boston 2025 (June 9-10): New speakers added
Hear from active senior developers at Netflix, CarGurus, Thoughtworks, and more. Get real-world insights on scalable architectures, resilience, AI, and platform engineering. Register before March 11 to save with early pricing. Meet the speakers & register now.
QCon San Francisco 2025 (November 17-21): Explore the provisional tracks
Get a first look at the key topics shaping QCon San Francisco. Provisional tracks explore scalable architectures, AI/ML in production, developer productivity, and more. Save with early pricing before March 11. Preview the tracks & register.
InfoQ Dev Summit Munich 2025 (October 15-16): Returning for its second year
Back for its second year, InfoQ Dev Summit Munich brings together 25+ senior developers to share how they're solving today's toughest software challenges. Save with launch pricing before March 11. Register now & secure the best price.
InfoQ Dev Summit New York 2025 (December 2025): Coming soon
A new city with the same focus on real-world software development insights. InfoQ Dev Summit New York will bring together senior developers to share practical strategies and lessons learned from experience. More details coming soon — stay updated.
