Medallion architecture is a design pattern that organizes a lakehouse into distinct layers: bronze, silver, and gold. 

The bronze layer deals with raw data; the silver layer cleans and validates the data, while the gold layer presents you with business-ready data. You see, the data quality improves at each layer from source to consumption. 

It is the perfect solution for data quality, management, and organization. 

This guide will help you understand medallion data architecture and its layers. 

What is Medallion Architecture?

Medallion architecture is a data design pattern used to organize and manage data in a lakehouse. Its ultimate goal is to improve the structure and quality as it flows through each layer of the architecture. 

It has three layers:

• Bronze
• Silver
• Gold

Note: Sometimes it is referred to as “multi-hop” architecture. 

The leading data and AI company, Databricks, coined the term in 2019 to describe the multi-layered system. 

People believe that Databricks may have named it, but the underlying concept builds on decades of traditional multi-stage data warehousing design patterns. 

Layers of Medallion Architecture

Now, it’s time for us to delve into the three layers of medallion architecture: Bronze, Silver, and Gold. 

They are distinct stages in the data refinement process. Each of these layers has its own purpose and characteristics. 

1. Bronze Layer 

This is the first layer of the medallion architecture. All the processes begin from this layer. 

You can consider this as a foundational repository for raw data in different formats like JSON and CSV. Now, you must know that the data stored in this layer is unaltered and sourced directly from the origin. 

The following attributes may characterize the Bronze layer. 

• Unvalidated Data: In this layer, data is stored in its original form, without any alteration. 
• Add-on Storage: This layer adds new data without modifying the existing data. 
• Preservation of Historical Data: Everything from the beginning is preserved in this layer. It comes in handy for point-in-time analysis and data reprocessing. 

Now, you may wonder: how do we get data into this layer? Well, batch data ingestion is one of the various ways to get data into this layer from different data sources. Real-time streaming data ingestion is another way.

As we’ve stated, the data remains unchanged in its original format, but that doesn’t mean you won’t be able to find it. To enable efficient data management, metadata is added. 

Typically, this metadata contains:

• Ingestion timestamp
• Source identifier
• Batch identifier

These things become essential for tracking data lineage. 

2. Silver Layer

The second layer of the medallion architecture is called the silver layer. This is where data goes through cleansing, deduplication, filtering, and transformation processes. 

This process is done so that we can make the data more usable for analysis, reporting, and machine learning. However, they tend to avoid complex aggregations at this stage. 

Here is a look at the characteristics of the silver layer. 

• Improvement in Data Quality: The issues with data quality are resolved. 
• Data Structuring: In this layer, data is organized in a way that allows users to query and analyze it easily. 
• Validity: To ensure data accuracy, fundamental data rules apply. 
• Enrichment: Might add metadata to provide additional context. It enhances the data. 

3. Gold Layer

This is the final layer of the data medallion architecture where data truly shines. 

Typically, in this layer, data is organized in a consumption-ready, enriched state. The data in this layer is transformed into formats that optimize query performance. We see a lot of star schema-based data models in this layer. 

This layer is defined by the following characteristics. 

• The data in the medallion pattern is extensively processed, cleaned, and enriched.
• Data aggregation happens according to business needs. 
• It follows the denormalized structure for easier querying and improved performance. 

What is the Difference Between ETL and Medallion Architecture?

Traditional extract, transform, load (ETL) transforms the data before loading it. On the other hand, the medallion approach follows the ELT format. Meaning, it ingests raw data in the bronze layer, then applies minimal transformations in the silver layer, and finally the gold layer is used for complex transformations. 

In the ETL process, you need to perform data cleaning before the loading process, which can be a time-consuming process. That’s where the medallion methodology improves. It mainly enhances data quality as the data moves through each layer. 

This architecture also addresses the issue of data changing as it moves through traditional ETL pipelines. Original files are often overwritten in traditional ETL pipelines. In contrast, the bronze layer of medallion architecture holds the original files in the exact format. 

Benefits of Medallion Architecture

A few benefits make medallion architecture an attractive choice for modern data management. 

• Better Data Quality: In this architecture, raw and processed data are kept in separate layers. It ensures that only highly accurate data reaches the final stage. 
• Easy to Use: It has a simple, clear, and intuitive structure. That makes it easy to understand, adopt, and scale. 
• Scalability: The layered approach allows you to scale data processing. 
• Optimized Performance: The gold layer receives refined and denormalized data that enables more efficient querying and real-time insights through business intelligence. 

Challenges of Medallion Architecture

Despite having many benefits, you must not think that all is hunky-dory with medallion architecture. 

It is fraught with its limitations and challenges. 

• Storage Cost: Well, you might have guessed that this architecture stores data across all three layers. So, the storage costs could skyrocket, especially for data-driven applications. 
• Data Management: It requires you to model and manage schemas and tables separately, effectively adding complexity in data management. 
• Compatibility: This is one of the biggest gripes of many businesses. Some of them find lakehouse architecture impractical for their business. That is why medallion architecture is not compatible with what they are trying to achieve. 

Conclusion

In this data-driven world, medallion architecture is just another way to organize and manage data. Its framework addresses several big data challenges. 

Architecture-wise, the Bronze, Silver, and Gold layers form a logical progression of data quality from raw sources to business-ready data. With this, the foundation for data governance and lineage tracking improves. 

So, if you want to take advantage of medallion architecture, you must fully understand it. We’ve covered almost everything you should know about medallion architecture. 

FAQs