Doris Data Model (Tables, Partitions, Column Families, etc.)

1. Table Structure

Doris uses table structures to store data. Table structures include:

  • Table Name: Each table has a unique name.
  • Row Count: The maximum row count for each table is limited to 10^18.
  • Column Families: Column families are a key concept in Doris, representing the types and attributes of a set of columns in a table.

2. Column Families

Column families are an important structure in Doris tables, defining the characteristics of each column, such as type and mutability. Column families are divided into the following parts:

  • Column: Each column has a unique name and data type.
  • Partition: Partitions are used to reduce data storage while ensuring data remains efficiently retrievable.

3. Partitioning

Doris supports the following partitioning strategies:

  • Range Partitioning: Partitions based on values, such as partitioning a table by year.
  • Hash Partitioning: Uses hash functions to distribute data across multiple physical storage units.
  • List Partitioning: Partitions data according to a specific order, such as time-ordered data partitions.

4. Primary Key and Unique Index

Doris supports primary keys and unique indexes, which are used for purposes such as uniqueness, non-repetition, and fast retrieval. A primary key is the type of a column in a table, while a unique index involves creating an index on certain columns in a table.

5. Index and View

Doris supports multiple indexing strategies, which are used to accelerate data retrieval, aggregation, and sorting. A view is an abstract layer of a table that can generate a temporary table based on specific query results to simplify complex data operations.

6. Partitioning Dimension and Partition Partitioning

Doris supports multiple partitioning dimensions, which are used to partition table data according to different dimensions. Similarly, partition partitioning can further partition data according to certain rules.

7. Historical Storage

Doris has a feature called historical storage, which can implement the concept of a time machine by replicating part or all of the data into a high-level historical table.

8. Snapshot Storage

Doris supports snapshot storage, which is used to quickly obtain snapshots of specific data. Snapshots can be created based on specific time ranges, conditions, etc.