Deployment and Configuration of Doris Clusters (Using Docker and Other Tools)

introduction

Doris is a distributed transactional SQL database that supports multi-tenancy mode and can achieve horizontal scaling through sharding. Cluster setup and configuration are core components of Doris, and they must adhere to the following principles: availability, reliability, and high performance.

Building a Cluster Based on Docker

First, we need to create a Docker image containing the core components of Doris. We can use the following command to create the image:

docker build -t doris-docker-image .

Then, we can use the following commands to start the cluster:

docker run -d --name doris-master \
  -p 8033:8033 \
  -v /path/to/data:/data \
  -e MASTER_STARTUP_TIMEOUT=30s \
  doris-docker-image

docker run -d --name doris-slave1 \
  -p 8034:8034 \
  -v /path/to/data:/data \
  -e SLAVE_STARTUP_TIMEOUT=30s \
  doris-docker-image

docker run -d --name doris-slave2 \
  -p 8035:8035 \
  -v /path/to/data:/data \
  -e SLAVE_STARTUP_TIMEOUT=30s \
  doris-docker-image

Building a Cluster Based on Kubernetes

We can use Kubernetes deployment YAML files to create a Doris cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: doris-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: doris-master
  template:
    metadata:
      labels:
        app: doris-master
    spec:
      containers:
      - name: doris-master
        image: doris-docker-image
        ports:
        - containerPort: 8033
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: doris-data



apiVersion: apps/v1
kind: Deployment
metadata:
  name: doris-slave1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: doris-slave1
  template:
    metadata:
      labels:
        app: doris-slave1
    spec:
      containers:
      - name: doris-slave1
        image: doris-docker-image
        ports:
        - containerPort: 8034
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: doris-data



apiVersion: apps/v1
kind: Deployment
metadata:
  name: doris-slave2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: doris-slave1
  template:
    metadata:
      labels:
        app: doris-slave1
    spec:
      containers:
      - name: doris-slave2
        image: doris-docker-image
        ports:
        - containerPort: 8035
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: doris-data

Building a Cluster Based on Docker Swarm

We can use the Docker Swarm API to create a Doris cluster:

docker swarm create --endpoint doris-master \
  --token  \
  --stack doris-cluster \
  -c '
    service doris-master {
      replicas: 1
      image: doris-docker-image
      ports:
        - "8033:8033"
      volumes:
        - /path/to/data:/data
    }

    service doris-slave1 {
      replicas: 1
      image: doris-docker-image
      ports:
        - "8034:8034"
      volumes:
        - /path/to/data:/data
    }

    service doris-slave2 {
      replicas: 1
      image: doris-docker-image
      ports:
        - "8035:8035"
      volumes:
        - /path/to/data:/data
    }
  '

Cluster Configuration

Cluster configuration includes Doris startup parameters, sharding information, data storage, and more. We can use the following command to view Doris startup parameters:

docker run --rm doris-docker-image bash -c "doris --show-config"

We can use the following command to view Doris sharding information:

docker exec -it doris-master bash -c "doris --show-partitions"

We can use the following command to view Doris data storage information:

docker run --rm doris-docker-image tar -xvf /data/backup.tar.gz

Conclusion

Building and configuring a Doris cluster is a complex process that requires adherence to relevant principles and best practices. By using tools such as Docker, Kubernetes, or Docker Swarm, we can create a Doris cluster with high availability, high reliability, and high performance.